addr2line keeps me sane

Doing kernel work, I end up with a lot of text dumps of things. It's typical to get lots of junk that looks like gibberish:

[71818.339389] bash            S 0000000000000000     0  3829   3753 0x00000000
[71818.339389]  ffff88007a5fdd38 0000000000000086 0000000000000001 0000000000011e80
[71818.339389]  0000000000000000 ffff88007af31080 ffff88007bc15040 ffff88007fc11e80
[71818.339389]  ffff88007af31080 0000000000000000 ffff88007fc11e80 0000000000000000
[71818.339389] Call Trace:
[71818.339389]  [] ? check_preempt_curr+0x7a/0x90
[71818.339389]  [] ? try_to_wake_up+0x1e5/0x280
[71818.339389]  [] schedule+0x45/0x60
[71818.339389]  [] schedule_timeout+0x14f/0x250
[71818.339389]  [] n_tty_read+0x2f0/0x810
[71818.339389]  [] ? try_to_wake_up+0x280/0x280
[71818.339389]  [] tty_read+0xa6/0xe0
[71818.339389]  [] vfs_read+0xcb/0x170
[71818.339389]  [] sys_read+0x55/0x90
[71818.339389]  [] system_call_fastpath+0x16/0x1b

Let's say you were trying to interpret this stack trace. Sometimes, the compiler will inline function calls and they might not show up in a stack trace, so it is not immediately apparent how tty_read() might call try_to_wake_up(). You can disassemble or use a debugger, but those both require skill. I prefer to replace having skill with tools instead, which is why I love addr2line. You need to feed it a vmlinux (not a vmlinuz or bzImage mind you), but its output is wonderful:

dave@kernel:~/linux-2.6.git$ addr2line -e vmlinux ffffffff81389ff6
/home/dave/work/linux-2.6.git/drivers/tty/tty_io.c:959
dave@kernel:~/linux-2.6.git$ vi /home/dave/work/linux-2.6.git/drivers/tty/tty_io.c +959

Which points to:

                i = (ld->ops->read)(tty, file, buf, count);
        else
                i = -EIO;
        tty_ldisc_deref(ld); <------------------
        if (i > 0)
                inode->i_atime = current_fs_time(inode->i_sb);
        return i;
}

and it's fairly easy to follow the call path from there.