Re: Differences of stack traces between linux kernel and systap

["Peter Teoh" <htmldeveloper at gmail dot com>]

On 5/3/08, Andy Gospodarek <gospo@redhat.com> wrote:
> On Sat, May 03, 2008 at 12:35:41AM +0800, Peter Teoh wrote:
>  > Notice the diference in the stack trace between systap:
>  >
>  >  0xc047ecf1 : slab_pad_check+0x1/0xb9
>  >  0xc064dd69 : kretprobe_trampoline_holder+0x18/0x38
>  >  0xc064dd69 : kretprobe_trampoline_holder+0x18/0x38
>  >  0xc064dd69 : kretprobe_trampoline_holder+0x18/0x38
>  >  0xc0434e33 : send_signal+0x4a/0x10e
>  >  0xc0435476 : __group_send_sig_info+0x6e/0x8c
>  >  0xc0435a8e : group_send_sig_info+0x43/0x58
>  >  0xc048d42a : send_sigio+0xe3/0x133
>  >  0xc048d4c9 : __kill_fasync+0x4f/0x52
>  >  0xc048dc78 : vfs_ioctl+0x0/0x6c
>  >  0xf8943ef5 : snd_timer_user_tinterrupt+0x11f/0x13c [snd_timer]
>  >  0xf8942c74 : snd_timer_interrupt+0x1de/0x23f [snd_timer]
>  >  0xf896667d : snd_pcm_period_elapsed+0xf0/0x134 [snd_pcm]
>  >  0xf8bfef49 : azx_interrupt+0x60/0xa4 [snd_hda_intel]
>  >  0xc0459eab : handle_IRQ_event+0x26/0x4d
>  >
>  >
>  > And that of the linux kernel (latest patch):
>  >
>  >  [<c042bad6>] warn_on_slowpath+0x46/0x56
>  >  [<c0415a33>] ? apic_wait_icr_idle+0x16/0x1d
>  >  [<c0415243>] ? __send_IPI_dest_field+0x50/0x54
>  >  [<c04020e5>] ? send_IPI_mask+0xd/0xf
>  >  [<c046773c>] ? get_pageblock_flags_group+0x50/0x6e
>  >  [<c046777e>] ? get_pageblock_migratetype+0x24/0x27
>  >  [<c0468472>] ? free_hot_page+0xf/0x11
>  >  [<c0468494>] ? __free_pages+0x20/0x2b
>  >  [<c047f471>] ? __free_slab+0xac/0xb4
>  >  [<c0480754>] kmem_cache_destroy+0xfe/0x108
>  >  [<f8d337c0>] nf_conntrack_cleanup+0x53/0x7a [nf_conntrack]
>  >  [<f8d3766d>] nf_conntrack_standalone_fini+0x1c/0x1e [nf_conntrack]
>  >  [<c044b56f>] sys_delete_module+0x177/0x1af
>  >  [<c0472c00>] ? remove_vma+0x31/0x53
>  >  [<c0473468>] ? do_munmap+0x182/0x19c
>  >  [<c0404bae>] sysenter_past_esp+0x6a/0x90
>  >  [<c0640000>] ? pci_scan_bridge+0x1dc/0x2eb
>  >
>  > The difference lies in the "?" in front of the function, which is to
>  > indicate that this function has just been executed and returned, so it
>  > is no longer on the stack.
>  >
>
>
> I'm pretty sure that on x86 arch's the kernel's stack traces are really
>  more like intelligent guesses rather than guarantees.  The kernel
>  searches back though it's stack and checks for possibly valid function
>  addresses and prints those out.

Yes, u are right, and I just asked this question in depth here:

http://mail.nl.linux.org/kernelnewbies/2008-05/msg00008.html

Hopefully someone can answer.

>
>  Other arches (like some PPC flavors) do this much better since they have
>  a dedicate register for storing stack frames and pre-defined stack frame
>  sizes and parameters.  Those are much more trustworthy, but I find the
>  backtraces on x86 to be quite usable.
>
>
>  > Is it possible to modify the systap stacktrace dump to be inlined with
>  > that of the kernel?   It this not more informative?
>
>
> Can you quantify the performance impact of something like this running
>  all the time?
>
>


-- 
Regards,
Peter Teoh