Re: Systemtap caching behavior

[Mikhail Zhukovskiy <mikhail dot zhukovskiy at oracle dot com>]

Hi Mark,

Again, thank you for all your help!

To give you a bit of background on my problem, after patching the kernel from 3.13 to 4.4, we started seeing periodic 1-2 minute events where the BIND DNS daemon goes into “uninterruptible sleep” and becomes unresponsive. My effort here is about figuring what what system calls BIND is waiting on when that happens.

I have the kernel -dbgsym package installed (doing this on Ubuntu).

vagrant@dns4-01-us1:~$ dpkg -l | grep dbgsym
ii  linux-image-4.4.0-133-generic-dbgsym 4.4.0-133.159~14.04.1             amd64        Linux kernel debug image for version 4.4.0 on 64 bit x86 SMP

Here is the link to all my source code and a complete run.

https://gist.github.com/misha354/15f74ad50c4fe67fe5ae2b9e046fe2f6

I am attaching the resulting flame graph. As you can see, the base of every call stack is entry_SYSCALL_64_fastpath(). I am hoping to see the BIND functions that are calling it. I’m not sure what other steps I need to take make that happen.


Thank you,
Mike


> On Jan 13, 2019, at 11:22 AM, Mark Wielaard <mark@klomp.org> wrote:
> 
> On Sun, Jan 13, 2019 at 09:58:39AM -0500, Mikhail Zhukovskiy wrote:
>> One more, perhaps naive question. What I am trying to do is generate fame graphs per https://urldefense.proofpoint.com/v2/url?u=http-3A__brendangregg.com_FlameGraphs_cpuflamegraphs.html&d=DwIBAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=1tDlnkfhOa3yKtPzIEgoBg8s2PlcIN5CQWpRGelmq-Y&m=ghzS2Dvh4MdfpRduK1G2gYXsy3n1CeV0-TRZLzr2Odo&s=P9ySoA2ptCOLPVUIQ7-82Zh_l0HeYpYklWFn7UETF34&e= . At the moment, I have succeeded in making the flame graphs in which the call stacks begin with entry_SYSCALL_64_fastpath. If I wanted to include the user space calls that made these system calls, would my next step be to re-compile the application that we are profiling (the BIND DNS daemon) with one of the gcc debug flags (https://urldefense.proofpoint.com/v2/url?u=https-3A__gcc.gnu.org_onlinedocs_gcc_Debugging-2DOptions.html-29-3F&d=DwIBAg&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=1tDlnkfhOa3yKtPzIEgoBg8s2PlcIN5CQWpRGelmq-Y&m=ghzS2Dvh4MdfpRduK1G2gYXsy3n1CeV0-TRZLzr2Odo&s=4NvgYM4LhMwJgO7Kgl8GAYMaWHCpMDmVHTg-m366who&e=
> 
> For just backtraces in user space on amd64 you shouldn't need to rebuild
> userspace, since amd64 includes unwind tables by default. So backtraces
> done from user space probes should work as is. But if you are unwinding
> from kernel space into user space you might need the kernel debuginfo
> installed (since the kernel normally doesn't include unwind tables,
> so the will need to come from the .debug_frame).
> 
> Cheers,
> 
> Mark