Getting user-space stack backtraces in more probe contexts

[William Cohen <wcohen at redhat dot com>]

One of the common tasks that people would like to use systemtap for is
to identify where lock contention is happening in user code.  The
futexes.stp example already in sytemtap will identify which processes
are waiting for locks, but it does not identify where in the code that
the waiting is occurring. What would like is to be able to get a
user-space stack backtrace to provide the developer some context where
the FUTEX_WAITs (and other syscalls) occur in the code.  Ideally,
would just like to be able to put a ubacktrace() in the syscall.futex
probe of futexes.stp. However, the ubacktrace() does not work in the
kernel contexts, e.g. kernel.function("*"); ubacktrace() only works in the
user-space process and profile probes where a pt_regs data struct
initialized to with userspace register values is available.

An alternative approach would be to have the kernel-space
syscall.futex probe record that a thread was waiting and then in
user-space record the information.  This is complicated by the
multiple sites in the glibc pthread library that make the futex system
call. It also means that futex use by other libraries might might be
missed unless they are instrumented in the same way as the glibc
library.

Previous work placed user-space probes in glibc pthread, but the
uprobes overhead was too high. An implementation with static markers
was generated to reduce that overhead:

http://sourceware.org/ml/systemtap/2009-q1/msg00502.html

However, these patches are not merged into the any of the packages or
the upstream glibc.

Took a closer look at how the systemtap runtime stack.c and uprobes
code operates to see whether there is some way to get the appropriate
location to start the userspace unwinding even when in a kprobe or
other kernel context.  Stack unwinding works for uprobes. The runtime
code uses the register_uprobes() and register_uretprobes() to insert
the breakpoint in the user-space code and adds a handler for that.

When the breakpoint from the registration is encounted in the
user-code a trap occurs. The CPU's user-mode registers are saved and a
SIGTRAP signal is generated. This signal is handled by a utrace engine
executing uprobe_report_signal() in uprobes.c.  The utrace engine
passes a pointer to the pt_regs into the uprobe_report_signal().

The main mechanism used by most user-space processes to get into
kernel code is syscalls using either an int 0x80, syscall, or
sysenter.  The mechanisms used for the syscalls do not store data in a
pt_regs structure.  the code that handles the syscall entry into the
kernel for x86_64 is located in linux/arch/x86/kernel/entry_64.S
ENTRY(system_call). Looking through the code for the syscall_call
entry see the old stackpointer getting saved and changed to the kernel
stack:

        movq    %rsp,PER_CPU_VAR(old_rsp)
        movq    PER_CPU_VAR(kernel_stack),%rsp

Also ran across the following comment and macro in entry_64.S creat
pt_regs for some functions:

 /*
  * Certain special system calls that need to save a complete full stack frame.
  */
         .macro PTREGSCALL label,func,arg

The syscalls using it are:

        PTREGSCALL stub_clone, sys_clone, %r8
        PTREGSCALL stub_fork, sys_fork, %rdi
        PTREGSCALL stub_vfork, sys_vfork, %rdi
        PTREGSCALL stub_sigaltstack, sys_sigaltstack, %rdx
        PTREGSCALL stub_iopl, sys_iopl, %rsi

Looked at how the backtrace mechanism worked in perf to see if
something from that mechanism can be borrowed. Has "perf record" has
"-g" option. This sets the PERF_SAMPLE_CALLCHAIN flag in the
attributes. The sampling mechanism in the kernel for perf includes a
pt_regs entry, but it looks like it only records backtrace for
hardware performance events, which cause interrupts.  When the perf
sample is recorded with with the "-g" option, the perf_callchain()
function is called. The perf_callchain_user() and
perf_callchain_user32() do the actual walk of the stack.  It looks to
be a simple minded frame-based mechanism.