Re: How to track the functions in self-written module using SystemTap?

[David Smith <dsmith at redhat dot com>]

On 12/04/2015 03:56 AM, Nan Xiao wrote:
> Hi Dave,
> 
>> It looks like you've got a couple of options:
> 
>> 1) Copy your module into the kernel module tree (located at
>> /lib/modules/`uname -r`/kernel) to use systemtap on it.
> 
> I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always
> executes error:
> 
> # cd /lib/modules/`uname -r`/kernel
> # pwd
> /lib/modules/3.10.0-123.el7.x86_64.debug/kernel
> # ls
> arch  crypto  drivers  fs  kernel  kex.ko  lib  mm  net  sound
> 
> # stap -v -e 'probe
> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
> { printf("%s\n", ppfunc()) }'
> Pass 1: parsed user script and 103 library script(s) using
> 214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms.
> semantic error: while resolving probe point: identifier 'module' at <input>:1:7
>         source: probe
> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
> { printf("%s\n", ppfunc()) }
>                       ^
> 
> semantic error: no match
> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
> global(s) using 372028virt/33024res/3412shr/30124data kb, in
> 20usr/770sys/804real ms.
> Pass 2: analysis failed.  [man error::pass2]
> 
> # stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n",
> ppfunc()) }'                                    Pass 1: parsed user
> script and 103 library script(s) using
> 214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms.
> semantic error: while resolving probe point: identifier 'module' at <input>:1:7
>         source: probe module("kex.ko").function("*") { printf("%s\n",
> ppfunc()) }
>                       ^
> 
> semantic error: no match
> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
> global(s) using 372032virt/33016res/3412shr/30128data kb, in
> 40usr/880sys/929real ms.
> Pass 2: analysis failed.  [man error::pass2]

I see I didn't explain well here. Once you copied your module to the
kernel's module directory, systemtap no longer would need a full path to
the module name - just "kex" should have worked.

>> 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1
>> system, and the modules_out_of_tree.exp test case passed there.)
> 
> I install the RHEL 7.2, and it really works! But it can't print actual
> stack traces:
> 
> # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") {
> print_backtrace() }'
> WARNING: no or bad debug frame hdr
> WARNING: No binary search table for debug frame, doing slow linear
> search for /root/kernel/105.ops/kex.ko
>  0xffffffffa006b000 : kex_init+0x0/0x0 [kex]
>  0xffffffff810020e8 (inexact)
>  0xffffffff810ed4ae (inexact)
>  0xffffffff81316880 (inexact)
>  0xffffffff810e9743 (inexact)
>  0xffffffff810ede66 (inexact)
>  0xffffffff81645909 (inexact)
>  0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex]
>  0xffffffff810eb23b (inexact)
>  0xffffffff81641113 (inexact)
>  0xffffffff81014b12 (inexact)
>  0xffffffff81645909 (inexact)
> 
> So it means the module isn't built with debuginfo, right?

Hmm, it could mean that we just need to make sure that the debuginfo is
loaded. Try adding the '-d  /root/kernel/105.ops/kex.ko' option to the
stap command line. (The '-d' option asks stap to load the symbol/unwind
information for a module even if stap doesn't think it needs it.)

-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)