Re: "lttng comparison with other tracers" page

[Mathieu Desnoyers <compudj at krystal dot dyndns dot org>]

* Josh Stone (jistone@redhat.com) wrote:
> Frank Ch. Eigler wrote:
> >> make clean
> >> ./configure --disable-pie
> >> make
> >> make install
> >>
> >> ok:/usr/local/bin# ./stap -e 'probe kernel.trace("*") {}' -w -t
> >> semantic error: cannot get type of tracepoint 'swap_in' parameter 'entry'
> >> Pass 2: analysis failed.  Try again with another '--vp 01' option.
> >> The swap_in tracepoint has the following parameters :
> >> void probe_swap_in(struct page *page, swp_entry_t entry)
> > 
> > I believe this can happen if the tracepoint declaration header file
> > does not #include all the headers needed to declare all of its types.
> > This is probably a problem worth fixing in your files.  Is that right,
> > Josh?
> 
> Actually, if the #includes were wrong, then our query module would fail
> altogether, so we wouldn't see ANY tracepoints.
> 
> The problem here is a bad assumption on my part, that the resolved types
> would always be either POD or pointers.  In this case, swp_entry_t is a
> "struct { unsigned long val; }" that is passed by *value*, which I
> didn't expect.  I suppose this one could be faked to a plain ulong, but
> it would be better to support the actual struct.  I've filed two bugs:
> 
>   tracepoints should gloss over unrecognized parameter types
>   http://sources.redhat.com/bugzilla/show_bug.cgi?id=9959
> 
>   tracepoints should support complex parameters by-value
>   http://sources.redhat.com/bugzilla/show_bug.cgi?id=9960
> 
> The first is a band-aid, and the second should make this work completely.
> 
> 
> >> So I created a small script that enables only part of the lttng
> >> instrumentation, ran with stap -t -w scripname :
> >>
> >> probe kernel.trace("irq_entry") {}
> >> probe kernel.trace("irq_exit") {}
> >> probe kernel.trace("syscall_entry") {}
> >> probe kernel.trace("syscall_exit") {}
> >> probe kernel.trace("sched_switch") {}
> >> probe kernel.trace("sched_process_fork") {}
> >> probe kernel.trace("sched_process_wait") {}
> >> probe kernel.trace("sched_process_free") {}
> >> probe kernel.trace("sched_process_exit") {}
> >>
> >> And as soon as I start the tbench test, stap stops with :
> >>
> >> WARNING: Number of errors: 0, skipped probes: 101
> >> WARNING: Skipped due to reentrancy: 101
> > 
> > You can try rerunning with -DMAXSKIPPED=99999 ish.  Josh was also
> > tracking down a problem with this area - is there a bz# for this one
> > yet?
> 
> I looked at it for a while, but I couldn't determine anything more than
> simple interrupt-reentrance.  The above tracepoints are hit often enough
> that even a low-rate of reentrance will add up quickly.
> 
> I did add a "-DINTERRUPTIBLE=0" option to disable interrupts during our
> handlers, which eliminated the problem as far as I could see.
> 
> 
> 
> Josh
> 

Hrm, the systemtap code should always expect reentrancy coming from
interrupts. The way to deal with this is either to _always_ disable
interrupts around the probe code, or to use atomic operations within the
probe and allow nested probe execution at any given point of the probe.

Mathieu

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68