Re: kernel summit session on systemtap

[Theodore Tso <tytso at mit dot edu>]

On Fri, Sep 26, 2008 at 03:51:25PM -0400, Frank Ch. Eigler wrote:
> 
> For improving debuginfo quality, yeah.  For subsetting or compressing,
> we can probably attack the problem with separate postprocessing tools
> that could be ready sooner.
> 

I'd recommend it something that could be done sooner if you really
want at least developers like me to use it when they want fast
compile, edit, debug cycles.  I think the right answer has to be not
postprocessing tools (since gcc still has to write out the gargantuan
.o files, which is most of the problem), but some way of extracting
the function type information in a way where function entry and exits
can decode the function arguments.  That's *the* one thing that
Systemtap has over ftrace; ftrace can only tell us that we've entered
a function, and not what arguments it has.  (Well, I guess the other
advantage Systemtap can have is that it if a function entry point is
called a lot, it can be more efficient about only printing information
on some kind of condition, instead of logging output at every single
trace point.  In some cases this will be critical; in other cases,
using "grep" as a postprocessing tool on tracing output can be
sufficient.)

> > > (By the way, do you build distro-style kernels on your laptop, with
> > > allmodconfig or somesuch, or something more linus-sized?)
> > 
> > I do both.  The distro-style kernels are the ones that I build with
> > debuginfo information, and it's been useful for playing around with
> > systemtap, but the moment I need do any serious development work, I
> > tend to fall back to a limited subset of compile options, generally
> > without any modules, and printk debugging.  
> 
> OK; is there some obstruction in the way of using systemtap on your
> 'serious development' kernels?

The main problem is that it takes too long to build the kernels with
-g debugging information.  If on average, the output object files grow
in size by a factor of 5, the the time it takes to build the kernel at
the end of the day tends to grow by somewhere between a factor of 3-5,
since compiles are often write-bound, especially if you are using
ccache.  I just got tired of the increased amount of time to compile
and install debuginfo, especially when more often than not I couldn't
set arbitrary trace points anyway.  Not being put them at static
function point entrypoints was a very rude wakeup call, since ext4 has
a lot of static functions.

> My recollection of the ksummit yak was that the sort of tapset that
> kernel people would be willing to help write/maintain consisted of
> compiled-in instrumentation like markers, or whatever event layer
> comes on top of the new ringbuffer widget.  If that's done right, it
> should not require debuginfo for systemtap to hook in.

No, but then in many cases it's not necessary to use systemtap,
either; we can just grep the output out of the circular ring buffer.
The only time we would need systemtap would be (a) when we can't
anticipant in advance when to put in the markers, and (b) where the
amount of tracing information is too much so to extract what we need
from the trace buffer, such that putting in compild-in conditional is
necessary.  So the real risk for Systemtap as a project is that if
people find they can solve 80-90% of their problems simply by using
ftrace plus markers and grep, there will be much less incentive to
accomodate Systemtap as potential solution that needs to be
accomodated moving forward.

The other question is how many tracepoints get added, and how quickly.
If it's only a core set of 30, then the flexibility of being able to
rely on debuginfo becomes much more important, especially if the cost
of debuginfo can be brought down significantly.  If Systemtap had a
very lightweight way of decoding function arguments without having to
build every single .o file with -g, that would be the equivalent of a
very large number of markers, and at the kernel summit, Linus very
much was against dumping in a large number of markers into the kernel;
and in particular, he didn't want to put *any* more markers or tracing
facilities in until there was tools that were simple and easy enough
to use that even kernel developers could use them could take advantage
of the existing markers and tracing facilities.

That being said, it could be that if we can get Google top-30
tracepoints, and we can figure out some cool ways Systemtap could use
those tracepoints that wouldn't necessarily be as easily by replicated
using LTTng plus grep/awk, it might be a good way of convincing people
that it's worthwhile to give Systemtap another try.

						- Ted