Re: architecture paper draft

[Vara Prasad <prasadav at us dot ibm dot com>]

Hi,

   I am not sure if my original comments reached through listserver or 
not.
   I am sending from IMAP account so that listserver wont bounce.

Hi Frank,

   Here are some of my comments on the draft paper.
  
   I am not sure if you have an updated version but my
   comments are based on the initial document that you sent me.
   As i have been writing these comments in my copious spare time
   it took me a while from the start to finish hence some of the comments
   might not have addressed latest discussions.

   General: Paper doesn't seem to refer systemtap as SystemTAP the original
   name we started with. Paper also seems to use breakpoint to refer to
   a probe location. I think it would be less confusing if we use probe 
point
   rather than break point.
  
   In the motivation section you mentioned "Red Hat is forming a new 
project named systemtap"
   don't you think you should include IBM here.
      
   You mentioned "The output should be available in multiple formats".  
Do you mean by output
   in text and graphics format or something else.

   In the requirements we should add either under simplicity or ease of 
use,
   Tool should be usable on systems which don't have Linux source code 
installed.
   Users should be able to trace the system without needing to develop 
their own probes.

   We should add a definitions section before we go to describe the 
architecture.
   The definitions we should include but not limited to
   Probe: A trap in the Kernel or user address space.
   Probe handler: A function to run when a probe point is hit.
           Probe handler runs in the kernel address space.
   Probe Group: A group of probes in a given functional area of the kernel.
                   For example probe handlers for all the system calls.

   How do we uniquely identify a probe in the system.
   <probe group>:<function >:<where>
   examples for kernel probes would be syscall:write:entry, vm:pagin:: etc.

   One issue with the above notation is how do we specify the probes for
   user space. From an implementation point of view inode and offset 
uniquely
   identifies a place to put the trap instruction but from a user point 
of view it should
   be more in terms of the process. We could go the route of PID 
provider Sun has
   taken but it needs more thought. Sun has module name in the above 
notation
   that they use to identify libraries while tracing user applications. 
I think we should
   be o.k without that but one has to specify individual function names 
in this nomenclature.

   Another disadvantage would be for subsystems where there several 
choices like filesystems,
   device drivers each of them have to implement their own probe group. 
It is o.k
   to do that but it doesn't have any correlation to the logical 
ordering of the subsystems
   in the kernel. What i mean by logical ordering is ext3 and raiserfs 
are filesystems and
   they are mutually exclusive and hook it up through VFS. A 
nominicleture of the kind
   that includes <subsystem>.<comoponent> would address this issue of 
representing
   relationship in the probe definition but the syntax would become too 
big and wont
   be simple like a script.

   Another related area that we need to have some details in the spec 
is user space
   probes and how we handle them, what happens during the fork etc.

   Probe language
   I like the idea of probe as the keyword rather than the break, 
similarly for globals
   identify a section called globals and declare all the globals in 
that section.
   Ofcourse we can have multiple global sections if we choose or to 
start with
   we could only allow one global section for the script.

   The syntax illustrated in the probe language section is more "C" 
like than script like.
   I like the syntaxt used in the dtr proof of concept more than the 
one in the probe language section.
   I would prefer to use self or this instead of user to refer to the 
caller. For frequently used
   datastructures like current and pid we should have a macro to refer 
to them like $CURRENT, $PID etc.
  
   We should also provide constructs like iterators to traverse lists 
of datastructures in the
   kernel that probes might access. We all printed lists while 
debugging, i think it would
   be useful.

   I think we should also look at scripting language like perl or 
python and adapt extensions from
   those, These languages have all the characteristics of awk from our 
needs point of view.
   Using awk would make people feel we are copying Sun, just a thought.

   The paper only seems to deal with function entry and exit probes 
which have well known
   addresses.  To start with that might be o.k. When i talk to 
developers they seem to feel
   they need to able to add probes in the middle of the function. We 
have to mention what
   is our strategy to deal with this issue. Nop markers is one way of 
identifying these locations,
   but we have to evaluate other options and need for these markers.

   Elaboration:
   The definition of SystemTAP provider in this section is not precise, 
one example refers to
   access to global data other example refers to probe location. I 
think we should come up with
   a definition that encompasses all aspects of providers. I understand 
providers could
   have varied functionalities.
  
   I have started some initial discussion about the probe group 
specification above.

   Translation
   What is a runaway-prevention logic?  You mean infinite loops.
   Paper mentions variables shared among probes, i am thinking you are 
referring to globals
   of the probe module here, am i right.
   Paper also mentions "Each group of local variables is placed into a 
synthetic call frame structure
   that keeps them off the tiny real kernel stacks.". It is not clear 
to me which local variables
   you are referring to. What do you mean by "tiny real kernel stacks". 
I think expanding this sentence
   possibly including some examples would be easy to follow.

   I agree that inode+offset way of specifying probe points is more of 
an implementation method
   of specifying rather than user way of specifying. I think users 
associate more easily with
   a process approach. As i mentioned above this presents some 
interesting issues to deal
   with fork etc.? Sun seems to have avoided this issue by saying PID 
provider can only be
   used on running processes.

   This section doesn't address how we are addressing the issue of 
statically defined structures
   which are passed in as arguments to the probing function and we are 
generating a jprobes
   module.

   Output
   I think getting the output via pritnk is in messagelog makes it 
difficult to parse the output, and display
   in a readable form. I would strongly suggest staying away from the 
printk approach.
   Paper mentions that systemtap infers relationship between arrays, 
where do you think we can
   do this in user space or kernel and i think we need to provide more 
details of how is implemented.

   Formatted output paper mentioned, like XML, i am assuming is done in 
user space not in Kernel,
   community will not approve XML formatting in kernel.

   Security
   I am not sure how many users really like to analyze their programs 
without admin privileges,
   hence i suggest systemtap should be runnable only by root users.

   The last paragraph tries to address the safety concerns but it was 
generic in nature.
   We should provide more specifics with examples of how we address the 
safety issues
   for common problems like divide by zero, accessing address space outside
   the scope of the process, etc. I am not sure there is any concern 
about C compilers
   reliability as mentioned in the paper, i think if any there will be 
more doubt on the
   reliability of an interpreter due to it's maturity. The main issue i 
think here is an interpreter
   has more control on the code being interpreted when there are errors 
unlike an executable code.
   We have to give more details about our exception handling to address 
this
   concern.

   Please let me know your thoughts and comments.
      
bye,
Vara Prasad

Frank Ch. Eigler wrote:

> Hi -
>
> I committed a first draft version of one proposed architecture in the
> systemtap CVS repository.  Check out the "archpaper" module using
> anoncvs [1].  Or use a real ssh account so you can eventually contribute
> directly (have you signed up yet?).  Just for gags, temporarily
> (since it's such an early draft) I put formatted version of the
> LaTeX paper up at home:
> <http://web.elastic.org/~fche/systemtap.pdf>
>
> - FChE
>
> [1]
> cvs -d :pserver:anoncvs@sources.redhat.com:/cvs/systemtap login
> {enter "anoncvs" as the password}
> cvs -d :pserver:anoncvs@sources.redhat.com:/cvs/systemtap co archpaper
>
>