Re: whitelist for safe-mode probes (or just a better blacklist?)

[Li Guanglei <guanglei at cn dot ibm dot com>]

Vara Prasad wrote:
> Martin Hunt wrote:
>
>
> It shouldn't be that difficult to use DWARF library to generate all 
> exported functions in the kernel. I am only focusing on exported 
> functions first as their interfaces are more stable then some internal 
> functions but this method can work on any function. If there happens to 
> be a function if one of our tapsets is probing that is not in the above 
> list we should add those functions as well. Once we have the function 
> names, generate a script that puts probes in some percentage of the 
> probes let us say 10% at each time in a sliding window. Loads the 
> generated module and runs a standard test like ltp for 10 mins. The 
> content of the probe handler should be to print the name of the 
> function, increment a counter and also print some golbal variables like 
> PID, GID etc. After being done with the whole list of the functions we 
> should then generate a script that puts the probe in all the functions 
> in the white list and runs few standard tests like ltp, fstest etc for 
> 30 min to make sure probing all of the functions doesn't cause any 
> instability problems.
>
> Once we agree upon a format we can run these tests as part of the weekly 
> test we are doing so we can catch problems early.  Over a period of few 
> weeks we can come up with a decent list that we feel comfortable. Once 
> we have a big enough of safe list translator by default for wild card 
> expansion consult this black list and white list and expand only to the 
> function names from this list. We should also provide a way for us to 
> indicate the translator i am testing i don't want you to restrict to 
> only white list so do the real expansion of wildcards.
>
> A side effect of this work could be after few weeks of results we can 
> identify safe to probe routines we could probably even go a head and put 
> some gcc magic macros in the kernel code itself that gives us info in 
> the ELF section to say what functions are deemed safe to put probes. 
> That way over a period of time we may not have to ship separate white 
> list, but that is for future (now i am day dreaming :-) ).
>
> Anyone got tomatoes?
>
> bye,
> Vara Prasad

Hi,
  I used: stap -e 'probe kernel.function("*") {}' -p2 -v  | grep 
"kernel.function" | wc -l, and it shows me 10827 functions will be probed.

  As suggested, we divide all the functions into groups. The number 
of group can't be too big since we must the run the test enough long 
for each group. So there will be quite some functions(~1000 maybe) in 
each group. How about if one of the groups crashes the kernel? In most 
cases we can't know which functions cause the problem so we have to 
shrink the scope by and by to put the functions inside this group 
gradually into the whitelist, but this will cause a lot of work. A bad 
situation is that all the groups will crash the kernel.

  Apparently those groups that pass the tests can't declare all 
functions contains inside them are safe. Maybe some functions were 
never triggered during the tests or only were triggered a few times 
and didn't came across the dead condition. If one day we find probing 
the whole whitelist crashes the Kernel, we have to take pains to find 
out which one in the whitelist has the problem. And found a suitable 
testcase that will trigger all the probes is a hard task.

  So after thinking about this topic, the whole work may not be an 
easy task. Maybe finally we find we spent too much time to get the 
whitelist.

  Just my random thoughts.

- Guanglei