Re: Hashtable

[Arkady <arkady dot miasnikov at gmail dot com>]

On Thu, Jul 6, 2017 at 9:24 PM, David Smith <dsmith@redhat.com> wrote:
> OK, The first thing to do is run your script with 1 addition. Add
> '-DSTP_ALIBI' to the stap command line. This will compile/run the
> script, but when a probe is hit it will return immediately.
>

I have been through this routine including the -t flag.
10 empty probes introduce ~8% overhead

> This will give you an idea of the overhead of kprobes itself.
>
> Another option would be to remove the '-DSTP_ALIBI' from the command
> line and '-t' to the command line. Here's a description of what that
> option does:
>
>        -t     Collect timing information on the number of times probe executes
>               and average amount of time spent in each probe-point. Also shows
>               the derivation for each probe-point.
>
> Once you've done that, we'll know more of what is going on.

The heaviest probes are thread creation related. The following output
is from the production code. The lines are ordered by MIN time.

index   state      hits  min*     avg       max name
│168     on            1 79908   79908     79908
begin(-9223372036854775808)
│  0     on            1 47496   47496     47496 begin
│114     on           50 16065   70322    528451
kernel.function("SyS_execve@/build/linux-EO9xOi/linux-4.4.0/fs/exec.c:1722").call
│171     on            1 14337   14337     14337
begin(-9223372036854775808)
│172     on            1 13149   13149     13149
begin(-9223372036854775808)
│132     on            1  9525    9525      9525
kernel.function("SyS_accept4@/build/linux-EO9xOi/linux-4.4.0/net/socket.c:1425").call
│121     on            1  8628    8628      8628
kernel.function("SyS_kill@/build/linux-EO9xOi/linux-4.4.0/kernel/signal.c:2847").call
│165     on          708  8247   18919    247540 timer.sec(1)
│145     on            2  6372    8148      9924
kernel.function("SyS_bind@/build/linux-EO9xOi/linux-4.4.0/net/socket.c:1363").call?
│170     on            1  5055    5055      5055
begin(-9223372036854775808)
│117     on           50  4731   11318     28385
kernel.function("SyS_execve@/build/linux-EO9xOi/linux-4.4.0/fs/exec.c:1722").return
│169     on            1  4416    4416      4416
begin(-9223372036854775808)
│124     on            5  4096    5494      7470
kernel.function("SyS_getcwd@/build/linux-EO9xOi/linux-4.4.0/fs/dcache.c:3244").return
│ 39     on          122  3088   11082     39507
kernel.function("SyS_sendto@/build/linux-EO9xOi/linux-4.4.0/net/socket.c:1619").call?

The following gist is a good representation of what I am doing in the
exec https://gist.github.com/larytet/b8155e9d01942984888e90c88c23c206

When I look into the generated by stap C code there are quite a number
of strlcpy() for the filename and args. Some of them are not strictly
necessary, but I understand that a generic framework has costs.

I figure that the arrays are a low hanging fruit. I hope that by
limiting the hashtable API I can make a lockless hashtable along the
lines  https://gist.github.com/larytet/c306d470f7b032c796efad15dcd609a9.

 I expect 15%-30% improvement in the performance overhead. The
improvement will come from the two things

* A single copy of the filename - I am going to copy the filename/args
from the user space directly to the share memory of the driver instead
of the map
* No spin locks are required for the hashmap

Thank you, Arkady.