Re: Unsafe mode for probes

[Arkady <arkady dot miasnikov at gmail dot com>]

David,

The SystemTap introduces overhead.

My typical probe looks like this

probe syscall.accept4.return ?
{
    %{HIT_MAP_INC(HIT_MAP_SYSCALL_ACCEPT4_RETURN)%}
    send_incident_result(%{INCDENT_TYPE_SYSCALL_ACCEPT4%}, $return);
}

where send_incident_result() is my C function. The function pushes ~16
bytes to a lockless FIFO in the shared memory.

The actual assembler of the probe will contain quite a bit of code
handling local variables, call to lock, etc
For short probes I can shave about 20% of CPU cycles.

Thank you, Arkady

On Mon, Aug 6, 2018 at 8:36 PM, David Smith <dsmith@redhat.com> wrote:
> For your "unsafe probes", why not just do something like the following?
>
> ====
> probe syscall.read {
> %{
> // C code here ...
> %}
> }
> ====
>
> On Sun, Aug 5, 2018 at 5:14 AM Arkady <arkady.miasnikov@gmail.com> wrote:
>>
>> Hi,
>>
>> I think that I have mentioned the problem before. I allocate
>> significant amount of virtual memory and create a shared memory
>> between my kernel drivers and code running in the user space. The goal
>> of the shared memory is to improve the throughput of the data channel
>> between the systemtap driver and the rest of the system.
>>
>> I started by using 'probe begin' to call my C initialization code.
>> This approach expectedly failed.
>>
>> My next step was a small kernel module which only initializes the
>> shared memory. Finally
>> I have patched the systemtap by adding two hooks
>> stp_user_init()/stp_user_close()  in
>> emit_module_init()/emit_module_exit() in the translate.cxx See, for
>> example, https://github.com/larytet/SystemTap/commit/6e4c99f96c9ecce9508f2c8612e8bace2ac91ae5
>> This worked just fine.
>>
>> Still performance wise the driver is not where I would like it to be.
>> I collect majority of the system calls  and install 100+ probes. The
>> code has to stand in tight performance constraints. I improve the
>> performance further by replacing the most frequently called probes
>> generated by the systemtap by custom C code. Every time the systemtap
>> version/kernel version changes I have to rewrite the probes. I do not
>> want to drop the systemtap because the framework solves lot of
>> compatibility related problems.
>>
>> I think to extend the STAP language by adding an "unsafe probe" - a
>> probe implemented in C. Unsafe probes do not enforce any checks. The
>> framework will "emit" access to the system call arguments and compile
>> the code. The framework will use the original stack for the local
>> variables.
>>
>> I understand that this is a long way from the SystemTap stated
>> intention of the system debug. I am breaking quite a few core
>> assumptions. Indeed I use the SystemTap to implement a high
>> performance system audit for Linux.
>>
>> I appreciate any feedback
>>
>> Thank you
>
>
>
> --
> David Smith
> Associate Manager
> Red Hat