Re: Proposal for simpler libc rseq registration ABI across libraries

[Mathieu Desnoyers <mathieu dot desnoyers at efficios dot com>]

----- On Mar 27, 2019, at 12:18 AM, Carlos O'Donell codonell@redhat.com wrote:

> On 3/23/19 11:15 AM, Mathieu Desnoyers wrote:
>> ----- On Mar 22, 2019, at 4:51 PM, Carlos O'Donell codonell@redhat.com wrote:
>> 
>>> On 3/22/19 4:27 PM, Mathieu Desnoyers wrote:
>>>> Hi,
>>>>
>>>> Here is a email thread dedicated to discuss a simpler ABI proposal for
>>>> interaction between libc and early adopter libraries/applications with
>>>> respect to registration of restartable sequences (rseq).
>>>>
>>>> This is based on my current proposal
>>>> "glibc: Perform rseq(2) registration at C startup and thread creation (v7)"
>>>> https://lore.kernel.org/patchwork/patch/1041324/
>>>>
>>>> We can move the __rseq_lib_abi TLS refcount and register_state to an external
>>>> library (librseq).
>>>>
>>>> glibc would expose a new global "int" variable symbol __rseq_handled acting as
>>>> a boolean. It would be initially 0. glibc would set it to 1 in its C startup
>>>> code when it effectively handles rseq registration. That symbol would _not_ be
>>>> a TLS (it's global).
>>>>
>>>> librseq would be a new library used by rseq early adopters. It would expose a
>>>> rseq register/unregister API, which internally would:
>>>>
>>>> - Check whether __rseq_handled is true. If so, it would do nothing, leaving
>>>>     rseq registration to the libc.
>>>> - If __rseq_handled is false, deal with many early adopters with TLS refcount
>>>>     and register_state variables internal to librseq.so.
>>>>
>>>> That should take care of minimizing those metrics:
>>>> - libc ABI complexity and maintenance burden in the long term,
>>>> - pain for rseq early adopters when upgrading to newer libc which supports rseq
>>>>     registration,
>>>>
>>>> Does that make sense ?
>>>
>>> Yes. Is the presence of __rseq_handled enough (depends on how you handle
>>> linkage)?
>> 
>> I would define a "int __rseq_handled" within both librseq and glibc.
>> 
>> Only glibc would set it to 1 in the C startup code.
>> 
>> librseq would only read its value to figure out if glibc handles registration.
>> 
>> It would require that the __rseq_handled symbol is not hidden, and that both
>> glibc and librseq populate their symbols into the global symbol table.
>> 
>> If glibc *and* librseq are linked into a process, then the dynamic linker
>> will end up choosing one instance of __rseq_handled and put it in the global
>> symbol table.
>> 
>> If either glibc or librseq are loaded, then we end up having a single
>> instance of the __rseq_handled symbol to place in the table, which is also
>> fine.
> 
> The choice of which __rseq_handled is used is based on the lookup scope
> for the object, so as long as your library is ahead of libc.so.6 in the
> lookup scope then you should be fine and it is in normally.

Even if librseq.so would happen to be after libc.so.6 in the lookup scope,
AFAIU it just means that the __rseq_handled address would sit within libc.so.6
rather than librseq.so. Given the C startup of libc would set __rseq_handled
to 1 before the constructor of librseq.so runs, glibc would "own" rseq
registration, which should all be fine.

> 
> This solution is fine with me.

I prefer the __rseq_handled solution to the rseq wrapper lookup.

The advantage of the __rseq_handled approach is that it can be used as an
ABI between early adopter libraries/applications as well to identify which
early adopter "owns" rseq registration. Basically, the first between constructors
and early main() to observe __rseq_handled == 0 set __rseq_handled to 1 owns
registration.

glibc vs early adopters just becomes an instance of these scenarios.

> 
>>>
>>> I think it's a reasonable design to use a symbol as feature detection for a
>>> feature that exposes no API.
>>>
>>> The alternative would be to implement an rseq wrapper and use the presence of
>>> that wrapper to determine if libc was doing registration. So we use a known and
>>> possibly required wrapper to implement the feature without needing another
>>> symbol.
>> 
>> I fear I'm not fully getting this last idea.
>> 
>> That wrapper would be a function with a public symbol, right ? Where would it be
>> implemented ? glibc or librseq ?
> 
> I'm suggesting that you add a rseq() syscall wrapper to glibc, like we do for all
> the other useful Linux system calls. The presence of the rseq symbol can have two
> meanings, that glibc provides a rseq wrapper for users to use, *and* that if rseq
> is present, glibc will do the automatic registration.
>  
>> Assuming it sits in glibc, how would librseq be able to link against an older
>> glibc which does not contain this symbol, and eventually upgrade to a newer
>> glibc
>> which contains this symbol (and handles rseq registration) ?
> 
> You check at runtime with dlsym (RTLD_DEFAULT, "rseq"); to see if glibc has the
> wrapper, and if it does you assume it is handling the registration, otherwise you
> do the registration yourself.

Got it. That approach would work with statically linked glibc as well.

> 
> Note: I'm avoiding using the weak-ref-and-check idiom here, but it could also be
> used (but has some issues) to support statically compiled applications.
>  
>> I'm clearly missing something here.
> 
> This idea won't work if we don't think rseq() is a useful function call that other
> application authors might want to use. If only glibc would ever use rseq() then
> there is no need to add it to the public API. Libraries need only use __rseq_abi
> to switch critical sections, so they never need to call rseq(). However, maybe in
> the future you forsee applications calling rseq() to do something else, something
> useful for applications, and so then we might just get ahead of that by creating
> the wrapper and using it to test for glibc handling of the registration.
> 
> So rather than:
> * global data symbol __rseq_abi
> * global data symbol __rseq_handled
> * global function symbol rseq()

When you say "global function symbol rseq()" here, so you mean we expose it
publicly from glibc ? I suspect it's not really useful as long as registration
is handled within glibc.

> 
> We have only:
> * global data symbol __rseq_abi
> * global function symbol rseq() / implies registration is handled

Indeed, I don't foresee that rseq(2) will be useful beyond registration, which
is only useful for glibc and early adopters.

Mainly because __rseq_handled allows many early-adopters to "reserve" ownership
of rseq registration, I'd favor the __rseq_handled approach.

Thanks,

Mathieu


> 
> --
> Cheers,
> Carlos.

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com