Re: Question about sparse arrays/atomic operations in systemtap

[Alan Conway <aconway at redhat dot com>]

On Tue, 2014-06-03 at 08:30 -0700, Josh Stone wrote:
> On 06/03/2014 06:34 AM, Alan Conway wrote:
> > I have a question about systemtap. I have a script (attached) for
> > measuring mutex contention (not the futex trick, I am trying to measure
> > directly from pthread_mutex_* calls). Aside from the merits of the
> > script, it has a problem which seems like it would be quite common, I'm
> > wondering if there is a well known solution.
> > 
> > To summarize: I am keeping count of the number of threads that are
> > attempting to lock a mutex at any given time but there is a race:
> > 
> > probe process("/lib64/libpthread.so.0").function("pthread_mutex_lock") {
> >     if (++mutex_locks[$mutex] > 1) // Contended
> >        ...
> > probe process("/lib64/libpthread.so.0").function("pthread_mutex_unlock")
> > {
> >     // TODO: There is a race here:
> >     // - this thread does --mutex_locks[$mutex] and sees 0
> >     // - another thread hits the lock probe and does
> > ++mutex_locks[$mutex]
> >     // - this thread does delete mutex_locks[$mutex] and nukes a non-0
> > value
> >     // Consequence: we could miss contentions. However we can't drop
> >     // the delete or we overrun the mutex_locks array.
> >     if (--mutex_locks[$mutex] <= 0)
> >       delete mutex_locks[$mutex];
> 
> All accessed globals are locked for the duration of the probe, so the
> entire handler is effectively atomic.  I want to think more about what
> might be your root cause, but there's shouldn't be any race.

Ok good to know! So systemtap is doing the right thing, what I'm seeing
is a different race/bug in my script. Will scratch head further.

> 
> > I think what I want is an atomic (decrement and delete if 0) operation
> > on an array.
> > 
> > Another way to solve it (which might have more general use) would be to
> > have a new array type: a 'sparse array' which automatically deletes
> > values when they are set to the array default (0, "", empty statistic
> > etc) Such an array type would be very useful for arrays that are
> > potentially huge (e.g. an entry for every mutex in a process) but are
> > mostly 0.
> > 
> > Is there already a solution to this problem in systemtap, or would the
> > sparse array be a useful addition?
> 
> This might still be a useful feature for brevity and performance.  I'll
> have to check, but ISTR the map runtime even has a flag to do this
> already, so we'd just have to express that desire somehow.
> 

It seems like it would be a common pattern, but given that systemtap is
doing the locking for you it's not that onerous to do the delete
yourself.