pthread_cond performence Discussion

[Carlos O'Donell]

On 3/18/20 8:53 AM, Torvald Riegel wrote:
> On Wed, 2020-03-18 at 08:12 -0400, Carlos O'Donell wrote:
>> On 3/16/20 3:30 AM, liqingqing wrote:
>>> The new condvar implementation that provides stronger ordering
>>> guarantees. For the waiters's ordering without expand the size of
>>> the
>>> struct of pthread_cond_t, It uses a little bits to maintain the
>>> state
>>> machine which has two different start group G1 and G2. This
>>> algorithm
>>> is very cleverly. But when I test MySQL performance and found that
>>> this new condvar implementation will affect the performance when
>>> there are many cores in one machine. the scenario is that in my arm
>>> server, test 200 terminals to read and write the database in 4P
>>> processor environment(totally 256 cores), and I found that It can
>>> get
>>> better performance when I use the old algorithm. 
>>
>> Are you able to look at any hardware performance counters to see if
>> there are increased cache line miss rates?
>>
>>> I think maybe there has too many cache false sharing when in my
>>> environment. Does anyone has the same problem? And is there room
>>> for
>>> optimization about the new algorithm?
>>
>> I have not seen anyone report a performance problem on large
>> machines.
>>
>> Unfortunately from an ABI perspective we cannot increase the size of
>> the structure, nor change the required alignment.
>>
>> We may be able to play with the order of the layout of elements
>> within the condvar. That's something you could experiment with and
>> report back to the list with your findings.
>>
>> For example:
>> - Make changes the layout by moving elements around to attempt to
>>   avoid cache-line sharing.
>> - Recompile glibc.
>> - Install into your system.
>>   - PTHREAD_COND_INITIALIZER should be all-zero bytes so you should
>>     not need to recompile applications.
>> - Retest performance.
> 
> The other thing I would recommend is to investigate whether you can
> improve the synchronization in MySQL.  Condition variables are just one
> way to do synchronization, and the root synchronization problem to
> solve is in the application.  This could potentially give you a much
> bigger performance improvement than any optimization of the condition
> variable implementation could. 
 
Agreed. The ABI limitations on the current interface are the biggest
problem. To fix the ABI issues would require structure allocation and
freeing in the background, which has it's own issues and performance
impact (requires fallback PSHARED handling). Therefore if you could
parallelize differently then it would certainly help. I expect that
this is harder than you think though since it requires refactoring
the locking in mysql. While making the current condvar layout more
cahceline friendly (even probablistically so) would improve all
applications using it.

-- 
Cheers,
Carlos.