Re: [PATCH] Improve string benchtests

[Wilco Dijkstra <Wilco dot Dijkstra at arm dot com>]

Hi Adhemerval,

> So my point is to which exactly should we compare on benchtests? Current we have:
>
>  1. Byte-oriented 'simple' implementation which, as we agree, should not be
>     used as a baseline.

That is for functions like memcpy where you'd expect any implementation -
including generic - to act on words rather than bytes. So for those comparing
with a byte-oriented version doesn't add anything useful.

A byte-oriented version does make sense when it is competitive with the
generic implementation. This is true for various wcs functions and strstr for
example.

>  2. Some named 'stupid' which are usually composed implementation that might
>     in fact be a faster implementation than some 'clever' ones.

In all cases I found the stupid ones were slower, mostly because they first
called strlen and then still processed the string one byte at a time instead of
calling memcpy with the known size. So they weren't adding useful data.

>  3. Compiler builtins, which also does not represent meaningful data for
>      libc optimization (it will either be inline, call libc implementation, or
>     mix both strategies).

Agreed - there are few cases where string functions with non-constant inputs are
inlined, so in most cases you're just benchmarking the libc version.

>   4. The libc implementations themselves, possible including all ifunc variations.
>
> So which really give us meaningful data for future optimization? Should we keep
> add multiple implementation as baselines to compare with?

Well it is useful to check different strategies as well as comparing similar string
functions. For example I noticed that on some microarchitectures memchr was
faster, but on others strlen is faster. Given the individual benchmarks use very
different inputs, you only notice this in a direct comparison. 

> What about an architecture that uses as baseline an arch-specific implementation,
> which might use non optimal strategy that a future generic implementation might
> use? We have examples for both string and math code on different architectures
> where the generic implementation ended up performing better than the arch-specific
> implementation.

So having more than 1 baseline is often useful since you can compare different
implementations and strategies.

> My point is using memchr_strlen as the *generic* implementation and also use
> it as the *baseline* for performance comparison shows to the developer that
> optimizing memchr would be a net gain in general than providing multiple
> different optimization for multiple symbols that can be built by memchr
> calls.

Well if you're only adding a target specific implementation for either strlen or
memchr then it makes sense to do memchr first. And yes we could make the 
generic strlen defer to memchr as that makes it easier to get good performance
on a new target using just a fast memchr.

> So I still think we should define better which exactly we need to compare
> in benchtests and use the generic implementation, which will be used as
> default for new ports, as the default basline. The file inclusion is just to 
> avoid code duplication, I don't have a strong opinion whether to include or 
> just copy-paste the code on benchtests. 

It's hard to come up with simple rules - what is best depends on the specific
function. Memcpy/memchr/strlen have no obvious "baseline", and given
most targets implement these in assembler (which should beat the generic
version), comparing against the generic implementations isn't really that useful.

For more complex string functions we've seen cases where an assembler
strcpy/strcat was slower than strlen/memcpy, so it makes sense to
benchmark against that as the baseline. The same is true for strlen, memchr
and strnlen as well as strchr and strlen/memchr.

My goal is to add the most obvious and useful comparisons to the benchtests
to make it much easier to find these unexpected performance flaws across
targets and microarchitectures.

Cheers,
Wilco