undefined references since newlib-3.2.0

[Dimitrios Glynos]

Hello,

I'm responsible for the CVE's that triggered these patches
(see [1]) and would just like to mention on-list that
during the initial (private) discussion of these issues with both
Jeff Johnston of newlib and Keith Packard of newlib-nano/picolibc
I had provided the following recommendation:

a) when the standard allows so, have the API return the NULL pointer
   to the caller with ENOMEM, so that the caller knows there's an
   out of memory (OOM) issue.

b) when the standard has no provision for ENOMEM, it would be
   good practice to introduce a function that would be called under
   OOM conditions.

OOM conditions are special, especially for embedded and
bare metal projects, as:
i) a project might not be able to simply call abort(), i.e. other
   important actions must be performed (e.g. communicate the error
   to the user or over some interface)
ii) a project might want to take special actions so that it survives
   this condition (e.g. relinquish some resources, unwind
   to some other state)

For these reasons, I had proposed for case (b) the introduction
of a function that would be implemented by the developer, and
would handle non-reported out-of-memory conditions. Developers
already supply implementations for system calls on bare bones devices,
and this could be a similar concept to that. There are two ways
of implementing this, one at compile-time (e.g. as you define
sbrk today) and another at runtime (similar to an atexit()
registration). However I feel that the compile-time introduction
of the function elaborates more clearly the message "that this
is something that you must take care of when using this library".

I don't recommend masquerading ENOMEM as some other error, as many
developers may miss that this is a point in the
code they should be covering points (i) and/or (ii).

CVE-wise newlib with the right compile-time configuration covers
the specific issues presented in the advisory, albeit with the
caveats you have all witnessed. It is wise to also inspect if
the callers of the reported functions handle NULL returns correctly
as in picolibc there were a number of callers that required a patch.

Finally, it would be beneficial for all if both projects
(picolibc and newlib) filed a comment to the standards group
stating the cases where an ENOMEM was found useful but was not
covered by the standard.

Hope this helps.

Best regards,

Dimitris

[1]
https://census-labs.com/news/2020/01/31/multiple-null-pointer-dereference-vulnerabilities-in-newlib/

On 14/6/20 8:02 μ.μ., Keith Packard via Newlib wrote:
> Josef Wolf <jw@raven.inka.de> writes:
> 
>>>         atof
>>>         atoff
>>> [ ... ]
>>>         strtod
>>>         strtof
>>>         strtold
>>>         wcstod
>>>         wcstold
>>>         strtodg
>>
>> Uh! Why on earth would those functions need to allocate memory?
> 
> Because they are performing string to float conversions using code
> written in 1991 by David Gay, based on research done by Will Clinger
> which shows that exact conversion from arbitrary strings of decimal
> digits to fixed precision binary requires arbitrary precision
> arithmetic.
> 
>         https://dl.acm.org/doi/10.1145/93548.93557
> 
>>> These now return infinity and set errno to ERANGE on allocation
>>> failure. (not ideal, but the options are limited)
>>>
>>> Here are some which do return a pointer, but do not document any errors:
>>>
>>>         ecvt
>>>         fcvt
>>
>> Maybe the documentation can be fixed?
> 
> The documentation is based on a standard, and fixing that standard
> involves a bit of process...
> 
>>>         gcvt
>>>         ecvtbuf
>>>         fcvtbuf
>>>         gcvtbuf
>>
>> Those get a pointer passed. No need to allocate memory.
> 
> These functions are using code also written by David Gay to perform
> float to string conversion, based on research done by Guy Steele and Jon
> White in how to print floating point numbers accurately (which happened
> to be presented at the same conference as the work above!). In this
> work, they showed that exact conversion could be done using 1050 bit
> arithmetic to generate a 64-bit double result:
> 
>         https://dl.acm.org/doi/10.1145/93548.93559
> 
> David Gay's code in newlib for both directions uses arbitrary precision
> arithmetic code found in newlib/libc/stdlib/mprec.c. This code allocates
> variable sized arrays of integers on the heap to hold all of the values.
> Before the eBalloc patch, none of these allocations were checked,
> leading to a rather long list of CVEs as the code could end up storing
> through a NULL pointer, which can cause security problems on some
> architectures.
> 
>>> And here's a list of functions which I feel reasonable applications
>>> should not expect an allocation error from:
>>
>> I don't think any application should expect those functions to call exit()
>> and/or abort() either.
> 
> I'm in complete agreement here. It's better to return an error that an
> application *might* check than to not give it any chance to recover at
> all.
> 
>>>         sprintf
>>>         snprintf
>>
>> Those should return -1 on failure.
>>
>>>         sscanf
>>
>> For this, ENOMEM is documented.
> 
> Yes, but as I suggested, applications probably aren't expecting a call
> to sscanf to return EOF and set errno to ENOMEM.
> 
> The real answer to your concerns is to replace the old arbitrary
> precision based float/string conversion code with code that uses results
> from new research by Ulf Adams.
> 
> That research improves on Steel & White by reducing the precision
> required for exact 64-bit float to string conversion from 1050 bits to
> 128 bits. Adams also presents an algorithm using a similar technique to
> perform (a slightly weaker form of) exact string to float conversion in
> the same precision:
> 
>         https://dl.acm.org/citation.cfm?doid=3296979.3192369
> 
> This reasonably small fixed precision can be statically allocated in
> memory, or allocated on the stack. Either of these solutions eliminates
> the use of the dynamic heap through malloc, and eliminates the need to
> change the specification of all of these functions to account for the
> heap usage in the existing newlib code.
> 
> Ulf Adams also published code to implement this algorithm on github:
> 
>         https://github.com/ulfjack/ryu
> 
> I've ported this code to picolibc, a fork of newlib designed for
> embedded systems. That library has an alternate stdio implementation
> that doesn't need to use malloc, and it made sense to add this
> malloc-free float/string conversion code to that (the previous
> float/string conversion code in this implementation was not exact). When
> compiled using that code, picolibc will not return errors from malloc
> failures in the above cases because it does not call malloc in those
> code paths.
> 
> The picolibc source repository also includes the stdio code from newlib
> which can be used in place of the default picolibc stdio code by setting
> a build option. That code has been modified to catch allocation
> failures and return the failures above. I did that in case someone
> wanted to use the original stdio code as I felt even this non-default
> code should not expose applications to arbitrary calls to abort from
> inside the library. I believe this code should be ported back to newlib
> so that at least newlib wouldn't call abort. Even better would be to
> have someone take a look at the Ryu paper and code and make that work in
> newlib.
> 
> (The definition of 'exact' used in Ulf Adams work offers the guarantee
> that you can print any floating point value, and then re-read that
> string to exactly reproduce the original floating point value in
> memory. This is weaker than what Clinger's research used; in that work,
> the goal was to generate the floating point value closest to an
> arbitrary string of decimal digits.)
>