malloc, vfork, and OpenJDK

[Florian Weimer <fweimer at redhat dot com>]

I recently looked at what would be needed to eliminate jspawnhelper from
OpenJDK.  It turns out we'd need closefrom support in posix_spawn (bug
10353).

jspawnhelper is a fairly recent addition on OpenJDK on Linux, though.
It turns out that the default vfork-based code for launching processes
calls opendir from the vfork'ed subprocess, thus calling malloc after
vfork.  This is needed for the OpenJDK-internal implementation of
closefrom.

Obviously, calling malloc after vfork is completely undefined, but
OpenJDK binaries have a very long shelf life, so future changes to
malloc will have to take this compatibility constraint into account.

The good news is that I think that our current malloc implementation has
this property.  Even if we expand the heap during malloc (with sbrk or
mmap), the vfork implementation in Linux ensures that the change is also
reflected in the parent process.  The futex-based locks do not use TIDs,
so the arena lock is acquired and released as expected.

The only bug I can see is that destroying the subprocess from Java
(which is supposed to be memory-safe) causes a memory leak in the parent
process (if it happens during closefrom) or a deadlock (if it happens
during malloc or free, while arena lock is held).  But this is a bug
that exists today, and it does not seem to matter much in practice.

And OpenJDK recently switched to posix_spawn + jspawnhelper for its
default process launch method, so that bug is going to go away
eventually.  But Java binaries tend to stick around for a long time.

Thanks,
Florian