[PATCH][RFC] Make malloc routines use mmap if heap is corrupt

[Siddhesh Poyarekar <spoyarek at redhat dot com>]

Hi,

If the heap is found to be corrupt, the default action currently is to
print a backtrace, the memory map and then abort() (configurable with
MALLOC_CHECK_). If the arena lock has been held during this time, this
can result in a deadlock since __backtrace calls routines in the
loader that may result in malloc calls.

With the patch attached, any malloc calls after a detected corruption
will only act on mmap'ed memory locations, thus preventing the malloc
routines from tripping over themselves. Even in a multi-threaded case,
after a heap corruption has been detected in some thread, other
threads will also end up using mmap till an abort() is actually
called. This is not necessarily a bad idea, since it prevents them
from messing up an already messed up heap and leaves a better
possibility of figuring out what went wrong from the resulting core
dump.

I was going to implement this using malloc hooks, but I found in
recent commits that the hooks will be deprecated.

Also inline is a small program that corrupts the heap to demonstrate a
resulting deadlock. Along with verifying for this reproducer (I
couldn't figure out how to make this into a test case for the
testsuite), I have run the test suite on x86_64 and the patch does not
seem to introduce any new failures.

Regards,
Siddhesh

ChangeLog:

2011-09-27  Siddhesh Poyarekar  <siddhesh@redhat.com>

	* malloc/malloc.c: New variable: heap_is_corrupted
	Use mmap for malloc, calloc, etc if it is set to 1.


reproducer:

#include <string.h>
#include <unistd.h>
#include <mcheck.h>
#include <malloc.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/shm.h>
#include <sys/mman.h>

int main(int argc, char *argv[])
{
       char *r=0;
       int i,j,ret;
       unsigned int *p,*q,*s;
       unsigned int *addr;

       p = (unsigned int *)malloc(100);
       memset(p,1,100);
       q = (unsigned int *)malloc(120);
       memset(q,2,100);

       free(q);
       memset(p,5,120);
       q = (unsigned int *)malloc(120);
       free(q);
       memset(q,6,132);

       return 0;
}

Attachment: malloc-heap-corrupt.patch
Description: Text document