This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: thread heap leak?
On Fri, 22 Mar 2019 14:01:14 -0400
"Carlos O'Donell" <codonell@redhat.com> wrote:
> On 3/22/19 1:53 PM, David Muse wrote:
> > On Fri, 22 Mar 2019 18:31:01 +0100
> > Florian Weimer <fw@deneb.enyo.de> wrote:
> >
> >> * David Muse:
> >>
> >>> Over time, we get more and more of them until top shows the app's VIRT
> >>> to be around 4G (I think, maybe just 2G). Then it crashes. The RES
> >>> is never more than a few MB.
> >>
> >> Do you have backtraces from the crash? How did you determine that the
> >> crashes and the anonymous mappings are related?
> >>
> >> How do you launch the detached threads? Do you use any other thread
> >> attributes?
> >>
> >
> > I've struggled to get backtraces. The app has a crash-handler that prints a backttrace to the log, but that also crashes inside of a malloc. Getting a core on the production system has been a challenge too. I'll see if I can get that.
> >
> > I'm not 100% sure that they are related.
> >
> > I have a monitor running that does a top every 30 seconds or so. The pattern is always that the app's VIRT grows to about 2G and then it crashes. We figured it was a memory leak, so we started running it through valgrind. We'd see the same memory usage, but valgrind would report no leaks, and only a few K of "still reachable" memory. This was the same whether the app crashed, or whether we just killed it after a few hours. I eventually noticed the anonymous segments in /proc/<pid>/maps, did some math on them, and them + [stack] roughly added up to the VIRT size. So, I figured they were responsible for the VIRT growth.
> >
> > It's not clear why the app is crashing at about 2G. I'd think that a 64-bit process ought to be able to address more than that. But, since it always crashes at that size, it seems like it's related.
> >
> > Code to launch detached threads:
> >
> > ... main ...
> >
> > cs->threadattr=new pthread_attr_t;
> > pthread_attr_init(cs->threadattr);
> > pthread_attr_setdetachstate(cs->threadattr,PTHREAD_CREATE_DETACHED);
> > ...
> > cs->threadhandle=new pthread_t;
> > if (pthread_create(cs->threadhandle,cs->threadattr,
> > (void *(*)(void *))clientThread,
> > (void *)cs)) {
> > ... error handling ...
> > }
> >
> >
> > ... inside of clientThread() ...
> >
> > pthread_attr_destroy(cs->threadattr);
> > ...
> > pthread_exit(NULL);
> >
> >
> > No other attributes.
>
> If the detached thread exits the memory should get reaped
> (eventually, the kernel is sometimes slow at reaping).
>
> You really need to instrument the thread stack/size and
> correlate the anonymous mappings to the threads, and figure
> out why you crash.
>
> When a thread exits we use madvise (MADV_DONTNEED) to mark
> the pages of the stack (minus PTHREAD_STACK_MIN) as unused.
> This means VmSIZE remains high, but VmRSS is reduced.
> However, eventually when the stack cache exceeds 40MiB we
> start unmapping the entire caches.
>
> I wonder if your OS has a customization to increase the stack
> cache size? Are you able to debug the value of 'stack_cache_maxsize'
> that you see from nptl/allocatestack.c?
>
> --
> Cheers,
> Carlos.
>
I probably can't check stack_cache_maxsize in production, but I can probably check it on a dev VM. I'd expect it to be the same. I'd think it would have to be really large to cause the problem I'm seeing, but I'll check.
David
david.muse@firstworks.com