This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: thread heap leak?
On Mon, 25 Mar 2019 11:37:58 -0400
David Muse <david.muse@firstworks.com> wrote:
> On Fri, 22 Mar 2019 13:45:42 -0400
> "Carlos O'Donell" <codonell@redhat.com> wrote:
>
> > On 3/22/19 1:39 PM, David Muse wrote:
> > > On Fri, 22 Mar 2019 13:03:51 -0400 "Carlos O'Donell"
> > > <codonell@redhat.com> wrote:
> > >
> > >> On 3/22/19 10:53 AM, David Muse wrote:
> > >>> We have this little server program that listens for client
> > >>> connections on an inet socket. When it receives a connection,
> > >>> it pthread_create()'s a detached thread to handle the request and
> > >>> the main thread waits for more client connections.
> > >>
> > >> The stack for the detached thread is recoverable only if the
> > >> thread exits, at which point it calls __free_tcb () (only if
> > >> detached, because otherwise only pthread_join() does this
> > >> recovery), enqueues the stack onto the free list and leaves it
> > >> there. Subsequent thread creation attemps to reuse the stack, but
> > >> must wait for the kernel to clear the registered tid memory
> > >> (CLONE_CHILD_CLEARTID) to mark the stack reusable (FREE_P). The
> > >> total size of the thread stack cache should only be ~40MiB, and new
> > >> stack creation triggers an automatic check to trim this cache back
> > >> down to 40MiB.
> > >>
> > >> Have you been able to reduce this to a smaller test case?
> > >>
> > >> The anonymous mappings might indeed be thread stacks, but you'd
> > >> have to verify that yourself, and you can do that by asking the
> > >> thread for the stack information and printing that.
> > >>
> > >> Are you using Amazon Linux or a known distro upstream?
> > >
> > > I did comment out lots of features in the program to narrow down the
> > > problem, but I haven't been able to reproduce the leak in a compact
> > > test case. I've been working on that though.
> >
> > Good luck :-)
> >
> > > How do I get the stack information from the thread? I know that I
> > > can do pthread_attr_getstacksize for the size, but what else can I
> > > get?
> >
> > You can use pthread_getattr_np() to take a snapshot of the thread's
> > attributes and look at those, that will get you the information about
> > stack address location and size.
> >
> > You should try using something like systemtap to put tap points on
> > your program and observe the behaviour of the detached threads.
> >
> >
> > > We're using OpenSUSE Leap 42.3. I've seen the problem directly
> > > there. I've gotten reports that it also occurs on RHEL 7, but I
> > > can't confirm that.
> >
> > Andreas,
> >
> > Have you seen reports like this?
> >
> > --
> > Cheers,
> > Carlos.
> >
>
> Thanks for the advice. I'll try pthread_getattr_np() and systemtap and see what info I can get.
>
> Thanks,
>
> David
> david.muse@firstworks.com
Ok, I added some thread-stack logging to my program and collected some data. Here's what I have so far.
It looks like during this run, only 4 threads were forked, and 3 of them reused the same stack:
thread: addr: 0000000089a63000 end: 0000000089c63000 - stack 1
thread: addr: 0000000089e65000 end: 000000008a065000 - stack 2
thread: addr: 0000000089e65000 end: 000000008a065000 - stack 2 (reused)
thread: addr: 0000000089e65000 end: 000000008a065000 - stack 2 (reused)
In /proc/<pid>/maps, I see the 2 stacks:
2aeb89a5e000-2aeb89a62000 rw-p 00000000 00:00 0
2aeb89a62000-2aeb89a63000 ---p 00000000 00:00 0
2aeb89a63000-2aeb89c63000 rw-p 00000000 00:00 0 - stack 1
2aeb89c63000-2aeb89c64000 ---p 00000000 00:00 0
2aeb89c64000-2aeb89e64000 rw-p 00000000 00:00 0
2aeb89e64000-2aeb89e65000 ---p 00000000 00:00 0
2aeb89e65000-2aeb8a065000 rw-p 00000000 00:00 0 - stack 2
2aeb8c000000-2aeb8c097000 rw-p 00000000 00:00 0
2aeb8c097000-2aeb90000000 ---p 00000000 00:00 0
2aeb90000000-2aeb90093000 rw-p 00000000 00:00 0
2aeb90093000-2aeb94000000 ---p 00000000 00:00 0
2aeb94000000-2aeb940a2000 rw-p 00000000 00:00 0
2aeb940a2000-2aeb98000000 ---p 00000000 00:00 0
But what are the rest of those anonymous segments?
This might not be a thread-stack leak, but some other kind of leak. What else creates anonymous segments like that?
David
david.muse@firstworks.com