Re: thread heap leak?

[David Muse <david dot muse at firstworks dot com>]

On Tue, 30 Apr 2019 19:55:54 -0400
"Carlos O'Donell" <codonell@redhat.com> wrote:

> On 4/30/19 5:48 PM, DJ Delorie wrote:
> > David Muse <david.muse@firstworks.com> writes:
> >> Ahh, ok.  I have a couple of questions then...
> > 
> > Some of the answers are in the wiki:
> > https://sourceware.org/glibc/wiki/MallocInternals
> > 
> >> Is the 64MB segment size a result of my app doing 64MB of allocations,
> >> or is that a default heap size?  Is that tuneable?
> 
> The first allocation of a new heap (attached to a given arena) creates
> the virtual-address reservation of a size that matches the default
> heap size, which is 64MiB for a 64-bit process.
> 
> This does not consume any memory though, it's just VA. RSS is used only
> when you start dirtying pages.
> 
> > Tunables are documented in the manual, or in manual/tunables.texi in the
> > source tree.  No tunables control the heap sizes, which are defined by
> > the two constants HEAP_MIN_SIZE (32kb) and HEAP_MAX_SIZE (1Mb for 32-bit
> > or 64Mb for 64-bit, IIRC)
> 
> And even if it was a tunable it would only change the increment of VA
> reservation, and not the RSS used by the application e.g. you would
> add X-sized heaps to the arena instead of Y-sized heaps.
>   
> >> Is there a way to inspect the base address of a thread's heap from
> >> within the app?
> > 
> > As per the wiki, the base address of the heap is just the pointer,
> > masked to the right number of bits.  All heaps begin at a power of two.
> 
> Also, there isn't anything to inspect, the heap is an internal detail
> of the implementation. You still haven't found the root cause of the
> failure you're seeing.
> 
> >> If a thread exits, is its heap marked as available?  If so, is this
> >> visible to the app?
> > 
> > I don't think any heap-specific information is available to the app.
> 
> Correct. The heaps are an implementation detail. We really really should
> have a heap dumper, a visualizer for the dump, and analysis tooling to
> allow developers to see how the heap is interacting with their allocation
> patterns... but we don't (yet).
> 
> >> At what point do new threads recycle old heaps?  Eg. Does a newly
> >> created thread just grab a heap if an unused one is available, or is
> >> there a minimum number of old heaps that have to pile up before they
> >> will be recycled?  If there's a minimum number, is that tuneable?
> > 
> > First off, don't confuse heaps with arenas.  There are a fixed max
> > number of arenas, but each arena can have more than one heap as more
> > memory is required.  Heaps may be returned to the kernel if all their
> > memory is free'd, arenas are put on a free list when they're not used.
> > New threads prefer to attach to an arena on the free list, else a new
> > one is created if the max isn't reached, else arenas are shared.
> 
> Yeah, there is a terminology issue here, most developers think of their
> application as having 2 things: a stack, and a heap. This isn't quite
> the truth in a modern OS and is perhaps an oversimplification.
> 
> >> In a 64-bit app (on suse linux running on AWS (Xen)), should a
> >> malloc() crash when the address space grows to 2G?
> > 
> > No.
> 
> Agreed. You need to do a root cause analysis of this failure, perhaps
> involving SUSE. However, Andreas (SUSE) commented upstream in this
> issue thathe hasn't seen anything like this.
>   
> > (technicality: malloc() should never *crash* anyway, but it may return
> > NULL and cause your application to crash ;)
> 
> David, I appreciate that you keep coming back to the list to discuss
> this issue, and I find it very interesting to watch your progress. I hope
> you get to the root cause!
> 
> -- 
> Cheers,
> Carlos.
> 

Thanks for all of the help and info guys.

I never got to the bottom of this problem, but it mysteriously just stopped happening, with no changes to the app, no system updates, and no changes to user behavior.  It was reliably happening 4 or 5 times a day, then just stopped happening, and hasn't happened in weeks.

The only thing I can imagine is that the VM host (Xen at AWS) was somehow causing 64-bit apps to crash, in the guest, when their VIRT hit 2Gb, and that was fixed behind-the-scenes.

I may have more questions if the problem crops up again, but for now, all is well.

Thanks again,

David Muse
david.muse@firstworks.com