On Fri, 22 Mar 2019 18:31:01 +0100
Florian Weimer <fw@deneb.enyo.de> wrote:
* David Muse:
Over time, we get more and more of them until top shows the app's VIRT
to be around 4G (I think, maybe just 2G). Then it crashes. The RES
is never more than a few MB.
Do you have backtraces from the crash? How did you determine that the
crashes and the anonymous mappings are related?
How do you launch the detached threads? Do you use any other thread
attributes?
I've struggled to get backtraces. The app has a crash-handler that prints a backttrace to the log, but that also crashes inside of a malloc. Getting a core on the production system has been a challenge too. I'll see if I can get that.
I'm not 100% sure that they are related.
I have a monitor running that does a top every 30 seconds or so. The pattern is always that the app's VIRT grows to about 2G and then it crashes. We figured it was a memory leak, so we started running it through valgrind. We'd see the same memory usage, but valgrind would report no leaks, and only a few K of "still reachable" memory. This was the same whether the app crashed, or whether we just killed it after a few hours. I eventually noticed the anonymous segments in /proc/<pid>/maps, did some math on them, and them + [stack] roughly added up to the VIRT size. So, I figured they were responsible for the VIRT growth.
It's not clear why the app is crashing at about 2G. I'd think that a 64-bit process ought to be able to address more than that. But, since it always crashes at that size, it seems like it's related.
Code to launch detached threads:
... main ...
cs->threadattr=new pthread_attr_t;
pthread_attr_init(cs->threadattr);
pthread_attr_setdetachstate(cs->threadattr,PTHREAD_CREATE_DETACHED);
...
cs->threadhandle=new pthread_t;
if (pthread_create(cs->threadhandle,cs->threadattr,
(void *(*)(void *))clientThread,
(void *)cs)) {
... error handling ...
}
... inside of clientThread() ...
pthread_attr_destroy(cs->threadattr);
...
pthread_exit(NULL);
No other attributes.