Script to identify contended mutex locks with a stack trace.

[Alan Conway <aconway at redhat dot com>]

I'm trying to write a systemtap script that will identify contended mutex locks 
and print a user stack trace to identify them. Two unsuccessful attempts are 
attached, I'd appreciate any tips on how to make them work.

futexes.stp is a modified version of the futexes example provided with stap. I 
added a pthread_mutex_lock.return trace point and I attempt to print the user 
stack trace there. However the stack trace is just a hex address. Also for my 
contention_test.c (attached) test it claims that there are 9 locks contended 1 
times each rather than 1 lock contended 9 times.

mutexes.stp sets probes on pthread_muted_lock and .return. This almost works but 
the stack trace for contention_test.c doesn't show functions main and foo as I 
expect, I get:
 0x0000003073408ca0 : __pthread_mutex_lock+0x0/0x135 [/lib64/libpthread-2.5.so]
 0x000000000040065e : _end+0x15de06/0x1e97a8 
[/home/remote/aconway/install/lib/libqpidbroker.so.2.0.0]

If I run mutexes.stp on anything non-trivial it prints a bunch of very promising 
looking stack traces but then falls over with:

ERROR: Skipped too many probes, check MAXSKIPPED or try again with stap -t for 
more details.
WARNING: Skipped due to global 'thread_blocktime' lock timeout: 101

Any tips on how to make these work would be greatly appreciated!

#include <pthread.h>
#include <stdlib.h>
#include <unistd.h>

pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;

void* sleeper(void* p) {
  pthread_mutex_lock(&lock);	/* Contended */
  usleep(1000);
  pthread_mutex_unlock(&lock);
}

void foo() {
  const int n=10;
  pthread_mutex_lock(&lock);
  pthread_t thread[n];
  int i;
  for (i = 0; i < n; ++i) 
    pthread_create(&thread[i], NULL, sleeper, NULL);
  usleep(5000);
  pthread_mutex_unlock(&lock);
  for (i = 0; i < n; ++i) 
    pthread_join(thread[i], 0);
}

int main(int argc,char** argv) {
  foo();
}

Attachment: futexes.stp
Description: Text document

Attachment: Makefile
Description: Text document

Attachment: mutexes.stp
Description: Text document