This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
[Bug testsuite/10791] parallelize systemtap testsuite
- From: "mcermak at redhat dot com" <sourceware-bugzilla at sourceware dot org>
- To: systemtap at sourceware dot org
- Date: Thu, 25 Feb 2016 10:03:39 +0000
- Subject: [Bug testsuite/10791] parallelize systemtap testsuite
- Auto-submitted: auto-generated
- References: <bug-10791-6586 at http dot sourceware dot org/bugzilla/>
https://sourceware.org/bugzilla/show_bug.cgi?id=10791
Martin Cermak <mcermak at redhat dot com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |mcermak at redhat dot com
Assignee|systemtap at sourceware dot org |mcermak at redhat dot com
--- Comment #1 from Martin Cermak <mcermak at redhat dot com> ---
My recent stats for systemtap's make installcheck are as follows:
------------------------------------------------------------------------------
rhel6 i386 MAX: 11:40:36 MEAN: 3:45:04 MEDIAN: 3:21:01
rhel6 x86_64 MAX: 6:44:16 MEAN: 3:19:03 MEDIAN: 3:00:46
rhel6 ppc64 MAX: 16:20:03 MEAN: 5:42:08 MEDIAN: 4:02:35
rhel6 s390x MAX: 3:37:31 MEAN: 2:10:49 MEDIAN: 2:09:37
rhel7 x86_64 MAX: 9:37:39 MEAN: 4:48:23 MEDIAN: 4:15:52
rhel7 ppc64 MAX: 25:05:10 MEAN: 6:10:03 MEDIAN: 4:23:52
rhel7 ppc64le MAX: 4:53:26 MEAN: 3:23:12 MEDIAN: 3:36:14
rhel7 s390x MAX: 4:09:53 MEAN: 2:14:52 MEDIAN: 2:24:33
rhel7 aarch64 MAX: 8:31:14 MEAN: 5:39:20 MEDIAN: 6:17:33
Shortening the run time (and not breaking the test results due to possible
races) would be beneficial. First I tried something as simple as this:
>
> =======
> # cat paralleltest.sh
> #!/bin/bash
>
> for dir in `find . -type d -name 'systemtap.*'`
> do
> make installcheck RUNTESTFLAGS="--outdir=$dir $dir/*exp >& $dir/output.log" &
> done
>
> wait
>
> ./dg-extract-results.sh systemtap.*/systemtap.sum > systemtap.sum
> ./dg-extract-results.sh -L systemtap.*/systemtap.log > systemtap.log
> #
> =======
>
> Where dg-extract-results.sh is taken from https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob;f=gdb/testsuite/dg-extract-results.sh
That works. The GDB testsuite testlog merger can seemingly be used as-is for
the systemtap testsuite. But the GDB testsuite's Makefile trick is nice! The
(slightly outdated) rhel7 gdb testsuite seems to group tests by directory
(similarly to the above), whereas the current upstream version tends to run
each testcase separately (with some exceptions for long-running tests).
Related thoughts in: commit 81208388898d0d561dc0c167b8bb7b2d77e40381, Author:
Tom Tromey. Using the GDB upstream approach in the systemtap testsuite's
Makefile (under the installcheck target):
=======
blah: site.exp
-rm -rf cache outputs temp
$(MAKE) -k do-check-parallel; \
result=$$?; \
$(SHELL) $(srcdir)/dg-extract-results.sh \
`find outputs -name systemtap.sum -print` > systemtap.sum; \
$(SHELL) $(srcdir)/dg-extract-results.sh -L \
`find outputs -name systemtap.log -print` > systemtap.log; \
sed -n '/=== systemtap Summary ===/,$$ p' systemtap.sum; \
exit $$result
all_tests := $(shell cd $(srcdir) && find systemtap.* -name '*.exp' -print)
TEST_TARGETS := $(addprefix check/,$(all_tests))
do-check-parallel: $(TEST_TARGETS)
@:
check/%.exp:
-mkdir -p outputs/$*
-$(MAKE) $(AM_MAKEFLAGS) check-DEJAGNU
RUNTESTFLAGS="--outdir=outputs/$* $*.exp $(RUNTESTFLAGS) --tool_opts \'install
$(TOOL_OPTS)\'"
=======
Using some random (probably not a representative sample) subset of testcases I
see:
make installcheck: 1m30
time make -j 1 blah: 2m32
time make -j 2 blah: 1m22
time make -j 3 blah: 1m2
time make -j 4 blah: 1m1
time make -j 5 blah: 1m3
time make -j 6 blah: 1m5
time make -j 7 blah: 0m44, 1m9, 1m10
time make -j 8 blah: 1m12
There is a significant overhead in lib/systemtap.exp:
echo "pass empty" > systemtap.printf/empty.exp
time make installcheck RUNTESTFLAGS=empty.exp
real 0m3.630s
After removing most of the flesh from it I'm (still) getting:
time make installcheck RUNTESTFLAGS=empty.exp
real 0m0.923s
From the GDB testsuite/README:
Parallel testing
****************
If not testing with a remote host (in DejaGnu's sense), you can run
the GDB test suite in a fully parallel mode. In this mode, each .exp
file runs separately and maybe simultaneously. The test suite ensures
that all the temporary files created by the test suite do not clash,
by putting them into separate directories. This mode is primarily
intended for use by the Makefile.
For GNU make, the Makefile tries to run the tests in parallel mode if
any -j option is given. For a non-GNU make, tests are not
parallelized.
If RUNTESTFLAGS is not empty, then by default the tests are
serialized. This can be overridden by either using the
`check-parallel' target in the Makefile, or by setting FORCE_PARALLEL
to any non-empty value:
make check-parallel RUNTESTFLAGS="--target_board=native-gdbserver"
make check RUNTESTFLAGS="--target_board=native-gdbserver"
FORCE_PARALLEL=1
If you want to use runtest directly instead of using the Makefile, see
the description of GDB_PARALLEL below.
--
You are receiving this mail because:
You are the assignee for the bug.