Re: [2.20] [1/6] Generate .test-result files for ordinary tests

["Carlos O'Donell" <carlos at redhat dot com>]

On 01/09/2014 09:11 PM, Joseph S. Myers wrote:
> This patch starts the process of generating explicit PASS or FAIL
> status for individual glibc tests.  It's based on Tomas Dohnalek's
> patch <https://sourceware.org/ml/libc-alpha/2012-10/msg00278.html>,
> but is deliberately more minimal: it doesn't try to cover any tests
> outside of $(tests) / $(xtests) (that's done in patch 2), nor does it
> put the result together in an overall summary file (patch 5): it just
> generates the .test-result files.
> 
> Thus, this patch keeps the overall logic for when a testsuite run
> finishes completely unchanged: a test failing will terminate the run.
> I think we *should* move to a more conventional approach where plain
> "make check" does not terminate for an individual test failure, unless
> e.g. you say "make stop-on-test-failure=y check", but that sort of
> policy change is best done as a separate patch (patch 6) once the
> infrastructure is in place to generate summary files for completed
> test runs (which will entirely consist of PASS and XFAIL lines if the
> testsuite run reaches the point of generating them, until such a
> policy change is made).
> 
> Tested x86_64.
> 
> 2014-01-10  Tomas Dohnalek  <tdohnale@redhat.com>
> 	    Joseph Myers  <joseph@codesourcery.com>
> 
> 	* Makeconfig (test-name): New variable.
> 	(evaluate-test): Likewise.
> 	* Makerules (do-test-clean): Remove .test-result files.
> 	(common-mostlyclean): Likewise.
> 	* Rules ($(objpfx)%.out): Use $(evaluate-test) in both rules.
> 	* scripts/evaluate-test.sh: New file.

Looks good to me.

> diff --git a/Makeconfig b/Makeconfig
> index 1908f27..a3aa65a 100644
> --- a/Makeconfig
> +++ b/Makeconfig
> @@ -1080,6 +1080,17 @@ defines += -D_LIBC_REENTRANT
>  libio-mtsafe = -D_IO_MTSAFE_IO
>  endif
>  
> +# The name to give to a test in test results summaries.
> +ifdef subdir
> +test-name = $(subdir)/$(@F:.out=)
> +else
> +test-name = $(@F:.out=)
> +endif

OK.

> +
> +# Command to output a test status line (such as PASS: test-name).
> +evaluate-test = $(..)scripts/evaluate-test.sh $(test-name) $$? \
> +		  > $(objpfx)$(@F:.out=).test-result

OK.

> +
>  endif # Makeconfig not yet included
>  
>  # Local Variables:
> diff --git a/Makerules b/Makerules
> index b7e556f..70a7108 100644
> --- a/Makerules
> +++ b/Makerules
> @@ -1296,7 +1296,10 @@ mostlyclean: common-mostlyclean
>  
>  do-tests-clean:
>  	-rm -f $(addprefix $(objpfx),$(addsuffix .out,$(tests) $(xtests) \
> -						      $(test-srcs)))
> +						      $(test-srcs)) \
> +				     $(addsuffix .test-result,$(tests) \
> +							      $(xtests) \
> +							      $(test-srcs)))
>  
>  # Remove the object files.
>  common-mostlyclean:
> @@ -1306,7 +1309,10 @@ common-mostlyclean:
>  						    $(test-srcs) $(others) \
>  						    $(sysdep-others)) \
>  				     $(addsuffix .out,$(tests) $(xtests) \
> -						      $(test-srcs)))
> +						      $(test-srcs)) \
> +				     $(addsuffix .test-result,$(tests) \
> +							      $(xtests) \
> +							      $(test-srcs)))
>  	-rm -f $(addprefix $(objpfx),$(extra-objs) $(extra-test-objs) \
>  				     $(install-lib) $(install-lib.so) \
>  				     $(install-lib.so:%.so=%_pic.a))
> diff --git a/Rules b/Rules
> index 49ca8ea..00e4a36 100644
> --- a/Rules
> +++ b/Rules
> @@ -183,9 +183,9 @@ make-test-out = $(test-wrapper-env) \
>  		GCONV_PATH=$(common-objpfx)iconvdata LC_ALL=C \
>  		$($*-ENV) $(host-test-program-cmd) $($*-ARGS)
>  $(objpfx)%.out: %.input $(objpfx)%
> -	$(make-test-out) > $@ < $(word 1,$^)
> +	$(make-test-out) > $@ < $(word 1,$^); $(evaluate-test)
>  $(objpfx)%.out: /dev/null $(objpfx)%	# Make it 2nd arg for canned sequence.
> -	$(make-test-out) > $@
> +	$(make-test-out) > $@; $(evaluate-test)
>  
>  endif	# tests
>  
> diff --git a/scripts/evaluate-test.sh b/scripts/evaluate-test.sh
> new file mode 100755
> index 0000000..e118a45
> --- /dev/null
> +++ b/scripts/evaluate-test.sh
> @@ -0,0 +1,32 @@
> +#! /bin/sh
> +# Output a test status line.
> +# Copyright (C) 2012-2014 Free Software Foundation, Inc.
> +# This file is part of the GNU C Library.
> +
> +# The GNU C Library is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU Lesser General Public
> +# License as published by the Free Software Foundation; either
> +# version 2.1 of the License, or (at your option) any later version.
> +
> +# The GNU C Library is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +# Lesser General Public License for more details.
> +
> +# You should have received a copy of the GNU Lesser General Public
> +# License along with the GNU C Library; if not, see
> +# <http://www.gnu.org/licenses/>.
> +
> +# usage: evaluate-test.sh test_name rc
> +
> +test_name=$1
> +rc=$2
> +
> +if [ $rc -eq 0 ]; then
> +  result="PASS"
> +else
> +  result="FAIL"
> +fi
> +
> +echo "$result: $test_name"
> +exit $rc
> 

OK.

Cheers,
Carlos.