gitlab.arm.com will be in the maintainance mode on Wednesday June 29th 01:00 - 10:00 (UTC+1). Repositories is read only during the maintainance.

Commit 6e1d3752 authored by Stefan Raspl's avatar Stefan Raspl Committed by Paolo Bonzini
Browse files

tap13: list testcases individually



The current tap13 support summarizes testcases on a rather coarse
granularity. Which sort of defeats the purpose, especially in CI
environments, where having results for individual tests to pinpoint
problems helps a lot.

To improve things, this patch makes TAP mode bypass print_result and extract
results for individual testcases as they are sent to the log file.  It then
introduces test case numbers and writes the plan at the end.

One could argue that post-processing the logs is a rather fragile approach.
Which is true - but apparently already the case, see e.g. extract_summary
in scripts/runtime.bash. Plus this is quite cheap, while a proper solution
would require to modify the kernels, so we can e.g. pass in a counter for
the test case number. And we would probably have to come up with reasonable
test case names, while this approach simply derives them from each test
output.

Before:
  $ ./run_tests.sh -t
  TAP version 13
  ok selftest-setup
  ok intercept
  ok emulator
  ok sieve
  ok sthyi
  ok skey
  ok diag10
  ok pfmf
  ok cmm
  ok vector
  ok gs # SKIP
  ok iep # SKIP
  1..12

After:
  $ ./run_tests.sh -t
  TAP version 13
  ok 1 - cmm: privileged: Program interrupt: expected(2)
  ok 2 - cmm: invalid ORC 8: Program interrupt: expected(6)
  ok 3 - diag10: lowcore freeing: 0x0000/0x0000: Program interrupt: expected(6)
  ok 4 - diag10: lowcore freeing: 0x1000/0x1000: Program interrupt: expected(6)
  ok 5 - diag10: lowcore freeing: 0x0000/0x1000: Program interrupt: expected(6)
  ok 6 - diag10: start/end: end < start: Program interrupt: expected(6)
  ok 7 - diag10: start/end: unaligned start: Program interrupt: expected(6)
  ok 8 - diag10: start/end: unaligned end: Program interrupt: expected(6)
  ok 9 - diag10: privileged: Program interrupt: expected(2)
  ok 10 - emulator: spm/ipm: cc=0,key=f: bit 32 and 33 set to zero
  ok 11 - emulator: spm/ipm: cc=0,key=f: bit 0-31, 40-63 unchanged
  ok 12 - emulator: spm/ipm: cc=0,key=f: cc and key applied
  ok 13 - emulator: spm/ipm: cc=1,key=9: bit 32 and 33 set to zero
  1..180
  [...]
Signed-off-by: default avatarStefan Raspl <raspl@linux.ibm.com>
[Almost rewritten, but still giving a lot of credit to Stefan. - Paolo]
Signed-off-by: default avatarPaolo Bonzini <pbonzini@redhat.com>
parent 231a2e75
......@@ -69,12 +69,50 @@ shift $((OPTIND - 1))
only_tests="$*"
# RUNTIME_log_file will be configured later
RUNTIME_log_stderr () { cat >> $RUNTIME_log_file; }
if [[ $tap_output == "no" ]]; then
process_test_output() { cat >> $RUNTIME_log_file; }
postprocess_suite_output() { cat; }
else
process_test_output() {
CR=$'\r'
while read -r line; do
line="${line%$CR}"
case "${line:0:4}" in
PASS)
echo "ok TEST_NUMBER - ${line#??????}" >&3
;;
FAIL)
echo "not ok TEST_NUMBER - ${line#??????}" >&3
;;
SKIP)
echo "ok TEST_NUMBER - ${line#??????} # skip" >&3
;;
*)
;;
esac
echo "${line}"
done >> $RUNTIME_log_file
}
postprocess_suite_output() {
test_number=0
while read -r line; do
case "${line}" in
ok*|"not ok"*)
(( test_number++ ))
echo "${line/TEST_NUMBER/${test_number}}" ;;
*) echo "${line}" ;;
esac
done
echo "1..$test_number"
}
fi
RUNTIME_log_stderr () { process_test_output; }
RUNTIME_log_stdout () {
if [ "$PRETTY_PRINT_STACKS" = "yes" ]; then
./scripts/pretty_print_stacks.py $1 >> $RUNTIME_log_file
./scripts/pretty_print_stacks.py $1 | process_test_output
else
cat >> $RUNTIME_log_file
process_test_output
fi
}
......@@ -84,7 +122,6 @@ function run_task()
if [ -z "$testname" ]; then
return
fi
test_number=$((test_number+1))
while (( $(jobs | wc -l) == $unittest_run_queues )); do
# wait for any background test to finish
......@@ -110,15 +147,17 @@ mkdir $unittest_log_dir || exit 2
echo "BUILD_HEAD=$(cat build-head)" > $unittest_log_dir/SUMMARY
if [[ $tap_output == "yes" ]]; then
test_number=0
echo "TAP version 13"
fi
trap "wait; exit 130" SIGINT
for_each_unittest $config run_task
(
# preserve stdout so that process_test_output output can write TAP to it
exec 3>&1
test "$tap_output" == "yes" && exec > /dev/null
for_each_unittest $config run_task
) | postprocess_suite_output
# wait until all tasks finish
wait
if [[ $tap_output == "yes" ]]; then
echo "1..$test_number"
fi
......@@ -61,23 +61,10 @@ function print_result()
local summary="$3"
local reason="$4"
if [[ $tap_output == "no" ]]; then
if [ -z "$reason" ]; then
echo "`$status` $testname $summary"
else
echo "`$status` $testname ($reason)"
fi
return
fi
if [[ $status == "FAIL" ]]; then
echo "not ok $testname $reason"
elif [[ $status == "PASS" ]]; then
echo "ok $testname"
elif [[ $status == "SKIP" ]]; then
echo "ok $testname # SKIP $reason"
if [ -z "$reason" ]; then
echo "`$status` $testname $summary"
else
echo "not ok # TODO unknown test status"
echo "`$status` $testname ($reason)"
fi
}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment