Skip to content
  • Stefan Raspl's avatar
    tap13: list testcases individually · 6e1d3752
    Stefan Raspl authored
    
    
    The current tap13 support summarizes testcases on a rather coarse
    granularity. Which sort of defeats the purpose, especially in CI
    environments, where having results for individual tests to pinpoint
    problems helps a lot.
    
    To improve things, this patch makes TAP mode bypass print_result and extract
    results for individual testcases as they are sent to the log file.  It then
    introduces test case numbers and writes the plan at the end.
    
    One could argue that post-processing the logs is a rather fragile approach.
    Which is true - but apparently already the case, see e.g. extract_summary
    in scripts/runtime.bash. Plus this is quite cheap, while a proper solution
    would require to modify the kernels, so we can e.g. pass in a counter for
    the test case number. And we would probably have to come up with reasonable
    test case names, while this approach simply derives them from each test
    output.
    
    Before:
      $ ./run_tests.sh -t
      TAP version 13
      ok selftest-setup
      ok intercept
      ok emulator
      ok sieve
      ok sthyi
      ok skey
      ok diag10
      ok pfmf
      ok cmm
      ok vector
      ok gs # SKIP
      ok iep # SKIP
      1..12
    
    After:
      $ ./run_tests.sh -t
      TAP version 13
      ok 1 - cmm: privileged: Program interrupt: expected(2)
      ok 2 - cmm: invalid ORC 8: Program interrupt: expected(6)
      ok 3 - diag10: lowcore freeing: 0x0000/0x0000: Program interrupt: expected(6)
      ok 4 - diag10: lowcore freeing: 0x1000/0x1000: Program interrupt: expected(6)
      ok 5 - diag10: lowcore freeing: 0x0000/0x1000: Program interrupt: expected(6)
      ok 6 - diag10: start/end: end < start: Program interrupt: expected(6)
      ok 7 - diag10: start/end: unaligned start: Program interrupt: expected(6)
      ok 8 - diag10: start/end: unaligned end: Program interrupt: expected(6)
      ok 9 - diag10: privileged: Program interrupt: expected(2)
      ok 10 - emulator: spm/ipm: cc=0,key=f: bit 32 and 33 set to zero
      ok 11 - emulator: spm/ipm: cc=0,key=f: bit 0-31, 40-63 unchanged
      ok 12 - emulator: spm/ipm: cc=0,key=f: cc and key applied
      ok 13 - emulator: spm/ipm: cc=1,key=9: bit 32 and 33 set to zero
      1..180
      [...]
    
    Signed-off-by: default avatarStefan Raspl <raspl@linux.ibm.com>
    [Almost rewritten, but still giving a lot of credit to Stefan. - Paolo]
    Signed-off-by: default avatarPaolo Bonzini <pbonzini@redhat.com>
    6e1d3752