Commit 5a28195f authored by Vincent Stehlé's avatar Vincent Stehlé
Browse files

Merge branch 'vincent/for-merge' into 'main'

Changes to parser, config and doc

See merge request !5
parents 27671453 d3f4d84c
...@@ -1214,6 +1214,42 @@ ...@@ -1214,6 +1214,42 @@
update: update:
result: IGNORED result: IGNORED
- rule: Force false-positive simple network protocol failure as ignored.
A full implementation of the UEFI general purpose networking ABIs is not
required (EBBR Table 2.4 Notable Deviations from UEFI § 2.6.2)
criteria:
descr: Simple Network Protocol Test
group: NetworkSupportTest
guid: 6880BD92-7004-41B8-9E43-7B271FD9AC2B
log: Status - Unsupported
name: EFI_SIMPLE_NETWORK_PROTOCOL.MCastIpToMac - Invoke MCastIpToMac() when
network interface not start.
result: FAILURE
revision: '0x00010001'
set guid: D924B177-7E44-445D-92AA-5E15C0FD2E27
sub set: MCastIpToMac_Conf
test set: SimpleNetworkProtocolTest
update:
result: IGNORED
- rule: Force false-positive simple network protocol failure (2) as ignored.
A full implementation of the UEFI general purpose networking ABIs is not
required (EBBR Table 2.4 Notable Deviations from UEFI § 2.6.2)
criteria:
descr: Simple Network Protocol Test
group: NetworkSupportTest
guid: 544B08C0-1D26-4462-9207-DD7EB754DC9E
log: Status - Unsupported, MAC address - 000000000000
name: EFI_SIMPLE_NETWORK_PROTOCOL.MCastIpToMac - Invoke MCastIpToMac() and
verify interface correctness within test case
result: FAILURE
revision: '0x00010001'
set guid: 3DE1C94C-4CBB-47C4-9407-0C0F20FA66E7
sub set: MCastIpToMac_Func
test set: SimpleNetworkProtocolTest
update:
result: IGNORED
# We force the following dropped or skipped tests result as `IGNORED' because # We force the following dropped or skipped tests result as `IGNORED' because
# they are run only when some platform features are present, which are not # they are run only when some platform features are present, which are not
# mandated by EBBR. # mandated by EBBR.
...@@ -2339,3 +2375,67 @@ ...@@ -2339,3 +2375,67 @@
comments: Check logs for messages such as "No EFI system partition" or comments: Check logs for messages such as "No EFI system partition" or
"Failed to persist EFI variables" and check that system has an EFI System "Failed to persist EFI variables" and check that system has an EFI System
Partition Partition
- rule: Add comments to failure due to ethernet cable unplugged
criteria:
descr: Simple Network Protocol Test
group: NetworkSupportTest
guid: EA3773EA-0E0F-45A3-82A0-64D485A10B52
log: Status - Not Ready
name: EFI_SIMPLE_NETWORK_PROTOCOL.Transmit - Invoke Transmit() with
HeaderSize is nonzero and not equal to MediaHeaderSize.
result: FAILURE
revision: '0x00010001'
set guid: 5DDF5C49-7ACA-4724-9553-614A53EE998A
sub set: Transmit_Conf
test set: SimpleNetworkProtocolTest
update:
comments: Check if ethernet cable is unplugged
- rule: Add comments to failure (2) due to ethernet cable unplugged
criteria:
descr: Simple Network Protocol Test
group: NetworkSupportTest
guid: DE544DE1-178E-4B5F-97D7-19119B1B7B18
log: Status - Not Ready
name: EFI_SIMPLE_NETWORK_PROTOCOL.Transmit - Invoke Transmit() with
BufferSize is less than MediaHeaderSize.
result: FAILURE
revision: '0x00010001'
set guid: 5DDF5C49-7ACA-4724-9553-614A53EE998A
sub set: Transmit_Conf
test set: SimpleNetworkProtocolTest
update:
comments: Check if ethernet cable is unplugged
- rule: Add comments to failure (3) due to ethernet cable unplugged
criteria:
descr: Simple Network Protocol Test
group: NetworkSupportTest
guid: A449842C-F5F8-47E9-987B-4B6141AEBD45
log: Status - Not Ready
name: EFI_SIMPLE_NETWORK_PROTOCOL.Transmit - Invoke Transmit() with
HeaderSize is nonzero and DestAddr is NULL.
result: FAILURE
revision: '0x00010001'
set guid: 5DDF5C49-7ACA-4724-9553-614A53EE998A
sub set: Transmit_Conf
test set: SimpleNetworkProtocolTest
update:
comments: Check if ethernet cable is unplugged
- rule: Add comments to failure (4) due to ethernet cable unplugged
criteria:
descr: Simple Network Protocol Test
group: NetworkSupportTest
guid: 2E3DD087-DD0C-426E-85BA-65E58310B1DE
log: Status - Not Ready
name: EFI_SIMPLE_NETWORK_PROTOCOL.Transmit - Invoke Transmit() with
HeaderSize is nonzero and Protocol is NULL.
result: FAILURE
revision: '0x00010001'
set guid: 5DDF5C49-7ACA-4724-9553-614A53EE998A
sub set: Transmit_Conf
test set: SimpleNetworkProtocolTest
update:
comments: Check if ethernet cable is unplugged
...@@ -9,19 +9,39 @@ It will proceed to generate a Markdown file listing number of failures, passes, ...@@ -9,19 +9,39 @@ It will proceed to generate a Markdown file listing number of failures, passes,
[UEFI SCT]: https://uefi.org/testtools [UEFI SCT]: https://uefi.org/testtools
## Dependencies
You need to install the [PyYAML] module for the configuration file to be loaded
correctly. Depending on your Linux distribution, this might be available as the
`python3-yaml` package.
It is also recommended to install the [packaging] library for smooth version
detection. Depending on your Linux distribution, this might be available as the
`python3-packaging` package.
See [Configuration file].
If you want to generate the pdf version of this documentation or convert
markdown results to HTML, you need to install [pandoc]. See [Usage] and
[Documentation].
[PyYAML]: https://github.com/yaml/pyyaml
[packaging]: https://github.com/pypa/packaging
[pandoc]: https://pandoc.org
## Quick Start ## Quick Start
If you're using this tool to analyze EBBR test results, use the following If you're using this tool to analyze EBBR test results, use the following
command. The parsed report can be found in `result.md`. command. The parsed report can be found in `result.md`.
``` {.sh} ``` {.sh}
$ ./parser.py --config EBBR.yaml \ $ ./parser.py \
</path/to/sct_results/Overall/Summary.ekl> \ </path/to/sct_results/Overall/Summary.ekl> \
contrib/v21.07_0.9_BETA/EBBR.seq contrib/v21.07_0.9_BETA/EBBR.seq
INFO apply_rules: Updated 200 test(s) out of 12206 after applying 124 rule(s) INFO apply_rules: Updated 200 test(s) out of 12206 after applying 124 rule(s)
INFO main: 0 dropped(s), 1 failure(s), 93 ignored(s), 106 known u-boot limitation(s), 12006 pass(s), 0 warning(s) INFO main: 0 dropped(s), 1 failure(s), 93 ignored(s), 106 known u-boot limitation(s), 12006 pass(s), 0 warning(s)
``` ```
(The `EBBR.yaml' configuration file is used to process results by default.)
## Usage ## Usage
Usage to generate a `result.md` is such: Usage to generate a `result.md` is such:
...@@ -38,12 +58,14 @@ $ ./parser.py --md out.md ... ...@@ -38,12 +58,14 @@ $ ./parser.py --md out.md ...
An online help is available with the `-h` option. An online help is available with the `-h` option.
The generated `result md` can be easily converted to HTML using `pandoc` with: The generated `result md` can be easily converted to HTML using [pandoc] with:
``` {.sh} ``` {.sh}
$ pandoc -oresult.html result.md $ pandoc -oresult.html result.md
``` ```
See [Dependencies].
### Custom search ### Custom search
For a custom Key:value search, the next two arguments *MUST be included together.* The program will search and display files that met that constraint, without the crosscheck, and display the names, guid, and key:value to the command line. `python3 parser.py <file.ekl> <file.seq> <search key> <search value>` For a custom Key:value search, the next two arguments *MUST be included together.* The program will search and display files that met that constraint, without the crosscheck, and display the names, guid, and key:value to the command line. `python3 parser.py <file.ekl> <file.seq> <search key> <search value>`
...@@ -165,21 +187,18 @@ $ ./parser.py --input-md 'result.md' ... ...@@ -165,21 +187,18 @@ $ ./parser.py --input-md 'result.md' ...
## Configuration file ## Configuration file
It is possible to use a configuration file with command line option `--config By default, the `EBBR.yaml` configuration file is used to process results. It is
<filename>`. intended to help triaging failures when testing specifically for [EBBR]
This configuration file describes operations to perform on the tests results, compliance. It describes operations to perform on the tests results,
such as marking tests as false positives or waiving failures. such as marking tests as false positives or waiving failures.
It is possible to specify another configuration file with the command line
option `--config <filename>`.
Example command for [EBBR]: You need to install the [PyYAML] module for the configuration file to be loaded
correctly, and installing the [packaging] library is recommended. See
``` {.sh} [Dependencies].
$ ./parser.py --config EBBR.yaml /path/to/Summary.ekl EBBR.seq ...
```
You need to install the [PyYAML] module for this to work.
[EBBR]: https://github.com/ARM-software/ebbr [EBBR]: https://github.com/ARM-software/ebbr
[PyYAML]: https://github.com/yaml/pyyaml
### Configuration file format ### Configuration file format
...@@ -250,8 +269,9 @@ This generated configuration can then be further edited manually. ...@@ -250,8 +269,9 @@ This generated configuration can then be further edited manually.
### EBBR configuration ### EBBR configuration
The `EBBR.yaml` file is a configuration file meant for [EBBR] testing. It can The `EBBR.yaml` file is the configuration file used by default. It is meant for
override the result of some tests with the following ones: [EBBR] testing and can override the result of some tests with the following
ones:
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
Result Description Result Description
...@@ -265,13 +285,37 @@ override the result of some tests with the following ones: ...@@ -265,13 +285,37 @@ override the result of some tests with the following ones:
filesystem implementation limitations and they do filesystem implementation limitations and they do
not prevent an OS to boot. not prevent an OS to boot.
`KNOWN ACS LIMITATION` Genuine bugs, which are fixed in a more recent version `KNOWN ACS LIMITATION` Genuine bugs, which are fixed in a more recent
of the ACS or which must ultimately be fixed and which version of the ACS or which must ultimately be fixed
we know about. and which we know about.
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
Some of the rules just add a `comments` field with some help text. Some of the rules just add a `comments` field with some help text.
Example command to see those comments:
``` {.sh}
$ ./parser.py \
--filter "x['result'] == 'FAILURE'" \
--fields 'count,result,name,comments' --uniq --print ...
```
### Database of sequence files
The `seq.db` file contains a list of known sequence files, which allows to
identify the input sequence file.
This database file contains lines describing each known sequence file in turn,
in the following format:
```
sha256 description
```
Everything appearing after a '#' sign is treated as a comment and ignored.
The database filename can be specified with the `--seq-db` option.
## Notes ## Notes
### Known Issues: ### Known Issues:
* "comment" is currently not implemented, as formatting is not currently consistent, should reflect the comments from the test. * "comment" is currently not implemented, as formatting is not currently consistent, should reflect the comments from the test.
...@@ -281,8 +325,8 @@ Some of the rules just add a `comments` field with some help text. ...@@ -281,8 +325,8 @@ Some of the rules just add a `comments` field with some help text.
### Documentation ### Documentation
It is possible to convert this `README.md` into `README.pdf` with pandoc using It is possible to convert this `README.md` into `README.pdf` with [pandoc] using
`make doc`. See `make help`. `make doc`. See `make help` and [Dependencies].
### Sanity checks ### Sanity checks
...@@ -384,7 +428,10 @@ convenience: ...@@ -384,7 +428,10 @@ convenience:
`v21.05_0.8_BETA-0/` EBBR sequence file from [ACS-IR v21.05_0.8_BETA-0]. `v21.05_0.8_BETA-0/` EBBR sequence file from [ACS-IR v21.05_0.8_BETA-0].
`v21.07_0.9_BETA/` EBBR sequence files from [ACS-IR v21.07_0.9_BETA]. `v21.07_0.9_BETA/` EBBR sequence files from [ACS-IR v21.07_0.9_BETA].
`v21.09_1.0` EBBR sequence files from [ACS-IR v21.09_1.0].
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
[ACS-IR v21.05_0.8_BETA-0]: https://github.com/ARM-software/arm-systemready/tree/main/IR/prebuilt_images/v21.05_0.8_BETA-0 [ACS-IR v21.05_0.8_BETA-0]: https://github.com/ARM-software/arm-systemready/tree/main/IR/prebuilt_images/v21.05_0.8_BETA-0
[ACS-IR v21.07_0.9_BETA]: https://github.com/ARM-software/arm-systemready/tree/main/IR/prebuilt_images/v21.07_0.9_BETA [ACS-IR v21.07_0.9_BETA]: https://github.com/ARM-software/arm-systemready/tree/main/IR/prebuilt_images/v21.07_0.9_BETA
[ACS-IR v21.09_1.0]: https://github.com/ARM-software/arm-systemready/tree/main/IR/prebuilt_images/v21.09_1.0
B[Test Case] B[Test Case]
B[Test Case] B[Test Case]
...@@ -8,16 +8,20 @@ import csv ...@@ -8,16 +8,20 @@ import csv
import logging import logging
import json import json
import re import re
import hashlib
import os
try: try:
from packaging import version from packaging import version
except ImportError: except ImportError:
print('No packaging...') print('No packaging. You should install python3-packaging...')
try: try:
import yaml import yaml
except ImportError: except ImportError:
print('No yaml...') print(
'No yaml. You should install PyYAML/python3-yaml for configuration'
' file support...')
if 'yaml' in sys.modules: if 'yaml' in sys.modules:
try: try:
...@@ -504,26 +508,29 @@ def gen_template(cross_check, filename): ...@@ -504,26 +508,29 @@ def gen_template(cross_check, filename):
def do_print(cross_check, fields): def do_print(cross_check, fields):
logging.debug(f'Print (fields: {fields})') logging.debug(f'Print (fields: {fields})')
# First pass to find the width for each field except the last one # First pass to find the width for each field
fm1 = fields[:len(fields) - 1]
w = {} w = {}
for f in fm1: for f in fields:
w[f] = len(f) w[f] = len(f)
for x in cross_check: for x in cross_check:
for f in fm1: for f in fields:
w[f] = max(w[f], len(str(x[f]) if f in x else '')) w[f] = max(w[f], len(str(x[f]) if f in x else ''))
# Second pass where we print # Second pass where we print
fm1 = fields[:len(fields) - 1]
lf = fields[len(fields) - 1] lf = fields[len(fields) - 1]
sep = ' '
print(' '.join([ print(sep.join([
*map(lambda f: f"{f:{w[f]}}", fm1), *map(lambda f: f"{f.capitalize():{w[f]}}", fm1),
lf])) lf.capitalize()]))
print(sep.join([*map(lambda f: '-' * w[f], fields)]))
for x in cross_check: for x in cross_check:
print(' '.join([ print(sep.join([
*map(lambda f: f"{x[f] if f in x else '':{w[f]}}", fm1), *map(lambda f: f"{x[f] if f in x else '':{w[f]}}", fm1),
x[lf] if lf in x else ''])) x[lf] if lf in x else '']))
...@@ -597,6 +604,50 @@ def combine_dbs(db1, db2): ...@@ -597,6 +604,50 @@ def combine_dbs(db1, db2):
return cross_check return cross_check
# Load the database of known sequence files.
def load_known_seq(seq_db):
known_seqs = {}
with open(seq_db, 'r') as f:
for line in f:
line = line.rstrip()
line = re.sub(r'#.*', '', line)
m = re.match(r'\s*([0-9a-fA-F]+)\s+(.*)', line)
if not m:
continue
kh = m.group(1)
d = m.group(2)
assert(kh not in known_seqs)
logging.debug(f'{kh} {d}')
known_seqs[kh] = d
logging.debug(f'{len(known_seqs)} known seq file(s)')
return known_seqs
# Try to identify the .seq file in a list of known versions using its sha256.
def ident_seq(seq_file, seq_db):
known_seqs = load_known_seq(seq_db)
# Hash seq file
hm = 'sha256'
hl = hashlib.new(hm)
with open(seq_file, 'rb') as f:
hl.update(f.read())
h = hl.hexdigest()
logging.debug(f'{hm} {h} {seq_file}')
# Try to identify the seq file
if h in known_seqs:
logging.info(f"""Identified `{seq_file}' as "{known_seqs[h]}".""")
else:
logging.debug(f"Could not identify `{seq_file}'...")
# Read the .ekl log file and the .seq file and combine them into a single # Read the .ekl log file and the .seq file and combine them into a single
# database, which we return. # database, which we return.
def read_log_and_seq(log_file, seq_file): def read_log_and_seq(log_file, seq_file):
...@@ -714,6 +765,8 @@ def read_md(input_md): ...@@ -714,6 +765,8 @@ def read_md(input_md):
if __name__ == '__main__': if __name__ == '__main__':
me = os.path.realpath(__file__)
here = os.path.dirname(me)
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(
description='Process SCT results.' description='Process SCT results.'
' This program takes the SCT summary and sequence files,' ' This program takes the SCT summary and sequence files,'
...@@ -740,6 +793,9 @@ if __name__ == '__main__': ...@@ -740,6 +793,9 @@ if __name__ == '__main__':
parser.add_argument( parser.add_argument(
'--print', action='store_true', help='Print results to stdout') '--print', action='store_true', help='Print results to stdout')
parser.add_argument('--input-md', help='Input .md filename') parser.add_argument('--input-md', help='Input .md filename')
parser.add_argument(
'--seq-db', help='Known sequence files database filename',
default=f'{here}/seq.db')
parser.add_argument('log_file', help='Input .ekl filename') parser.add_argument('log_file', help='Input .ekl filename')
parser.add_argument('seq_file', help='Input .seq filename') parser.add_argument('seq_file', help='Input .seq filename')
parser.add_argument('find_key', nargs='?', help='Search key') parser.add_argument('find_key', nargs='?', help='Search key')
...@@ -749,7 +805,8 @@ if __name__ == '__main__': ...@@ -749,7 +805,8 @@ if __name__ == '__main__':
# could actually import yaml. # could actually import yaml.
if 'yaml' in sys.modules: if 'yaml' in sys.modules:
parser.add_argument( parser.add_argument(
'--config', help='Input .yaml configuration filename') '--config', help='Input .yaml configuration filename',
default=f'{here}/EBBR.yaml')
parser.add_argument('--yaml', help='Output .yaml filename') parser.add_argument('--yaml', help='Output .yaml filename')
parser.add_argument( parser.add_argument(
'--template', help='Output .yaml config template filename') '--template', help='Output .yaml config template filename')
...@@ -765,6 +822,10 @@ if __name__ == '__main__': ...@@ -765,6 +822,10 @@ if __name__ == '__main__':
else: else:
# Command line argument 1 is the ekl file to open. # Command line argument 1 is the ekl file to open.
# Command line argument 2 is the seq file to open. # Command line argument 2 is the seq file to open.
# Try to identify the sequence file
ident_seq(args.seq_file, args.seq_db)
# Read both and combine them into a single cross_check database. # Read both and combine them into a single cross_check database.
cross_check = read_log_and_seq(args.log_file, args.seq_file) cross_check = read_log_and_seq(args.log_file, args.seq_file)
......
# List of known sequence files and their sha256
6a381192057c511b2b69282c58d6107c1daeaf0b95038605d4c58383eb5cc88b Test sample.seq
6ae5a92f0b83b3a1469c89f6a02b0771ae3818a424e7118a28bba053be394078 Supplied SBBR.seq
6b83dbfbd1f07fc61a918297f02f449591a72131b64ac746f969a4210f97aee8 ACS-IR v21.05_0.8_BETA-0 EBBR.seq
c06684b3f8b35871e37b9447f609f9aab6070a7ca1c4ba63a52e029c018c9b73 ACS-IR v21.07_0.9_BETA EBBR.seq
d66485b5e436409ef8c0667baf5250e784cbf292f2b9ef1b3893d474a0585fae ACS-IR v21.09_1.0 EBBR.seq
ec730fd81eada415278b39533fe7cc21147b39183447dc11fa77ad4419d13969 Supplied EBBR.seq
f7793d53c10106c1c275a4992e1710ce9863e210dd07581a3d783c4f4cf2312b ACS-IR v21.07_0.9_BETA or v21.09_1.0 EBBR_manual.seq
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment