- 31 May, 2021 1 commit
-
-
Vincent Stehlé authored
Try to semi-automate the writing of yaml configuration files by generating a template yaml configuration from tests data, which can be further edited manually. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
- 18 May, 2021 1 commit
-
-
Jeffrey Booher-Kaeding authored
A fix for old python3 yaml versions
-
- 11 May, 2021 1 commit
-
-
Vincent Stehlé authored
The yaml module has the FullLoader class since version 5.1 only; handle this gracefully at runtime. This fixes the following error: AttributeError: module 'yaml' has no attribute 'FullLoader' Reported-by:
Zhifei Yang <zhifei.yang@arm.com> Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
- 24 Mar, 2021 1 commit
-
-
Jeffrey Booher-Kaeding authored
Adding the rules processing & fixing ekl parsing.
-
- 18 Mar, 2021 1 commit
-
-
Vincent Stehlé authored
We add a configuration file with rules for testing the EBBR specification. Some tests correspond to cases not mandated by the EBBR specification and we want to ignore them. Some other tests are either already fixed with a recent U-Boot or still present with the sandbox or qemu, and we want to issue waivers for those. Original result Forced result False-positive FAILURE/WARNING/DROPPED IGNORED Waived FAILURE/WARNING/SKIPPED WAIVED The list has been established in part by comparing SCT runs done with the EBBR.seq sequence file, running on arm64 U-Boot sandbox and qemu, from commit ddaa94978583 and version v2021.04-rc2. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
- 15 Mar, 2021 5 commits
-
-
The libyaml implementation is significantly faster than the pure python YAML parser. If it is present, then use CDumper when dumping the results. Signed-off-by:
Grant Likely <grant.likely@arm.com> [Rebased, added conditional.] Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
Now that we are using yaml to load configurations, we might as well dump our test data in yaml, too. This helps writing the configuration rules. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
Add a --config option to Parser.py, to use a configuration file. This configuration file can contain a list of transformations to apply to the test results. This will allow to mark tests as false positives or to waive failures. We describe the YAML configuration file format in details in the README.md. We also add a sample configuration, to demonstrate usage of configuration files with the already provided sample log and sequence files. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
A skip test set is a test set, which was considered but had zero of its tests run. This is not the same as a dropped test set, which we do not find in the log at all. Generate artificial test entries with the result set to SKIPPED to keep track of the former. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
Currently we are dropping the end of the device path in the ekl when it contains `|' characters, as in sample.elk for example. Rework the ekl parsing to fix this. Before: `Acpi(PNP0A08,0)/Pci(0' After: `Acpi(PNP0A08,0)/Pci(0|0)/Pci(0|0)' Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
- 11 Mar, 2021 6 commits
-
-
Jeffrey Booher-Kaeding authored
From Vincent: Lowercase change from JSON output Sorting command line option Internal changes, to prepare for configuration/filtering: detect values & combined databases
-
Vincent Stehlé authored
Currently we read db1 and db2 from the log and sequence files respectively. Then we compute a cross_check database with all the tests, which have run, as well as a would_not_run database with all the tests sets, which have not run. Combine those databases into a unified cross_check database and get rid of would_not_run. We create artificial test entries for the dropped tests sets. While at it, we keep the spurious run tests, if any, and mark them as such. The benefits are that we do not need special handling for dropped tests sets down the pipe anymore, and that our future configuration file filtering rules will also apply to spurious and dropped tests. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
In the future, we will be able to override tests result values with a config file and it will become more convenient to auto-detect the test result values rather than hardcoding. This changes a bit the summary table and group sections order and fields, but is still arguably acceptable. While at it, print a one line summary during execution. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
Add a --sort command line option to Parser.py, to allow sorting the tests data in a configurable manner before producing the outputs. Also, document usage in the README.md and fix parser.py case. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
Add a --json command line option to dump our data in json format. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Mostly an aesthetic change, but having the filename in lowercase is more friendly when running from case sensitive filesystems, like on Linux. Signed-off-by:
Grant Likely <grant.likely@secretlab.ca>
-
- 10 Mar, 2021 2 commits
-
-
Jeffrey Booher-Kaeding authored
Usability upgrades Rework command line using argparse Add logging and debug Add csv output Use iconv to cat logs
-
Vincent Stehlé authored
Currently we `cat' all the discrete .ekl log files into a big, single file directly as binaries. This adds UTF-16 two bytes BOM everywhere in the middle of the resulting cat_summary.ekl. Fix that with iconv: we convert all the log files to UTF-8 on the fly and back to UTF-16 while concatenating. While at it, handle an optional command line argument to specify the folder to search for logs. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
- 03 Mar, 2021 5 commits
-
-
Vincent Stehlé authored
Use logging to output debug messages and add a --debug command line option to change the log level. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
Add the --csv command line option to generate a .csv file. This is handy to manually triage issues in a spreadsheet. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
Use argparse for command line arguments. This gives us the online help and will give use more flexibility to add command line options. We preserve the current behaviour with up to two positional arguments for the ekl and seq input files, and up to two arguments for the key:value search. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
We capture the group iteration, start date and time, revision, description and device path. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
This is easier for argument passing and more flexible for adding fields in the future. We take care to preserve current fields ordering in output result.md. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
- 01 Mar, 2021 13 commits
-
-
Jeffrey Booher-Kaeding authored
Added License
-
Jeffrey Booher-Kaeding authored
-
Jeffrey Booher-Kaeding authored
-
Vincent Stehlé authored
We generate a table for the summary rather than sections, as this is arguably more readable. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
Avoid duplicating the GUID, result and name in the log. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
Currently the parser tries to record the test service as 'group' and the test protocol as 'test set'. However, some tests have only a protocol and no service (e.g. LoadedImageProtocolTest). In this case, the parser will record the protocol as 'group' and the full line dictionary as 'test set'. Fix this case by recording an empty 'group' and the protocol as 'test set', more in line with the generic case. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
Strip .ekl input lines, to avoid capturing "\n" in the log field. Adapt the empty lines skip test accordingly. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
This allows to execute Parser.py directly with: $ ./Parser.py <arguments...> Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
The json module is unused: remove it. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
Specify language of the python bloc. This enables syntax coloring with pandoc. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
Remove spurious whitespaces at end of line in Parser.py and README.md. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
We add a Makefile and a configuration file to automate the generation of README.pdf with pandoc. Also, update the .gitignore accordingly. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
For the moment we ignore result.md, the output of Parser.py. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
- 23 Feb, 2021 1 commit
-
-
Jeffrey Booher-Kaeding authored
Fixes
-
- 19 Feb, 2021 3 commits
-
-
Vincent Stehlé authored
The title is already after an ATX-style #, which makes the following setext-style underline superfluous. Remove the underline to fix the rendering with pandoc. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-
Vincent Stehlé authored
This repairs the run with default sample filenames on Linux. Signed-off-by:
Vincent Stehlé <vincent.stehle@arm.com>
-