Try to semi-automate the writing of yaml configuration files by generating
a template yaml configuration from tests data, which can be further edited
manually.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

A fix for old python3 yaml versions

The yaml module has the FullLoader class since version 5.1 only; handle
this gracefully at runtime.

This fixes the following error:

  AttributeError: module 'yaml' has no attribute 'FullLoader'
Reported-by: Zhifei Yang <zhifei.yang@arm.com>
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Adding the rules processing & fixing ekl parsing.

We add a configuration file with rules for testing the EBBR specification.

Some tests correspond to cases not mandated by the EBBR specification and
we want to ignore them. Some other tests are either already fixed with a
recent U-Boot or still present with the sandbox or qemu, and we want to
issue waivers for those.

                  Original result          Forced result
  False-positive  FAILURE/WARNING/DROPPED  IGNORED
          Waived  FAILURE/WARNING/SKIPPED  WAIVED

The list has been established in part by comparing SCT runs done with the
EBBR.seq sequence file, running on arm64 U-Boot sandbox and qemu, from
commit ddaa94978583 and version v2021.04-rc2.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Grant Likely authored Mar 10, 2021 and Vincent Stehlé committed Mar 15, 2021

The libyaml implementation is significantly faster than the pure python
YAML parser. If it is present, then use CDumper when dumping the
results.
Signed-off-by: Grant Likely <grant.likely@arm.com>
[Rebased, added conditional.]
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Now that we are using yaml to load configurations, we might as well dump
our test data in yaml, too. This helps writing the configuration rules.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a --config option to Parser.py, to use a configuration file.

This configuration file can contain a list of transformations to apply to
the test results. This will allow to mark tests as false positives or to
waive failures.

We describe the YAML configuration file format in details in the README.md.
We also add a sample configuration, to demonstrate usage of configuration
files with the already provided sample log and sequence files.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

A skip test set is a test set, which was considered but had zero of its
tests run. This is not the same as a dropped test set, which we do not find
in the log at all.
Generate artificial test entries with the result set to SKIPPED to keep
track of the former.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Currently we are dropping the end of the device path in the ekl when it
contains `|' characters, as in sample.elk for example. Rework the ekl
parsing to fix this.

  Before: `Acpi(PNP0A08,0)/Pci(0'
   After: `Acpi(PNP0A08,0)/Pci(0|0)/Pci(0|0)'
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

From Vincent:
Lowercase change from
JSON output
Sorting command line option
Internal changes, to prepare for configuration/filtering: detect values & combined databases

Currently we read db1 and db2 from the log and sequence files respectively.
Then we compute a cross_check database with all the tests, which have run,
as well as a would_not_run database with all the tests sets, which have not
run.

Combine those databases into a unified cross_check database and get rid of
would_not_run. We create artificial test entries for the dropped tests
sets.
While at it, we keep the spurious run tests, if any, and mark them as such.

The benefits are that we do not need special handling for dropped tests
sets down the pipe anymore, and that our future configuration file
filtering rules will also apply to spurious and dropped tests.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

In the future, we will be able to override tests result values with a
config file and it will become more convenient to auto-detect the test
result values rather than hardcoding.

This changes a bit the summary table and group sections order and fields,
but is still arguably acceptable.

While at it, print a one line summary during execution.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a --sort command line option to Parser.py, to allow sorting the tests
data in a configurable manner before producing the outputs.

Also, document usage in the README.md and fix parser.py case.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a --json command line option to dump our data in json format.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Grant Likely authored Mar 10, 2021 and Vincent Stehlé committed Mar 11, 2021

Mostly an aesthetic change, but having the filename in lowercase is more
friendly when running from case sensitive filesystems, like on Linux.
Signed-off-by: Grant Likely <grant.likely@secretlab.ca>

Usability upgrades

Rework command line using argparse
Add logging and debug
Add csv output
Use iconv to cat logs

Currently we `cat' all the discrete .ekl log files into a big, single file
directly as binaries. This adds UTF-16 two bytes BOM everywhere in the
middle of the resulting cat_summary.ekl.

Fix that with iconv: we convert all the log files to UTF-8 on the fly and
back to UTF-16 while concatenating.

While at it, handle an optional command line argument to specify the
folder to search for logs.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Use logging to output debug messages and add a --debug command line option
to change the log level.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add the --csv command line option to generate a .csv file. This is handy to
manually triage issues in a spreadsheet.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Use argparse for command line arguments. This gives us the online help and
will give use more flexibility to add command line options.

We preserve the current behaviour with up to two positional arguments for
the ekl and seq input files, and up to two arguments for the key:value
search.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We capture the group iteration, start date and time, revision, description
and device path.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is easier for argument passing and more flexible for adding fields in
the future.

We take care to preserve current fields ordering in output result.md.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Added License

We generate a table for the summary rather than sections, as this is
arguably more readable.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Avoid duplicating the GUID, result and name in the log.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Currently the parser tries to record the test service as 'group' and the
test protocol as 'test set'.

However, some tests have only a protocol and no service (e.g.
LoadedImageProtocolTest). In this case, the parser will record the protocol
as 'group' and the full line dictionary as 'test set'.

Fix this case by recording an empty 'group' and the protocol as 'test set',
more in line with the generic case.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Strip .ekl input lines, to avoid capturing "\n" in the log field. Adapt
the empty lines skip test accordingly.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This allows to execute Parser.py directly with:

    $ ./Parser.py <arguments...>
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

The json module is unused: remove it.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Specify language of the python bloc. This enables syntax coloring with
pandoc.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Remove spurious whitespaces at end of line in Parser.py and README.md.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We add a Makefile and a configuration file to automate the generation of
README.pdf with pandoc.

Also, update the .gitignore accordingly.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

For the moment we ignore result.md, the output of Parser.py.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Fixes

The title is already after an ATX-style #, which makes the following
setext-style underline superfluous. Remove the underline to fix the
rendering with pandoc.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This repairs the run with default sample filenames on Linux.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>