We were already ignoring those tests, when they resulted in failures. They
result in warnings when running the ACS-IR; add rules to ignore those as
well.

As the parser matches only plain text (no regex) we have to duplicate the
rules.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This happens when running the ACS-IR. EBBR does not require authenticated
EFI variables; we can therefore ignore those failures.
Reported-by: Masami Hiramatsu <masami.hiramatsu@linaro.org>
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This happends when running the ACS-IR. EBBR does not require authenticated
EFI variables; we can therefore ignore those failures.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We know about those failures.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Stop waiving those tests as we should really run with the right terminal
attached, but add comments to help debug.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

EBBR requires GetWakeupTime and SetWakeupTime only when wakeup is
supported. Ignore the warning raised when it is not, instead of waiving it.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

EBBR does not require watchdog timer, therefore ignore the skipped test
rather than waive it.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

EBBR specifically lifts requirements on ConnectController(). Ignore those
failures instead of waiving them.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

U-Boot v2021.04 is our baseline: stop waiving tests for older U-Boot
versions.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Ignore the skipped network tests rather than waiving them. Some network
driver features are optional and we don't have to waive anything here. We
mark the tests as ignored to distinguish them from a skipped test, which we
don't know about.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Ignore the LoadImage() HII failures rather than waiving them. This is a
notable EBBR deviation from UEFI.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Yaml Linting

Fix all the yamllint warnings found after running `make check'.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This runs yamllint for the moment, but can be extended to do more in the
future.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Deal with the case where not all records have a given field more gracefully
during printing. This happens for those fields, which are generated
on-the-fly by configuration rules for example.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Parser changes

This is useful to print a quick summary, after filtering. We document some
usage in the README.md.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is similar in its principle to the UNIX `uniq -c' and is handy for
triaging results, in conjunction with the `--fields' option.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Keeping only some fields is handy to generate summaries for triaging.

This does not apply to the markdown and config template formats, which
depend on certain fields being present.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is helpful to debug, especially with large SCT logs.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Handling missing Package, Updating Doc, Fixing Typos

Fix a few typos, some of which made us waive some failures instead of
ignoring them.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

It is always convenient to see the effect of a filter in a quantitative
manner.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

The packaging library is not always installed even though it is standard.
Try to handle its absence gracefully.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Document how the `--filter' option can be somewhat (ab)used to apply
transformations to tests data.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

cat.sh and dropped entries have been verified to be fine on runs with more
than 10k tests.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Template and filter

Allow to filter tests data using a python expression. This complements the
existing key/value filtering mechanism.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Try to semi-automate the writing of yaml configuration files by generating
a template yaml configuration from tests data, which can be further edited
manually.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

A fix for old python3 yaml versions

The yaml module has the FullLoader class since version 5.1 only; handle
this gracefully at runtime.

This fixes the following error:

  AttributeError: module 'yaml' has no attribute 'FullLoader'
Reported-by: Zhifei Yang <zhifei.yang@arm.com>
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Adding the rules processing & fixing ekl parsing.

We add a configuration file with rules for testing the EBBR specification.

Some tests correspond to cases not mandated by the EBBR specification and
we want to ignore them. Some other tests are either already fixed with a
recent U-Boot or still present with the sandbox or qemu, and we want to
issue waivers for those.

                  Original result          Forced result
  False-positive  FAILURE/WARNING/DROPPED  IGNORED
          Waived  FAILURE/WARNING/SKIPPED  WAIVED

The list has been established in part by comparing SCT runs done with the
EBBR.seq sequence file, running on arm64 U-Boot sandbox and qemu, from
commit ddaa94978583 and version v2021.04-rc2.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Grant Likely authored Mar 10, 2021 and Vincent Stehlé committed Mar 15, 2021

The libyaml implementation is significantly faster than the pure python
YAML parser. If it is present, then use CDumper when dumping the
results.
Signed-off-by: Grant Likely <grant.likely@arm.com>
[Rebased, added conditional.]
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Now that we are using yaml to load configurations, we might as well dump
our test data in yaml, too. This helps writing the configuration rules.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a --config option to Parser.py, to use a configuration file.

This configuration file can contain a list of transformations to apply to
the test results. This will allow to mark tests as false positives or to
waive failures.

We describe the YAML configuration file format in details in the README.md.
We also add a sample configuration, to demonstrate usage of configuration
files with the already provided sample log and sequence files.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

A skip test set is a test set, which was considered but had zero of its
tests run. This is not the same as a dropped test set, which we do not find
in the log at all.
Generate artificial test entries with the result set to SKIPPED to keep
track of the former.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Currently we are dropping the end of the device path in the ekl when it
contains `|' characters, as in sample.elk for example. Rework the ekl
parsing to fix this.

  Before: `Acpi(PNP0A08,0)/Pci(0'
   After: `Acpi(PNP0A08,0)/Pci(0|0)/Pci(0|0)'
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

From Vincent:
Lowercase change from
JSON output
Sorting command line option
Internal changes, to prepare for configuration/filtering: detect values & combined databases