GitLab

Update the documentation accordingly. While at it, remove obsolete mention
of default behaviour.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a file to run `make check' on github.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We can stress the markdown generation code a bit more now that we can
re-read previously generated markdown result. Make markdown generation more
robust to missing keys to avoid exceptions.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Refine the documentation of the result.md and explain how to generate HTML
with pandoc.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is useful to perform further processing.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This simplifies the main path and prepares for future conditional .md
generation.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

dict_2_md implicitly relies on the tests all having the same order for
their dict keys during printing. This works well with all the tests read
from the .ekl log file but does not work with the "artificial" SPURIOUS
test entries, which end up having their keys in a different order.

Fix the keys order to fix .md printing.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Simplify the printing by group, to help debugging. As a bonus, this is more
efficient.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a checker for python scripts and a checker for shell scripts to the
`make check' target. Fix all issues reported by the checker in parser.py.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This makes the main path more concise and prepares for future re-reading of
the .md summary.

While at it, reformat a bit.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This reduces the use of db1 in the code for clarity.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

It is convenient to have those sequence files at hand to analyse results
obtained with the different ACS-IR published versions.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Usability updates

See merge request !1

The sample files are just for testing. Using them by default could hide
from the user that they are calling the script wrong. Make it required
to supply the .ekl and .seq files.
Signed-off-by: Grant Likely <grant.likely@arm.com>

Use the typical __init__ == '__main__' pattern to determine if the code
is being run as a script, and only execute the main code if true. This
makes the code more portable to other projects.
Signed-off-by: Grant Likely <grant.likely@arm.com>

Adds some simple instructions to the top of the README file so that
a users doesn't need to go searching for the instructions.
Signed-off-by: Grant Likely <grant.likely@arm.com>

Update EBBR config

Start to document the EBBR.yaml configuration file specificities.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We put the links definitions in the same section as the first use, to
facilitate reading the README.md directly.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Ignore monotonic count failures as it is not required by EBBR. A change to
EBBR to explicitly clarify those cases is pending.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Ignore daylight and time zone failures as they are not required by EBBR.
A change to EBBR to explicitly clarify those cases is pending.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Those failures happen when the system does not have an EFI System Partition
and is not capable of persisting EFI variables across reset. Add comments
in that case, to help debugging.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

The boot services memory map test has a known issue with the ACS-IR
v21.05_0.8_BETA-0. The real fix is to update the ACS-IR test code, but we
add a rule already to add some comments for that case, to help triaging.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We want to fix those failures in U-Boot.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We know about this failure.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This simple network protocol failure is not showing up anymore. Remove this
rule, as we want to reduce the list of WAIVED to zero.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Simple Text Input Ex Protocol tests can be skipped when the SCT is unable
to locate the console device bound to the protocol. This is the case with
U-Boot.

EBBR has requirements on the console, but it does not specifically require
that a device be always bound to the Simple Text Input Ex Protocol.

Simply ignore this skipped test instead of waiving it.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is not mandated by EBBR and may be skipped safely. Ignore the skipped
tests rather than waiving them.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We were already ignoring those tests, when they resulted in failures. They
result in warnings when running the ACS-IR; add rules to ignore those as
well.

As the parser matches only plain text (no regex) we have to duplicate the
rules.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This happens when running the ACS-IR. EBBR does not require authenticated
EFI variables; we can therefore ignore those failures.
Reported-by: Masami Hiramatsu <masami.hiramatsu@linaro.org>
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This happends when running the ACS-IR. EBBR does not require authenticated
EFI variables; we can therefore ignore those failures.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We know about those failures.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Stop waiving those tests as we should really run with the right terminal
attached, but add comments to help debug.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

EBBR requires GetWakeupTime and SetWakeupTime only when wakeup is
supported. Ignore the warning raised when it is not, instead of waiving it.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

EBBR does not require watchdog timer, therefore ignore the skipped test
rather than waive it.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

EBBR specifically lifts requirements on ConnectController(). Ignore those
failures instead of waiving them.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

U-Boot v2021.04 is our baseline: stop waiving tests for older U-Boot
versions.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Ignore the skipped network tests rather than waiving them. Some network
driver features are optional and we don't have to waive anything here. We
mark the tests as ignored to distinguish them from a skipped test, which we
don't know about.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Ignore the LoadImage() HII failures rather than waiving them. This is a
notable EBBR deviation from UEFI.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>