New "known sct limitation"

See merge request !7

A specific case where SetState() return "unsupported" makes the SCT emit a
warning. As this is a valid return value, add a rule to force this warning
as "known sct limitation".

While at it, document this new "known sct limitation" category in the
README.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

New "known u-boot limitation" revealed by ACS-IR v1.0

See merge request !6

The QueryVariableInfo() implementation in U-Boot does not check properly
for all the invalid inputs conditions and we already mark some failures as
"known u-boot limitations".

The new ACS-IR v1.0 uses a new SCT version with commit 814b2af535d9
("uefi-sct/SctPkg: uefi-sct:QueryVariableInfo(EFI_VARIABLE_NON_VOLATILE)"),
which causes a new failure.

Mark this new failure as "known u-boot limitation".
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Changes to parser, config and doc

See merge request !5

Now that ACS-IR v1.0 is out, add its sequence files to the `contrib' folder
as well as to the `seq.db' database for identification.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Make our dependencies a bit clearer in the documentation and in the warning
messages we print from the parser.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Give an example command, to quickly see the help comments added by some
rules to help debugging.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Ignore a couple of network failures, which happen with edk2. Network
functions are allowed to return "unsupported" as EBBR does not require a
full implementation of the UEFI general purpose networking ABIs.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Some simple network failures happen when the ethernet cable is unplugged.
Add comments in that case, to help debugging.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Triaging results when testing for EBBR compliance is the main use case of
the parser; let's make it the default.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We add a database of the sequence files we know about with their sha256,
and we try to identify the input sequence file among those.

A `--seq-db' option is added to allow specifying the database.

Update documentation accordingly. While at it, fix spacing.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Make printing a bit closer to markdown, with header line, wider separations
between columns and capitalized header fields.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

To avoid issues with the sequence file getting modified in the test report,
recommend using the copy of EBBR.seq that is stored in this repo.
Signed-off-by: Grant Likely <grant.likely@arm.com>

A couple EBBR.yaml changes

See merge request !3

The QueryVariableInfo() implementation in U-Boot does not check properly
for all the invalid inputs conditions, which causes conformance failures.
Mark those failures as known U-Boot limitations.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

The ACS emits warnings when it detects an identity mapping between virtual
and physical memory, which is the case with U-Boot. Mark those warnings as
known ACS limitations.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Mucho changes

See merge request !2

Add a rule to mark four ACS-IR BETA 0 CreateEvent failures as "known acs
limitations". The SCT in this ACS version erroneously tests for Tpl level
high.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Handle more cases gracefully during printing, where not all the fields are
present for some of the records.

This makes the original commit 29d85950 ("parser.py: add an option for
printing") more robust and fixes a bug in the first hardening attempt of
commit f153e735

 ("parser.py: more robust printing").
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Ignore four SetTime failures happening with U-Boot RTC emulation. EBBR has
requirements on SetTime only when an RTC is present.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Classify three UpdateCapsule conformance failures as known u-boot
limitations. They are known bugs, which should be fixed in U-Boot
ultimately. Those failures are waived and do not prevent certification.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Let's classify the ACS-IR BETA 0 MemoryMap failure automatically.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Update the documentation accordingly. While at it, remove obsolete mention
of default behaviour.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a file to run `make check' on github.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We can stress the markdown generation code a bit more now that we can
re-read previously generated markdown result. Make markdown generation more
robust to missing keys to avoid exceptions.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Refine the documentation of the result.md and explain how to generate HTML
with pandoc.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is useful to perform further processing.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This simplifies the main path and prepares for future conditional .md
generation.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

dict_2_md implicitly relies on the tests all having the same order for
their dict keys during printing. This works well with all the tests read
from the .ekl log file but does not work with the "artificial" SPURIOUS
test entries, which end up having their keys in a different order.

Fix the keys order to fix .md printing.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Simplify the printing by group, to help debugging. As a bonus, this is more
efficient.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a checker for python scripts and a checker for shell scripts to the
`make check' target. Fix all issues reported by the checker in parser.py.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This makes the main path more concise and prepares for future re-reading of
the .md summary.

While at it, reformat a bit.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This reduces the use of db1 in the code for clarity.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

It is convenient to have those sequence files at hand to analyse results
obtained with the different ACS-IR published versions.
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Usability updates

See merge request !1

The sample files are just for testing. Using them by default could hide
from the user that they are calling the script wrong. Make it required
to supply the .ekl and .seq files.
Signed-off-by: Grant Likely <grant.likely@arm.com>

Use the typical __init__ == '__main__' pattern to determine if the code
is being run as a script, and only execute the main code if true. This
makes the code more portable to other projects.
Signed-off-by: Grant Likely <grant.likely@arm.com>

Adds some simple instructions to the top of the README file so that
a users doesn't need to go searching for the instructions.
Signed-off-by: Grant Likely <grant.likely@arm.com>

Update EBBR config