We add a database of the sequence files we know about with their sha256,
and we try to identify the input sequence file among those.

A `--seq-db' option is added to allow specifying the database.

Update documentation accordingly. While at it, fix spacing.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Make printing a bit closer to markdown, with header line, wider separations
between columns and capitalized header fields.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Handle more cases gracefully during printing, where not all the fields are
present for some of the records.

This makes the original commit 29d85950 ("parser.py: add an option for
printing") more robust and fixes a bug in the first hardening attempt of
commit f153e735

 ("parser.py: more robust printing").

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We can stress the markdown generation code a bit more now that we can
re-read previously generated markdown result. Make markdown generation more
robust to missing keys to avoid exceptions.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is useful to perform further processing.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This simplifies the main path and prepares for future conditional .md
generation.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

dict_2_md implicitly relies on the tests all having the same order for
their dict keys during printing. This works well with all the tests read
from the .ekl log file but does not work with the "artificial" SPURIOUS
test entries, which end up having their keys in a different order.

Fix the keys order to fix .md printing.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Simplify the printing by group, to help debugging. As a bonus, this is more
efficient.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a checker for python scripts and a checker for shell scripts to the
`make check' target. Fix all issues reported by the checker in parser.py.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This makes the main path more concise and prepares for future re-reading of
the .md summary.

While at it, reformat a bit.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This reduces the use of db1 in the code for clarity.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

The sample files are just for testing. Using them by default could hide
from the user that they are calling the script wrong. Make it required
to supply the .ekl and .seq files.

Signed-off-by: Grant Likely <grant.likely@arm.com>

Use the typical __init__ == '__main__' pattern to determine if the code
is being run as a script, and only execute the main code if true. This
makes the code more portable to other projects.

Signed-off-by: Grant Likely <grant.likely@arm.com>

Deal with the case where not all records have a given field more gracefully
during printing. This happens for those fields, which are generated
on-the-fly by configuration rules for example.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is useful to print a quick summary, after filtering. We document some
usage in the README.md.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is similar in its principle to the UNIX `uniq -c' and is handy for
triaging results, in conjunction with the `--fields' option.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Keeping only some fields is handy to generate summaries for triaging.

This does not apply to the markdown and config template formats, which
depend on certain fields being present.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is helpful to debug, especially with large SCT logs.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

It is always convenient to see the effect of a filter in a quantitative
manner.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

The packaging library is not always installed even though it is standard.
Try to handle its absence gracefully.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Allow to filter tests data using a python expression. This complements the
existing key/value filtering mechanism.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Try to semi-automate the writing of yaml configuration files by generating
a template yaml configuration from tests data, which can be further edited
manually.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

The yaml module has the FullLoader class since version 5.1 only; handle
this gracefully at runtime.

This fixes the following error:

  AttributeError: module 'yaml' has no attribute 'FullLoader'

Reported-by: Zhifei Yang <zhifei.yang@arm.com>
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Grant Likely authored Mar 10, 2021 and Vincent Stehlé committed Mar 15, 2021

The libyaml implementation is significantly faster than the pure python
YAML parser. If it is present, then use CDumper when dumping the
results.

Signed-off-by: Grant Likely <grant.likely@arm.com>
[Rebased, added conditional.]
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Now that we are using yaml to load configurations, we might as well dump
our test data in yaml, too. This helps writing the configuration rules.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a --config option to Parser.py, to use a configuration file.

This configuration file can contain a list of transformations to apply to
the test results. This will allow to mark tests as false positives or to
waive failures.

We describe the YAML configuration file format in details in the README.md.
We also add a sample configuration, to demonstrate usage of configuration
files with the already provided sample log and sequence files.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

A skip test set is a test set, which was considered but had zero of its
tests run. This is not the same as a dropped test set, which we do not find
in the log at all.
Generate artificial test entries with the result set to SKIPPED to keep
track of the former.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Currently we are dropping the end of the device path in the ekl when it
contains `|' characters, as in sample.elk for example. Rework the ekl
parsing to fix this.

  Before: `Acpi(PNP0A08,0)/Pci(0'
   After: `Acpi(PNP0A08,0)/Pci(0|0)/Pci(0|0)'

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Currently we read db1 and db2 from the log and sequence files respectively.
Then we compute a cross_check database with all the tests, which have run,
as well as a would_not_run database with all the tests sets, which have not
run.

Combine those databases into a unified cross_check database and get rid of
would_not_run. We create artificial test entries for the dropped tests
sets.
While at it, we keep the spurious run tests, if any, and mark them as such.

The benefits are that we do not need special handling for dropped tests
sets down the pipe anymore, and that our future configuration file
filtering rules will also apply to spurious and dropped tests.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

In the future, we will be able to override tests result values with a
config file and it will become more convenient to auto-detect the test
result values rather than hardcoding.

This changes a bit the summary table and group sections order and fields,
but is still arguably acceptable.

While at it, print a one line summary during execution.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a --sort command line option to Parser.py, to allow sorting the tests
data in a configurable manner before producing the outputs.

Also, document usage in the README.md and fix parser.py case.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a --json command line option to dump our data in json format.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Grant Likely authored Mar 10, 2021 and Vincent Stehlé committed Mar 11, 2021

Mostly an aesthetic change, but having the filename in lowercase is more
friendly when running from case sensitive filesystems, like on Linux.

Signed-off-by: Grant Likely <grant.likely@secretlab.ca>

Use logging to output debug messages and add a --debug command line option
to change the log level.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add the --csv command line option to generate a .csv file. This is handy to
manually triage issues in a spreadsheet.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Use argparse for command line arguments. This gives us the online help and
will give use more flexibility to add command line options.

We preserve the current behaviour with up to two positional arguments for
the ekl and seq input files, and up to two arguments for the key:value
search.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We capture the group iteration, start date and time, revision, description
and device path.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is easier for argument passing and more flexible for adding fields in
the future.

We take care to preserve current fields ordering in output result.md.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We generate a table for the summary rather than sections, as this is
arguably more readable.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Avoid duplicating the GUID, result and name in the log.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>