Commit 0cc9573d authored by Grant Likely's avatar Grant Likely
Browse files

Merge branch 'vincent/master' into 'main'

Mucho changes

See merge request !2
parents d56fa37f 9d1ddc37
---
name: CI
# yamllint disable-line rule:truthy
on: [push]
jobs:
check:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Install dependencies and check
run: |
sudo apt install flake8
make check
...@@ -1553,12 +1553,88 @@ ...@@ -1553,12 +1553,88 @@
update: update:
result: IGNORED result: IGNORED
- rule: Force false-positive time services set time failure as ignored.
SetTime does not retain year properly with U-Boot RTC emulation.
SetTime is required only when an RTC is present (EBBR Table 2.6
EFI_RUNTIME_SERVICES Implementation Requirements).
criteria:
descr: Time Services Test
device path: No device path
group: RuntimeServicesTest
guid: 3B96A20C-2B1F-44EA-BAA9-F96FEE131D05
log: Status - Success, TPL - 4
name: RT.SetTime - Verify year after change
result: FAILURE
revision: '0x00010000'
set guid: 603B46BE-7E14-408A-93D7-DD9DEC732968
sub set: SetTime_Func
test set: TimeServicesTest
update:
result: IGNORED
- rule: Force false-positive time services set time failure (2) as ignored.
SetTime does not retain year properly with U-Boot RTC emulation.
SetTime is required only when an RTC is present (EBBR Table 2.6
EFI_RUNTIME_SERVICES Implementation Requirements).
criteria:
descr: Time Services Test
device path: No device path
group: RuntimeServicesTest
guid: 2D5CDBE5-1055-4EF6-8E90-0C993F93F698
log: Status - Success, TPL - 4
name: RT.SetTime - Verify month after change
result: FAILURE
revision: '0x00010000'
set guid: 603B46BE-7E14-408A-93D7-DD9DEC732968
sub set: SetTime_Func
test set: TimeServicesTest
update:
result: IGNORED
- rule: Force false-positive time services set time failure (3) as ignored.
SetTime does not retain year properly with U-Boot RTC emulation.
SetTime is required only when an RTC is present (EBBR Table 2.6
EFI_RUNTIME_SERVICES Implementation Requirements).
criteria:
descr: Time Services Test
device path: No device path
group: RuntimeServicesTest
guid: E664E1D7-B733-410D-BC53-D4CFF2464355
log: Status - Success, TPL - 8
name: RT.SetTime - Verify year after change
result: FAILURE
revision: '0x00010000'
set guid: 603B46BE-7E14-408A-93D7-DD9DEC732968
sub set: SetTime_Func
test set: TimeServicesTest
update:
result: IGNORED
- rule: Force false-positive time services set time failure (4) as ignored.
SetTime does not retain year properly with U-Boot RTC emulation.
SetTime is required only when an RTC is present (EBBR Table 2.6
EFI_RUNTIME_SERVICES Implementation Requirements).
criteria:
descr: Time Services Test
device path: No device path
group: RuntimeServicesTest
guid: DA4B19E7-F605-4FB9-A181-CCD335290BFE
log: Status - Success, TPL - 8
name: RT.SetTime - Verify month after change
result: FAILURE
revision: '0x00010000'
set guid: 603B46BE-7E14-408A-93D7-DD9DEC732968
sub set: SetTime_Func
test set: TimeServicesTest
update:
result: IGNORED
############################################################################### ###############################################################################
# Known U-Boot limitations # # Known U-Boot limitations #
############################################################################### ###############################################################################
# We force the following tests result as `KNOWN U-BOOT LIMITATION'. They are # We force the following tests result as `KNOWN U-BOOT LIMITATION'. They are
# genuine bugs, which much ultimately be fixed. We know about them; they are due # genuine bugs, which must ultimately be fixed. We know about them; they are due
# to U-Boot FAT filesystem implementation limitations and they do not prevent an # to U-Boot FAT filesystem implementation limitations and they do not prevent an
# OS to boot. # OS to boot.
...@@ -1864,6 +1940,109 @@ ...@@ -1864,6 +1940,109 @@
update: update:
result: KNOWN U-BOOT LIMITATION result: KNOWN U-BOOT LIMITATION
- rule: Force misc runtime services update capsule conformance failure as known
U-Boot limitation
UpdateCapsule implementation in U-Boot up to (at least) v2021.07 does not
check properly for all invalid inputs conditions
criteria:
descr: Misc Runtime Services Test
device path: No device path
group: RuntimeServicesTest
guid: 304F6960-79D0-4F17-8811-620FC6BDB0D4
log: Status - Unsupported
name: RT.UpdateCapsule - invoke UpdateCapsule with invalid ScatterGatherList
result: FAILURE
revision: '0x00010000'
set guid: 7227CFAC-CA96-4680-9314-E3FBC60A2A61
sub set: UpdateCapsule_Conf
test set: MiscRuntimeServicesTest
update:
result: KNOWN U-BOOT LIMITATION
- rule: Force misc runtime services update capsule conformance failure (2) as
known U-Boot limitation
UpdateCapsule implementation in U-Boot up to (at least) v2021.07 does not
check properly for all invalid inputs conditions
criteria:
descr: Misc Runtime Services Test
device path: No device path
group: RuntimeServicesTest
guid: 18F86BF8-76CF-4225-8E3E-1B1F63432600
log: Status - Unsupported
name: RT.UpdateCapsule - invoke UpdateCapsule with invalid Flags
result: FAILURE
revision: '0x00010000'
set guid: 7227CFAC-CA96-4680-9314-E3FBC60A2A61
sub set: UpdateCapsule_Conf
test set: MiscRuntimeServicesTest
update:
result: KNOWN U-BOOT LIMITATION
- rule: Force misc runtime services update capsule conformance failure (3) as
known U-Boot limitation
UpdateCapsule implementation in U-Boot up to (at least) v2021.07 does not
check properly for all invalid inputs conditions
criteria:
descr: Misc Runtime Services Test
device path: No device path
group: RuntimeServicesTest
guid: 145E4790-3342-4C8C-99F2-7F6500FD26E5
log: Status - Unsupported
name: RT.UpdateCapsule - invoke UpdateCapsule with invalid Flags
result: FAILURE
revision: '0x00010000'
set guid: 7227CFAC-CA96-4680-9314-E3FBC60A2A61
sub set: UpdateCapsule_Conf
test set: MiscRuntimeServicesTest
update:
result: KNOWN U-BOOT LIMITATION
###############################################################################
# Known ACS limitations #
###############################################################################
# We force the following tests result as `KNOWN ACS LIMITATION'. They are
# genuine bugs, which are fixed in a more recent version of the ACS or which
# must ultimately be fixed and which we know about.
- rule: Force boot services memory map test failure with ACS-IR
v21.05_0.8_BETA-0 as known ACS limitation
criteria:
descr: SbbrBootServices Tests
device path: No device path
group: BootServicesTest
guid: 58A44F17-760E-478B-BAEF-E20BFBCD7457
log: MemoryMap Not Found
name: MemoryMap
result: FAILURE
revision: '0x00010000'
set guid: 8540C12D-5413-4A0B-A545-89A8E3C7DA4B
sub set: MemoryMap
test set: SbbrBootServices
update:
result: KNOWN ACS LIMITATION
- rule: Force event services create event test failures with ACS-IR
v21.05_0.8_BETA-0 as known ACS limitation.
The ACS BETA 0 has an old version of the SCT, which erroneously creates
events with Tpl level high.
This is fixed in ACS BETA 1 and up.
criteria:
descr: Event, Timer, and Task Priority Services Test
device path: No device path
group: BootServicesTest
guid: EF317ADE-8668-456F-BED9-766056672DFF
log: NotifyTpl - 31
name: BS.CreateEvent - Create event with all valid event type and supported
TPL.
result: FAILURE
revision: '0x00010000'
set guid: 83CF7F0B-C274-4918-AEA6-413DA9CF68CF
sub set: CreateEvent_Func
test set: EventTimerandPriorityServicesTest
update:
result: KNOWN ACS LIMITATION
############################################################################### ###############################################################################
# Comments # # Comments #
############################################################################### ###############################################################################
...@@ -1976,23 +2155,6 @@ ...@@ -1976,23 +2155,6 @@
# We add comments to the following tests failures for which we know a likely # We add comments to the following tests failures for which we know a likely
# cause, to help triaging/debugging. # cause, to help triaging/debugging.
- rule: Add comments to boot services memory map test failure with ACS-IR
v21.05_0.8_BETA-0
criteria:
descr: SbbrBootServices Tests
device path: No device path
group: BootServicesTest
guid: 58A44F17-760E-478B-BAEF-E20BFBCD7457
log: MemoryMap Not Found
name: MemoryMap
result: FAILURE
revision: '0x00010000'
set guid: 8540C12D-5413-4A0B-A545-89A8E3C7DA4B
sub set: MemoryMap
test set: SbbrBootServices
update:
comments: This is a known issue with ACS-IR v21.05_0.8_BETA-0
- rule: Add comments to failure due to missing ESP - rule: Add comments to failure due to missing ESP
criteria: criteria:
descr: Image Services Test descr: Image Services Test
......
...@@ -6,7 +6,8 @@ all: doc ...@@ -6,7 +6,8 @@ all: doc
help: help:
@echo 'Targets:' @echo 'Targets:'
@echo ' all' @echo ' all'
@echo ' check Perform sanity checks (currently yamllint)' @echo ' check Perform sanity checks'
@echo ' (currently yamllint, shellcheck and flake8)'
@echo ' clean' @echo ' clean'
@echo ' doc Generate README.pdf' @echo ' doc Generate README.pdf'
@echo ' help Print this help.' @echo ' help Print this help.'
...@@ -18,6 +19,8 @@ doc: README.pdf ...@@ -18,6 +19,8 @@ doc: README.pdf
check: check:
yamllint . yamllint .
shellcheck $$(find -name '*.sh')
flake8
clean: clean:
-rm -f README.pdf -rm -f README.pdf
...@@ -23,12 +23,27 @@ INFO main: 0 dropped(s), 1 failure(s), 93 ignored(s), 106 known u-boot limitatio ...@@ -23,12 +23,27 @@ INFO main: 0 dropped(s), 1 failure(s), 93 ignored(s), 106 known u-boot limitatio
``` ```
## Usage ## Usage
Usage to generate a "result md" is such. `python3 parser.py <log_file.ekl> <seq_file.seq>`
If you do no provided any command line arguments it will use `sample.ekl` and `sample.seq`. Usage to generate a `result.md` is such:
The output filename can be specified with `--md <filename>`.
``` {.sh}
$ python3 parser.py <log_file.ekl> <seq_file.seq>
```
The output filename can be specified with the `--md` option:
``` {.sh}
$ ./parser.py --md out.md ...
```
An online help is available with the `-h` option. An online help is available with the `-h` option.
The generated `result md` can be easily converted to HTML using `pandoc` with:
``` {.sh}
$ pandoc -oresult.html result.md
```
### Custom search ### Custom search
For a custom Key:value search, the next two arguments *MUST be included together.* The program will search and display files that met that constraint, without the crosscheck, and display the names, guid, and key:value to the command line. `python3 parser.py <file.ekl> <file.seq> <search key> <search value>` For a custom Key:value search, the next two arguments *MUST be included together.* The program will search and display files that met that constraint, without the crosscheck, and display the names, guid, and key:value to the command line. `python3 parser.py <file.ekl> <file.seq> <search key> <search value>`
...@@ -131,6 +146,23 @@ $ ./parser.py \ ...@@ -131,6 +146,23 @@ $ ./parser.py \
--fields 'count,result,name' --uniq --print ... --fields 'count,result,name' --uniq --print ...
``` ```
### Re-reading markdown results
It is possible to re-read a previously generated markdown results file with the
`--input-md` option. This can be useful to perform further processing on the
tests.
Example command to read a previously generated markdown:
``` {.sh}
$ ./parser.py --input-md 'result.md' ...
```
* By default an output markdown is still generated, except in the case where the
input and output markdown have the same filename.
* The generated markdown results do not contain the "passed" tests. They can
therefore not be re-read.
## Configuration file ## Configuration file
It is possible to use a configuration file with command line option `--config It is possible to use a configuration file with command line option `--config
...@@ -190,13 +222,13 @@ tests. ...@@ -190,13 +222,13 @@ tests.
### Sample ### Sample
A `sample.yaml` configuration file is provided as example, to use with the In the folder `sample`, a `sample.yaml` configuration file is provided as
`sample.ekl` and `sample.seq` files. example, to use with the `sample.ekl` and `sample.seq` files.
Try it with: Try it with:
``` {.sh} ``` {.sh}
$ ./parser.py --config sample.yaml ... $ ./parser.py --config sample/sample.yaml sample/sample.ekl sample/sample.seq
``` ```
### Generating a configuration template ### Generating a configuration template
...@@ -232,6 +264,10 @@ override the result of some tests with the following ones: ...@@ -232,6 +264,10 @@ override the result of some tests with the following ones:
We know about them; they are due to U-Boot FAT We know about them; they are due to U-Boot FAT
filesystem implementation limitations and they do filesystem implementation limitations and they do
not prevent an OS to boot. not prevent an OS to boot.
`KNOWN ACS LIMITATION` Genuine bugs, which are fixed in a more recent version
of the ACS or which must ultimately be fixed and which
we know about.
------------------------------------------------------------------------------- -------------------------------------------------------------------------------
Some of the rules just add a `comments` field with some help text. Some of the rules just add a `comments` field with some help text.
...@@ -250,8 +286,18 @@ It is possible to convert this `README.md` into `README.pdf` with pandoc using ...@@ -250,8 +286,18 @@ It is possible to convert this `README.md` into `README.pdf` with pandoc using
### Sanity checks ### Sanity checks
To perform sanity checks, run `make check`. For the moment this runs `yamllint`, To perform sanity checks, run `make check`. It runs a number of checkers and
which will inspect all [YAML] files and report errors. See `make help`. reports errors:
-------------------------------
Checker Target
------------- ----------------
`flake8` Python scripts.
`yamllint` [YAML] files.
`shellcheck` Shell scripts.
-------------------------------
See `make help`.
### db structure: ### db structure:
...@@ -326,3 +372,19 @@ their test run according to the log file. ...@@ -326,3 +372,19 @@ their test run according to the log file.
We create artificial tests entries for those dropped tests sets, with the We create artificial tests entries for those dropped tests sets, with the
"result" fields set to "SKIPPED". "result" fields set to "SKIPPED".
## Contributed files
A few contributed files are stored in sub-folders under `contrib` for
convenience:
-------------------------------------------------------------------------------
Sub-folder Contents
--------------------- --------------------------------------------------------
`v21.05_0.8_BETA-0/` EBBR sequence file from [ACS-IR v21.05_0.8_BETA-0].
`v21.07_0.9_BETA/` EBBR sequence files from [ACS-IR v21.07_0.9_BETA].
-------------------------------------------------------------------------------
[ACS-IR v21.05_0.8_BETA-0]: https://github.com/ARM-software/arm-systemready/tree/main/IR/prebuilt_images/v21.05_0.8_BETA-0
[ACS-IR v21.07_0.9_BETA]: https://github.com/ARM-software/arm-systemready/tree/main/IR/prebuilt_images/v21.07_0.9_BETA
B[Test Case] B[Test Case]
B[Test Case] B[Test Case]
B[Test Case] B[Test Case]
#!/usr/bin/env python3 #!/usr/bin/env python3
#SCT log parser # SCT log parser
import sys import sys
...@@ -33,7 +33,7 @@ else: ...@@ -33,7 +33,7 @@ else:
yaml_load_args = {} yaml_load_args = {}
#based loosley on https://stackoverflow.com/a/4391978 # based loosley on https://stackoverflow.com/a/4391978
# returns a filtered dict of dicts that meet some Key-value pair. # returns a filtered dict of dicts that meet some Key-value pair.
# I.E. key="result" value="FAILURE" # I.E. key="result" value="FAILURE"
def key_value_find(list_1, key, value): def key_value_find(list_1, key, value):
...@@ -44,21 +44,25 @@ def key_value_find(list_1, key, value): ...@@ -44,21 +44,25 @@ def key_value_find(list_1, key, value):
return found return found
#Were we intrept test logs into test dicts # Were we intrept test logs into test dicts
def test_parser(string, current): def test_parser(string, current):
test_list = { test_list = {
"name": string[2], #FIXME:Sometimes, SCT has name and Description, "name": string[2],
"result": string[1], # FIXME:Sometimes, SCT has name and Description,
**current, "result": string[1],
"guid": string[0], #FIXME:GUID's overlap **current,
#"comment": string[-1], #FIXME:need to hash this out, sometime there is no comments "guid": string[0],
"log": ' '.join(string[3:]) # FIXME:GUID's overlap
# "comment": string[-1], # FIXME:need to hash this out,
# sometime there is no comments
"log": ' '.join(string[3:])
} }
return test_list return test_list
#Parse the ekl file, and create a map of the tests
def ekl_parser (file): # Parse the ekl file, and create a map of the tests
#create our "database" dict def ekl_parser(file):
# create our "database" dict
temp_list = list() temp_list = list()
# All tests are grouped by the "HEAD" line, which precedes them. # All tests are grouped by the "HEAD" line, which precedes them.
current = {} current = {}
...@@ -122,14 +126,17 @@ def ekl_parser (file): ...@@ -122,14 +126,17 @@ def ekl_parser (file):
'device path': '|'.join(split_line[13:]), 'device path': '|'.join(split_line[13:]),
} }
#FIXME:? EKL file has an inconsistent line structure, # FIXME:? EKL file has an inconsistent line structure,
# sometime we see a line that consits ' dump of GOP->I\n' # sometime we see a line that consits ' dump of GOP->I\n'
#easiest way to skip is check for blank space in the first char # easiest way to skip is check for blank space in the first char
elif split_line[0] != '' and split_line[0][0] != " ": elif split_line[0] != '' and split_line[0][0] != " ":
try: try:
#deliminiate on ':' for tests # deliminiate on ':' for tests
split_test = [new_string for old_string in split_line for new_string in old_string.split(':')] split_test = [new_string for old_string in
#put the test into a dict, and then place that dict in another dict with GUID as key split_line for new_string in
old_string.split(':')]
# put the test into a dict, and then place that dict in another
# dict with GUID as key
tmp_dict = test_parser(split_test, current) tmp_dict = test_parser(split_test, current)
temp_list.append(tmp_dict) temp_list.append(tmp_dict)
n += 1 n += 1
...@@ -144,74 +151,85 @@ def ekl_parser (file): ...@@ -144,74 +151,85 @@ def ekl_parser (file):
return temp_list return temp_list
#Parse Seq file, used to tell which tests should run.
# Parse Seq file, used to tell which tests should run.
def seq_parser(file): def seq_parser(file):
temp_dict = list() temp_dict = list()
lines=file.readlines() lines = file.readlines()
magic=7 #a test in a seq file is 7 lines, if not mod7, something wrong.. magic = 7
if len(lines)%magic != 0: # a test in a seq file is 7 lines, if not mod7, something wrong..
if len(lines) % magic != 0:
sys.exit("seqfile cut short, should be mod7") sys.exit("seqfile cut short, should be mod7")
#the utf-16 char makes this looping a bit harder, so we use x+(i) where i is next 0-6th # the utf-16 char makes this looping a bit harder, so we use x+(i) where i
for x in range(0,len(lines),magic): #loop ever "7 lines" # is next 0-6th
#(x+0)[Test Case] # loop ever "7 lines"
#(x+1)Revision=0x10000 for x in range(0, len(lines), magic):
#(x+2)Guid=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX # (x+0)[Test Case]
#(x+3)Name=InstallAcpiTableFunction # (x+1)Revision=0x10000
#(x+4)Order=0xFFFFFFFF # (x+2)Guid=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
#(x+5)Iterations=0xFFFFFFFF # (x+3)Name=InstallAcpiTableFunction
#(x+6)(utf-16 char) # (x+4)Order=0xFFFFFFFF
#currently only add tests that are supposed to run, should add all? # (x+5)Iterations=0xFFFFFFFF
#0xFFFFFFFF in "Iterations" means the test is NOT supposed to run # (x+6)(utf-16 char)
if not "0xFFFFFFFF" in lines[x+5]: # currently only add tests that are supposed to run, should add all?
# 0xFFFFFFFF in "Iterations" means the test is NOT supposed to run
if "0xFFFFFFFF" not in lines[x + 5]:
seq_dict = { seq_dict = {
"name": lines[x+3][5:-1],#from after "Name=" to end (5char long) # from after "Name=" to end (5char long)
"guid": lines[x+2][5:-1],#from after"Guid=" to the end, (5char long) "name": lines[x + 3][5:-1],
"Iteration": lines[x+5][11:-1],#from after "Iterations=" (11char long) # from after"Guid=" to the end, (5char long)
"rev": lines[x+1][9:-1],#from after "Revision=" (9char long) "guid": lines[x + 2][5:-1],
"Order": lines[x+4][6:-1]#from after "Order=" (6char long) # from after "Iterations=" (11char long)
"Iteration": lines[x + 5][11:-1],
# from after "Revision=" (9char long)
"rev": lines[x + 1][9:-1],
# from after "Order=" (6char long)
"Order": lines[x + 4][6:-1]
} }
temp_dict.append(seq_dict) #put in a dict based on guid # put in a dict based on guid
temp_dict.append(seq_dict)
return temp_dict return temp_dict
#group items by key, and print by key
#we slowly iterate through the list, group and print groups # Print items by "group"
def key_tree_2_md(input_list,file,key): def key_tree_2_md(input_list, file):
#make a copy so we don't destroy the first list. h = {}
temp_list = input_list.copy()
while temp_list: # Bin by group
test_dict = temp_list.pop() for t in input_list:
found, not_found = [test_dict],[] g = t['group']
#go through whole list looking for key match
while temp_list: if g not in h:
next_dict = temp_list.pop() h[g] = []
if next_dict[key] == test_dict[key]: #if match add to found
found.append(next_dict) h[g].append(t)
else: #else not found
not_found.append(next_dict) # Print each group
temp_list = not_found #start over with found items removed for g in sorted(h.keys()):
file.write("### " + test_dict[key]) file.write("### " + g)