Commit 0e87dd09 authored by Vincent Stehlé's avatar Vincent Stehlé
Browse files

EBBR.yaml: add configuration with ignored and waived



We add a configuration file with rules for testing the EBBR specification.

Some tests correspond to cases not mandated by the EBBR specification and
we want to ignore them. Some other tests are either already fixed with a
recent U-Boot or still present with the sandbox or qemu, and we want to
issue waivers for those.

                  Original result          Forced result
  False-positive  FAILURE/WARNING/DROPPED  IGNORED
          Waived  FAILURE/WARNING/SKIPPED  WAIVED

The list has been established in part by comparing SCT runs done with the
EBBR.seq sequence file, running on arm64 U-Boot sandbox and qemu, from
commit ddaa94978583 and version v2021.04-rc2.
Signed-off-by: Vincent Stehlé's avatarVincent Stehlé <vincent.stehle@arm.com>
parent 99c98e9c
###############################################################################
# EBBR configuration file #
###############################################################################
# Rules defined in this configuration are applied in order to the tests.
# Only the first matching rule does actually update the test.
# Criteria fields are matched in a "relaxed" manner (substrings can match).
# See README.md for details.
###############################################################################
# Ignored #
###############################################################################
# We force the following false-positive tests result as `IGNORED' because they
# are not mandated by EBBR, and they are too fine-grained to be removed from
# `EBBR.seq'.
- rule: Force false-positive simple network protocol ReceiveFilters failure as
pass (EBBR Table 2.4 Notable Deviations from UEFI § 2.6.2)
criteria:
descr: Simple Network Protocol Test
group: NetworkSupportTest
guid: 2143092E-03DD-4806-9FD6-08D42B9ABFC6
name: EFI_SIMPLE_NETWORK_PROTOCOL.ReceiveFilters - Invoke ReceiveFilters()
to reset multicast receive filters list and verify interface correctness
within test case
log: Status - Unsupported, Filter - 0, Mask - 0,Count - 0(0), Address - 0,
0, 0
result: FAILURE
revision: '0x00010001'
set guid: 7A1E587D-53AC-444E-A5A1-81CD3F4F04D4
sub set: ReceiveFilter_Func
test set: SimpleNetworkProtocolTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure as ignored (EBBR Table 2.2
Notable omissions from UEFI § 2.6.1)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 02C017D7-1557-47D9-BCE9-87182D07910C
log: EfiCompliantBBTestRequired_uefi.c
name: UEFI Compliant - DECOMPRESS protocol must exist
result: FAILURE
revision: '0x00010001'
set guid: 117C9ABC-489D-4504-ACDB-12AACE8F505B
sub set: RequiredElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (2) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 0CCD5843-5BB5-4FC2-A732-DB17C414A43D
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - USB Bus support protocols must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (3) as ignored (Table 2.3 UEFI
Platform-Specific Required Elements)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 1D0A2F2A-0924-4B8C-9FC7-B185CC22E118
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - EBC Interpreter protocol must exist in this platform
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (4) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 213A75C9-7F3D-42DB-B32A-02DBD698319D
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - PCI Bus support protocols must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (5) as ignored (Table 2.3 UEFI
Platform-Specific Required Elements
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 28C068F2-F398-488A-B059-534E982D9C85
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - Boot from SCSI peripheral
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (6) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 2B83418F-E7FB-4528-B6FF-C9D487AE2EFF
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - Ext SCSI Pass Thru protocol must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (7) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 2E6D1733-6D39-49AB-A886-1B6DE44566A8
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - DNS4 support is required
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (8) as ignored (Table 2.3 UEFI
Platform-Specific Required Elements)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 329027CE-406E-48C8-8AC1-A02C1A6E3983
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - Platform Driver Override protocols must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (9) as ignored (Table 2.3 UEFI
Platform-Specific Required Elements)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 3EE22696-0875-46F4-8884-BA124C7EAFF0
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - Debug support protocols must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (10) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 4C82EB2D-C785-410C-95D1-AE27122144C8
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - UEFI V6 General Network Application required
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (11) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 517BCBEB-4982-4A7E-8551-CA847DDC21C2
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - UART protocols must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (12) as ignored (Table 2.3 UEFI
Platform-Specific Required Elements)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 563F654F-ABA8-4539-804B-506305072623
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - Ata Pass Thru protocols must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (13) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 5AEA7246-BCF9-4BA4-81D2-832C984146F3
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - NVM Express Pass Thru protocol must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failures (14) as ignored (Table 2.3 UEFI
Platform-Specific Required Elements)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 5CB0CDB5-AC80-4983-B710-4B0BF0191563
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - Boot from block-oriented NVMe peripheral
result: IGNORED
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (15) as ignored (Table 2.3 UEFI
Platform-Specific Required Elements)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 6B7077A6-4B13-4E13-9B1F-0C4B3A8669E2
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - Boot from iSCSI peripheral
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (16) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 72BA0E86-58E5-48DD-8529-88C68383118D
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - Graphic Console Device
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (17) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 76A6A1B0-8C53-407D-8486-9A6E6332D3CE
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - UEFI General Network Application required
result: WARNING
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (18) as ignored (Table 2.3 UEFI
Platform-Specific Required Elements)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 87E50392-F5A2-42B8-8112-68BEC902B9BC
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - BlueTooth Classic Support protocols must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (19) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: B27660E2-0E87-4794-82F1-E6BDBD8B7442
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - IPsec protocols must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (20) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: CB6F7B77-0B15-43F7-A95B-8C7F9FD70B21
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - TLS support is required
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (21) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: E02A6EF3-4B70-40EC-AA23-50B7B972B065
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - DNS6 support is required
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (22) as ignored (Table 2.3 UEFI
Platform-Specific Required Elements)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: EFF461EB-4F56-44A5-895E-EE5EE42AD309
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - BlueTooth LE Support protocols must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (23) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: F0DC12FA-3C4B-43F7-A69E-A5BE6FCC90A1
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - EAP Support protocols must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (24) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: BF38A3FD-58AC-419A-ABC2-C60BAE9CFE67
log: Block IO - Yes, Disk IO - No, Simple FS - Yes, Unicode Collation - Yes
name: UEFI Compliant - Boot from disk protocols must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (25) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 5CB0CDB5-AC80-4983-B710-4B0BF0191563
log: NVMExpressPassThru - No
name: UEFI Compliant - Boot from block-oriented NVMe peripheral
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (26) as ignored (Table 2.3 UEFI
Platform-Specific Required Elements)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 18670DB1-89FB-4DE4-B10F-898E047D952A
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - Pointer protocols must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (27) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
This test checks the availability of the http service binding protocol and
the http utilities protocol. Even though EBBR requires HTTP Boot when the
platform supports network booting, EBBR does no require those specific
protocols.
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 77FDDB95-5969-4FB4-A218-5C0C760B0564
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - HTTP support is required
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (28) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: B7CD2D76-EA43-4013-B7D1-59EB2EC9BF1B
log: HiiDatabase - Yes, HiiString - Yes, HiiConfigRouting - No,
HiiConfigAccess - No
name: UEFI Compliant - Hii protocols must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
- rule: Force false-positive uefi compliant failure (29) as ignored (EBBR Table 2.4
Notable Deviations from UEFI § 2.6.2)
criteria:
descr: UEFI Compliant Test
device path: No device path
group: GenericTest
guid: 98551AE7-5020-4DDD-861A-CFFFB4D60382
log: EfiCompliantBBTestPlatform_uefi.c
name: UEFI Compliant - Validating a boot image received through a network device
must be implemented
result: FAILURE
revision: '0x00010001'
set guid: A0A8BED3-3D6F-4AD8-907A-84D52EE1543B
sub set: PlatformSpecificElements
test set: EFICompliantTest
update:
result: IGNORED
# We force the following dropped tests result as `IGNORED' because they are run
# only when some platform features are present, which are not mandated by EBBR.
- rule: Force dropped RNG as ignored (Table 2.3 UEFI Platform-Specific Required
Elements)
The EFI_RNG_PROTOCOL is required only if the platform has a hardware entropy
source
criteria:
result: DROPPED
revision: '0x10000'
set guid: 39FF9C71-4B41-4E5B-AED7-87C794187D66
sub set: GetInfo_Func
update:
result: WAIVED
- rule: Force dropped RNG (2) as ignored (Table 2.3 UEFI Platform-Specific
Required Elements)
The EFI_RNG_PROTOCOL is required only if the platform has a hardware entropy
source
criteria:
result: DROPPED
revision: '0x10000'
set guid: 861F4A3B-8125-4A5A-99CC-DA7779C2B020
sub set: GetInfo_Conf
update:
result: IGNORED
- rule: Force dropped RNG (3) as ignored (Table 2.3 UEFI Platform-Specific
Required Elements)
The EFI_RNG_PROTOCOL is required only if the platform has a hardware entropy
source
criteria:
result: DROPPED
revision: '0x10000'
set guid: DC5B2884-EDFB-4078-A288-4DFDA849A08D
sub set: GetRNG_Func
update:
result: WAIVED
- rule: Force dropped RNG (4) as ignored (Table 2.3 UEFI Platform-Specific
Required Elements)
The EFI_RNG_PROTOCOL is required only if the platform has a hardware entropy
source
criteria:
result: DROPPED
revision: '0x10000'
set guid: DDBBE5AB-206E-4F35-9556-186DA87C2A86
sub set: GetRNG_Conf
update:
result: WAIVED
###############################################################################
# Waived #
###############################################################################
# We force the following tests result as `WAIVED', as they are known to work
# with a recent U-Boot version.
# This section should become ultimately empty, as devices stop to use old U-Boot
# versions.
- rule: Force simple file system protocol failure working with U-Boot
v2021.04-rc2 as waived
criteria:
descr: Testing For Simple File System Protocol and EFI_FILE Protocol
group: MediaAccessTest
guid: C7900513-E931-404A-A5E3-E348407AB2A2
log: SimpleFileSystemBBTestConformance.c
name: Flush() Conformance Test - checkpoint2
result: FAILURE
revision: '0x00010000'
set guid: 41E00D7C-E0F7-455F-A617-FC6356182414
sub set: Flush_Conf
test set: SimpleFileSystemProtocolTest
update:
result: WAIVED
- rule: Force simple file system protocol failures (2) working with U-Boot
v2021.04-rc2 as waived
criteria:
descr: Testing For Simple File System Protocol and EFI_FILE Protocol
group: MediaAccessTest
guid: 033273AE-2471-4C08-B08D-EBD9DDBD5781
log: SimpleFileSystemExBBTestConformance.c
name: OpenEx() Conformance Test - checkpoint3
result: FAILURE
revision: '0x00010000'
set guid: 614CF090-E3A6-49B6-AF93-31BABA0761B4
sub set: OpenEx_Conf
test set: SimpleFileSystemProtocolTest
update:
result: WAIVED
- rule: Force simple file system protocol failures (3) working with U-Boot
v2021.04-rc2 as waived
criteria:
descr: Testing For Simple File System Protocol and EFI_FILE Protocol
group: MediaAccessTest
guid: AD02D93D-F2E8-4F25-93CE-940677B6E1B2
log: SimpleFileSystemExBBTestConformance.c
name: OpenEx() Conformance Test - checkpoint1
result: FAILURE
revision: '0x00010000'
set guid: 614CF090-E3A6-49B6-AF93-31BABA0761B4
sub set: OpenEx_Conf
test set: SimpleFileSystemProtocolTest