gitlab.arm.com will be in the maintainance mode on Wednesday June 29th 01:00 - 10:00 (UTC+1). Repositories is read only during the maintainance.

Commit 17177f8a authored by Vincent Stehlé's avatar Vincent Stehlé
Browse files

parser.py: add an option to collapse duplicates



This is similar in its principle to the UNIX `uniq -c' and is handy for
triaging results, in conjunction with the `--fields' option.
Signed-off-by: Vincent Stehlé's avatarVincent Stehlé <vincent.stehle@arm.com>
parent 8bd1de3c
...@@ -78,6 +78,22 @@ $ ./parser.py --fields 'result,sub set,descr,name,log' ... ...@@ -78,6 +78,22 @@ $ ./parser.py --fields 'result,sub set,descr,name,log' ...
The csv format can retain the fields order. The csv format can retain the fields order.
### Collapsing duplicates
It is possible to "collapse" duplicate tests data into a single entry using the
`--uniq` option, much similar in principle to the UNIX `uniq -c` command.
This step happens after tests and fields filtering, and it adds a "count" field.
Example command, suitable for triaging:
``` {.sh}
$ ./parser.py \
--filter "x.update({'log': re.sub(r'/.*/', '', x['log'])}) \
or x['result'] != 'PASS'" \
--fields 'count,result,sub set,descr,name,log' --uniq ...
```
## Configuration file ## Configuration file
It is possible to use a configuration file with command line option `--config It is possible to use a configuration file with command line option `--config
......
...@@ -361,6 +361,35 @@ def keep_fields(cross_check, fields): ...@@ -361,6 +361,35 @@ def keep_fields(cross_check, fields):
del x[k] del x[k]
# Do a "uniq" pass on the data
# All duplicate entries are collapsed into a single one
# We add a "count" field
def uniq(cross_check):
logging.debug("Collapsing duplicates")
# First pass to count all occurences
h = {}
for x in cross_check:
i = ''
for k in sorted(x.keys()):
i += f"{k}:{x[k]} "
if i not in h:
h[i] = {
'count': 0,
**x,
}
h[i]['count'] += 1
# Transform back to list
r = list(h.values())
logging.info(f"{len(r)} unique entries")
return r
# Discover fields # Discover fields
# The fields can be supplied as a comma-separated list # The fields can be supplied as a comma-separated list
# Order is preserved # Order is preserved
...@@ -542,6 +571,8 @@ def main(): ...@@ -542,6 +571,8 @@ def main():
parser.add_argument('--filter', help='Python expression to filter results') parser.add_argument('--filter', help='Python expression to filter results')
parser.add_argument( parser.add_argument(
'--fields', help='Comma-separated list of fields to write') '--fields', help='Comma-separated list of fields to write')
parser.add_argument(
'--uniq', action='store_true', help='Collapse duplicates')
parser.add_argument( parser.add_argument(
'log_file', nargs='?', default='sample.ekl', 'log_file', nargs='?', default='sample.ekl',
help='Input .ekl filename') help='Input .ekl filename')
...@@ -656,6 +687,10 @@ def main(): ...@@ -656,6 +687,10 @@ def main():
if args.fields is not None: if args.fields is not None:
keep_fields(cross_check, args.fields) keep_fields(cross_check, args.fields)
# Do a `uniq` pass if requested
if args.uniq:
cross_check = uniq(cross_check)
# Auto-discover the fields and take the option into account # Auto-discover the fields and take the option into account
fields = discover_fields(cross_check, args.fields) fields = discover_fields(cross_check, args.fields)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment