Commit 17177f8a authored by Vincent Stehlé's avatar Vincent Stehlé
Browse files add an option to collapse duplicates

This is similar in its principle to the UNIX `uniq -c' and is handy for
triaging results, in conjunction with the `--fields' option.

Signed-off-by: Vincent Stehlé's avatarVincent Stehlé <>
parent 8bd1de3c
......@@ -78,6 +78,22 @@ $ ./ --fields 'result,sub set,descr,name,log' ...
The csv format can retain the fields order.
### Collapsing duplicates
It is possible to "collapse" duplicate tests data into a single entry using the
`--uniq` option, much similar in principle to the UNIX `uniq -c` command.
This step happens after tests and fields filtering, and it adds a "count" field.
Example command, suitable for triaging:
``` {.sh}
$ ./ \
--filter "x.update({'log': re.sub(r'/.*/', '', x['log'])}) \
or x['result'] != 'PASS'" \
--fields 'count,result,sub set,descr,name,log' --uniq ...
## Configuration file
It is possible to use a configuration file with command line option `--config
......@@ -361,6 +361,35 @@ def keep_fields(cross_check, fields):
del x[k]
# Do a "uniq" pass on the data
# All duplicate entries are collapsed into a single one
# We add a "count" field
def uniq(cross_check):
logging.debug("Collapsing duplicates")
# First pass to count all occurences
h = {}
for x in cross_check:
i = ''
for k in sorted(x.keys()):
i += f"{k}:{x[k]} "
if i not in h:
h[i] = {
'count': 0,
h[i]['count'] += 1
# Transform back to list
r = list(h.values())"{len(r)} unique entries")
return r
# Discover fields
# The fields can be supplied as a comma-separated list
# Order is preserved
......@@ -542,6 +571,8 @@ def main():
parser.add_argument('--filter', help='Python expression to filter results')
'--fields', help='Comma-separated list of fields to write')
'--uniq', action='store_true', help='Collapse duplicates')
'log_file', nargs='?', default='sample.ekl',
help='Input .ekl filename')
......@@ -656,6 +687,10 @@ def main():
if args.fields is not None:
keep_fields(cross_check, args.fields)
# Do a `uniq` pass if requested
if args.uniq:
cross_check = uniq(cross_check)
# Auto-discover the fields and take the option into account
fields = discover_fields(cross_check, args.fields)
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment