parser.py: add an option to collapse duplicates

This is similar in its principle to the UNIX `uniq -c' and is handy for
triaging results, in conjunction with the `--fields' option.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

README.md

@@ -78,6 +78,22 @@ $ ./parser.py --fields 'result,sub set,descr,name,log' ...
 
 The csv format can retain the fields order.
 
+### Collapsing duplicates
+
+It is possible to "collapse" duplicate tests data into a single entry using the
+`--uniq` option, much similar in principle to the UNIX `uniq -c` command.
+
+This step happens after tests and fields filtering, and it adds a "count" field.
+
+Example command, suitable for triaging:
+
+``` {.sh}
+$ ./parser.py \
+      --filter "x.update({'log': re.sub(r'/.*/', '', x['log'])}) \
+                or x['result'] != 'PASS'" \
+      --fields 'count,result,sub set,descr,name,log' --uniq ...
+```
+
 ## Configuration file
 
 It is possible to use a configuration file with command line option `--config

parser.py

@@ -361,6 +361,35 @@ def keep_fields(cross_check, fields):
                 del x[k]
 
 
+# Do a "uniq" pass on the data
+# All duplicate entries are collapsed into a single one
+# We add a "count" field
+def uniq(cross_check):
+    logging.debug("Collapsing duplicates")
+
+    # First pass to count all occurences
+    h = {}
+
+    for x in cross_check:
+        i = ''
+
+        for k in sorted(x.keys()):
+            i += f"{k}:{x[k]} "
+
+        if i not in h:
+            h[i] = {
+                'count': 0,
+                **x,
+            }
+
+        h[i]['count'] += 1
+
+    # Transform back to list
+    r = list(h.values())
+    logging.info(f"{len(r)} unique entries")
+    return r
+
+
 # Discover fields
 # The fields can be supplied as a comma-separated list
 # Order is preserved
@@ -542,6 +571,8 @@ def main():
     parser.add_argument('--filter', help='Python expression to filter results')
     parser.add_argument(
         '--fields', help='Comma-separated list of fields to write')
+    parser.add_argument(
+        '--uniq', action='store_true', help='Collapse duplicates')
     parser.add_argument(
         'log_file', nargs='?', default='sample.ekl',
         help='Input .ekl filename')
@@ -656,6 +687,10 @@ def main():
     if args.fields is not None:
         keep_fields(cross_check, args.fields)
 
+    # Do a `uniq` pass if requested
+    if args.uniq:
+        cross_check = uniq(cross_check)
+
     # Auto-discover the fields and take the option into account
     fields = discover_fields(cross_check, args.fields)