Skip to content

exekall rework (part 2)

Douglas Raillard requested to merge github/fork/douglas-raillard-arm/_pr27 into next
  • Mostly rework/cleanup the ValueDB where exekall stores the objects that were computed for each "stages" of the test (what is displayed in STRUCTURE file in tests' artifact folder). That DB now has a simple API to allow reloading object instances in a script or notebook. Documentation/examples coming soon.

  • bisector is now integrated with exekall's ValueDB, which can be stored directly inside bisector's report. That allows to remove the xUnit ad-hoc file since bisector can now deal with lisa objects directly (mostly ResultBundle and Result). As a consequence, opening a report using bisector report will likely require the lisa package to be importable, which means lisa should installed as well.

  • exekall learns a exekall compare subcommand that allows comparing two VALUE_DB.pickle.xz files. It will expect multiple values for each test, and look for significant regressions. In order to have more than one value for each test in one DB, use exekall merge or bisector report myreport.yml.gz -oexport-db=merged_db.pickle.xz

  • exekall compare is implemented in term of functions/classes found in lisa/regression.py, which can be readily used for other purposes if needed.

breaking changes:

  • Since xUnit is replaced by ValueDB in bisector's reports, that version won't be able to open previous lisa-next reports. lisa legacy reports should still work fine.
  • ValueDB structure has changed, so reloading previously-generated DB won't work.

note: Since a lot more objects are now stored in bisector's report, the report loading time has increased. Use bisector report --cache to use a pickle cache of the YAML report, which is around ~10 times faster to open.

Merge request reports