trappy merge requestshttps://gitlab.arm.com/tooling/trappy/-/merge_requests2017-09-25T20:50:48Zhttps://gitlab.arm.com/tooling/trappy/-/merge_requests/269Thermal plotting additions2017-09-25T20:50:48ZDarryl GreenThermal plotting additions*Created by: valschneider*
This PR is about:
- Changing the drawstyle of plotting methods to 'steps-post'
- Adding cdev_state plotting methods for devfreq and cpufreq cooling devices.
- Adding xlim/ylim parameters to new and existing...*Created by: valschneider*
This PR is about:
- Changing the drawstyle of plotting methods to 'steps-post'
- Adding cdev_state plotting methods for devfreq and cpufreq cooling devices.
- Adding xlim/ylim parameters to new and existing plotting methods (mostly to help in my next LISA PR)
This is what the plots look like: https://gist.github.com/valschneider/81a247b6acb69cccfe6e0ad7682914a8#file-thermal_plotting-ipynbhttps://gitlab.arm.com/tooling/trappy/-/merge_requests/277ftrace: fix windowing KeyErrors2018-10-01T12:16:00ZDarryl Greenftrace: fix windowing KeyErrors*Created by: derkling*
For some traces and events (e.g. SchedSwitch) we get a KeyError in case
the first event index is bigger then window[0]. For example:
trace_class: <trappy.sched.SchedSwitch object at 0x7f3afe425f10>
win...*Created by: derkling*
For some traces and events (e.g. SchedSwitch) we get a KeyError in case
the first event index is bigger then window[0]. For example:
trace_class: <trappy.sched.SchedSwitch object at 0x7f3afe425f10>
window: [11793.005501, None]
__comm __cpu __line __pid next_comm next_pid [...]
Time
11793.005507 <...> 0 2 6803 shell svc 6799 6800 [...]
11793.005565 <...> 0 4 6800 adbd 2648 [...]
11793.005701 adbd 0 7 2648 ->transport 2650 [...]
11793.005782 ->transport 0 11 2650 trace-cmd 6803 [...]
11793.005831 <...> 0 17 6803 kworker/u17:0 367 [...]
[...]
KeyError: 11793.005501
Where the min window value [11793.005501] is defined by the first valid event
in the trace. Oddly this does not always happens, expecially it seems to not
happen with trappy.dynamic events.
However, since in these cases the dataframe should not be sliced at all,
let's add some boundary checks to ensure we always provide "valid" slicing
index values. This has been verified to fix the above slicing issue in some
test traces.
Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>https://gitlab.arm.com/tooling/trappy/-/merge_requests/276trappy: Add support for 'print:' ftrace fallback event2018-09-05T10:09:01ZDarryl Greentrappy: Add support for 'print:' ftrace fallback event*Created by: bjackman*
This event is just like tracing_mark_write but presents a small difficulty in
that its name is a Python keyword. This is solved by appending '_' to the
atribute name of the FTrace object.*Created by: bjackman*
This event is just like tracing_mark_write but presents a small difficulty in
that its name is a Python keyword. This is solved by appending '_' to the
atribute name of the FTrace object.https://gitlab.arm.com/tooling/trappy/-/merge_requests/254Add support for trace event processing using callbacks2017-06-29T21:08:14ZDarryl GreenAdd support for trace event processing using callbacks*Created by: cobrien7*
Some trace analysis tasks are most easily expressed by iterating over
events in chronological order. Add a run_custom_analysis() method to
the GenericFTrace class that will run user-defined functions on trace
e...*Created by: cobrien7*
Some trace analysis tasks are most easily expressed by iterating over
events in chronological order. Add a run_custom_analysis() method to
the GenericFTrace class that will run user-defined functions on trace
events in the order they occurred.
Signed-off-by: Connor O'Brien <connoro@google.com>https://gitlab.arm.com/tooling/trappy/-/merge_requests/253Remove raw trace output2017-06-27T10:12:43ZDarryl GreenRemove raw trace output*Created by: credp*
In the quest to speed up trappy parsing, I noticed that we can use the -r option for trace-cmd report so that we can use per-event raw parsing instead of generating a whole new text file.
This set of patches sligh...*Created by: credp*
In the quest to speed up trappy parsing, I noticed that we can use the -r option for trace-cmd report so that we can use per-event raw parsing instead of generating a whole new text file.
This set of patches slightly alters the initialisation of parse objects so that we can have the set of events to use ready before we create the file, and then scans them to find which need raw formatting or not.
Then I use that to generate a single file. The patches look a little scrappy, but it works for me using the
trappy example trace.https://gitlab.arm.com/tooling/trappy/-/merge_requests/267Urgent: Fix parsing when a bad trace.txt exists2017-10-20T13:43:45ZDarryl GreenUrgent: Fix parsing when a bad trace.txt exists*Created by: bjackman*
TRAPpy requires that trace.txt files are created with certain events
printed raw, rather than formated. When given a trace.dat, it
currently creates a trace.txt that fits that requirement, then uses
that for pa...*Created by: bjackman*
TRAPpy requires that trace.txt files are created with certain events
printed raw, rather than formated. When given a trace.dat, it
currently creates a trace.txt that fits that requirement, then uses
that for parsing.
It used to create one trace.txt with non-raw formatting, and a
trace.raw.txt with raw formatting. Therefore it currently uses the
presence of trace.raw.txt as a hint to suggest that the trace.txt
contains non-raw formatting (as created by an older version of
TRAPpy), and raises an error in this case.
However bad (non-raw) trace.txt files may have been created by the
user or other tools such as Workload Automation[1].
Therefore, this patch presents a new approach to finding and creating
input files:
- Ignore trace.raw.text
- When given a trace.dat file, convert it to a trace.trappy.txt
file with all the required events formatted as raw (unless this
has already been done). We can hope that this file is not created
by any other tool.
- When the trace.dat is not present but a trace.trappy.txt is
present, assume that TRAPpy created the trace.trappy.txt and that
the user has since deleted the corresponding trace.dat. Warn, and
continue using trace.trappy.txt.
- If only a trace.txt is present, warn the user (because there are
likely non-raw events in it, in which case we cannot recover and
will crash cryptically), but continue anyway, so that if the user
has manually created a correctly formatted trace.txt then they
can still use that.
[1] https://github.com/ARM-software/workload-automationhttps://gitlab.arm.com/tooling/trappy/-/merge_requests/261cache: Fix issue where single missing CSV deletes all2017-07-14T09:27:14ZDarryl Greencache: Fix issue where single missing CSV deletes all*Created by: joelagnel*
There's a bug in trappy caching where corruption or removal of single CSVs
causes all CSVs to get removed. Fix this and add a test-case to cover it.
Also test that only missing items are regenerated.
Change-...*Created by: joelagnel*
There's a bug in trappy caching where corruption or removal of single CSVs
causes all CSVs to get removed. Fix this and add a test-case to cover it.
Also test that only missing items are regenerated.
Change-Id: Iea41c8e0a005515790b580e7f78f06bdd60141a3
Signed-off-by: Joel Fernandes <joelaf@google.com>https://gitlab.arm.com/tooling/trappy/-/merge_requests/256trappy: 30% speed up goodies !!!2017-06-29T21:12:24ZDarryl Greentrappy: 30% speed up goodies !!!*Created by: joelagnel*
Some speed enhancements, summarizing, altogether giving 30-35% improvement in performance:
* Optimize unique word search.
* Optimize search for array pattern in data.
* Optimize integer conversion in generate_...*Created by: joelagnel*
Some speed enhancements, summarizing, altogether giving 30-35% improvement in performance:
* Optimize unique word search.
* Optimize search for array pattern in data.
* Optimize integer conversion in generate_data_dict.https://gitlab.arm.com/tooling/trappy/-/merge_requests/255trappy/ftrace: add support for boot clock timestamp2017-06-21T19:11:31ZDarryl Greentrappy/ftrace: add support for boot clock timestamp*Created by: derkling*
FTrace can be configured to report events timestamp using different
clock sources which can be selected via the trace_clock sysfs attribute
as described in:
https://www.kernel.org/doc/Documentation/trace/ftra...*Created by: derkling*
FTrace can be configured to report events timestamp using different
clock sources which can be selected via the trace_clock sysfs attribute
as described in:
https://www.kernel.org/doc/Documentation/trace/ftrace.txt
The global clock source reports time in [s] with a [us] resolution.
Other sources instead, like for example the boot clock, uses [ns].
Thus, in these last cases we do not have decimals in the timestamp.
Let's update the special fields regexp to match both [s] and [ns]
formatted times. This also update the base test to add a set of trace
events which are expressed in [ns] resolution.
Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>https://gitlab.arm.com/tooling/trappy/-/merge_requests/252systrace: line test and regex fix2017-06-21T20:11:21ZDarryl Greensystrace: line test and regex fix*Created by: joelagnel*
I promised @derkling I would write this so here you go.
Signed-off-by: Joel Fernandes <joelaf@google.com>*Created by: joelagnel*
I promised @derkling I would write this so here you go.
Signed-off-by: Joel Fernandes <joelaf@google.com>https://gitlab.arm.com/tooling/trappy/-/merge_requests/251trappy: Add a line number column2017-06-14T22:20:10ZDarryl Greentrappy: Add a line number column*Created by: joelagnel*
Useful for joining DataFrames that have timestamp collisions or for iterating
through a group of DataFrames in line order.
Signed-off-by: Joel Fernandes <joelaf@google.com>*Created by: joelagnel*
Useful for joining DataFrames that have timestamp collisions or for iterating
through a group of DataFrames in line order.
Signed-off-by: Joel Fernandes <joelaf@google.com>https://gitlab.arm.com/tooling/trappy/-/merge_requests/249systrace parsing new edition2017-06-14T22:06:31ZDarryl Greensystrace parsing new edition*Created by: joelagnel*
Hi @derkling ,
I went ahead and re-wrote #243. This is very fresh so I am just posting it early for any comments. This approach is more cleaner and also doesn't have issues with #243. It also doesn't require dro...*Created by: joelagnel*
Hi @derkling ,
I went ahead and re-wrote #243. This is very fresh so I am just posting it early for any comments. This approach is more cleaner and also doesn't have issues with #243. It also doesn't require dropping custom strings written by tracing_mark_write that systrace doesn't recognize . I didn't add your test case but I ran some scripts to see it working and existing nosetests are passing. I would appreciate if you can integrate your test case from #243 to work with this PR and feel free to repost this PR with that. We can also close that old PR and work with this one. I borrowed your regex from the systrace format patch and given you credit in the relevant patch, thanks. CC @bjackmanhttps://gitlab.arm.com/tooling/trappy/-/merge_requests/248[RFC] trappy: Add tgid information from systrace2017-06-14T04:37:16ZDarryl Green[RFC] trappy: Add tgid information from systrace*Created by: joelagnel*
Add tgid information if available during parsing traces
Change-Id: I7d280a9dc95d48a2730983e5ddb474d266ebf0fd
Signed-off-by: Joel Fernandes <joelaf@google.com>*Created by: joelagnel*
Add tgid information if available during parsing traces
Change-Id: I7d280a9dc95d48a2730983e5ddb474d266ebf0fd
Signed-off-by: Joel Fernandes <joelaf@google.com>https://gitlab.arm.com/tooling/trappy/-/merge_requests/247trappy: Speed up trappy by caching trace parsing2017-06-29T20:53:57ZDarryl Greentrappy: Speed up trappy by caching trace parsing*Created by: joelagnel*
Pandas is extremely fast at parsing csv to data frames. Astonishingly it takes
< 1s to serialize/deserialize a 100MB work of traces with 430000 events to/from
csv. We leverage this and write out a data frames i...*Created by: joelagnel*
Pandas is extremely fast at parsing csv to data frames. Astonishingly it takes
< 1s to serialize/deserialize a 100MB work of traces with 430000 events to/from
csv. We leverage this and write out a data frames into a csv file when they are
created for the first time. Next time we read it out if it exists. To make
sure, the cache isn't stale, we take the md5sum of the trace file and also
ensure all CSVs exist before reading from the cache. I get a speed up of 16s to
1s when parsing a 100MB trace.
Signed-off-by: Joel Fernandes <joelaf@google.com>https://gitlab.arm.com/tooling/trappy/-/merge_requests/241[RFC] trappy: add support for event callbacks2017-07-08T17:03:28ZDarryl Green[RFC] trappy: add support for event callbacks*Created by: joelagnel*
This series adds event callbacks support to trappy so that trappy users can register callbacks for trace events, that can be used to drive a state machine to do some analysis.
Example analysis script using thi...*Created by: joelagnel*
This series adds event callbacks support to trappy so that trappy users can register callbacks for trace events, that can be used to drive a state machine to do some analysis.
Example analysis script using this is here:
https://github.com/joelagnel/lisa/blob/40047eab82ed43152ef9d587ba89c5145cdb5d5b/get_wlat_cbs.py
(disclaimer, the example script in the above link is very wip and needs to cleaned up and polished properly)https://gitlab.arm.com/tooling/trappy/-/merge_requests/230Constraint: Fix __str__ for non-string column keys2016-11-04T10:44:09ZDarryl GreenConstraint: Fix __str__ for non-string column keys*Created by: bjackman*
Pandas DataFrame columns may be named with any hashable object.
I haven't changed the code for when the data comes from a Trappy trace. I don't think it's necessary (because self.column is always a string - rig...*Created by: bjackman*
Pandas DataFrame columns may be named with any hashable object.
I haven't changed the code for when the data comes from a Trappy trace. I don't think it's necessary (because self.column is always a string - right?) and I haven't figured out how to test it.Javi MerinoJavi Merinohttps://gitlab.arm.com/tooling/trappy/-/merge_requests/229Add fill_alpha param to ILinePlot2016-11-06T16:22:11ZDarryl GreenAdd fill_alpha param to ILinePlot*Created by: bjackman*
Example use case: https://gist.github.com/bjackman/559553d565b033ad3cd7799bf9f9c0de
(Javascript doesn't get run from Gists so you'll need to download it)*Created by: bjackman*
Example use case: https://gist.github.com/bjackman/559553d565b033ad3cd7799bf9f9c0de
(Javascript doesn't get run from Gists so you'll need to download it)Javi MerinoJavi Merinohttps://gitlab.arm.com/tooling/trappy/-/merge_requests/245ftrace: Improve error messages when failing to parse trace2017-05-15T12:59:55ZDarryl Greenftrace: Improve error messages when failing to parse trace*Created by: bjackman*
*Created by: bjackman*
https://gitlab.arm.com/tooling/trappy/-/merge_requests/213ILinePlot: Performance Improvements for ILinePlot v32016-08-20T18:46:44ZDarryl GreenILinePlot: Performance Improvements for ILinePlot v3*Created by: sinkap*
Newer version for #212 solving issue #103
*Created by: sinkap*
Newer version for #212 solving issue #103
Javi MerinoJavi Merinohttps://gitlab.arm.com/tooling/trappy/-/merge_requests/212ILinePlot: Performance Improvements for ILinePlot v22016-08-20T18:46:46ZDarryl GreenILinePlot: Performance Improvements for ILinePlot v2*Created by: sinkap*
v2 for #207 and fixing issue #103
*Created by: sinkap*
v2 for #207 and fixing issue #103