It hasn't worked for a long time

The current python packaging user guide suggests a simpler way of
building packages. Adopt it and simplify building of the package.

Since python 3.5, the high-level API recommends using subprocess.run() instead
of call(), check_call() and check_output().  Use the new API.

ftrace: fix endtime

Make sure endtime is populated even if no events need to be parsed.

ftrace: Add support to parse function tracing ('function' tracer)

The non-raw version of this event is of the shape

    "CALLEE <-- CALLER"

which trappy obviously doesn't like. Force this event to be parsed raw,
which gives us:

      ip=<addr> parent_ip=<addr>

doc: Update Dynamic traces

ftrace: Only add dynamic classes on non-custom scopes

Stop recommending register_dynamic_trace, and use events

We already have basetime, let's add endtime. This allows getting a stable
end time, regardless of what even is being parsed, since that's necessary for a
number of time based analysis.

Redirect the output of trace-cmd straight to a file rather than keeping it in
memory before dumping.

Custom scope is supposed to give only the events that are explicitly listed in
`events`. Therefore, avoid adding dynamic events in this case, to speed up
parsing and avoid consuming memory.

ftrace: Add support for funcgraph_entry and funcgraph_exit events

These events are generated by the function_graph tracer on function entry and
exit.

Signed-off-by: Douglas RAILLARD <douglas.raillard@arm.com>

add support to parse syscall traces

ftrace: Ensure timestamp uniqueness

Call trace-cmd report with "-t" option, so that the full available resolution is
used. This reduces the number of conflicting timestamps, but does not completely
avoid them.

The timestamp of consecutive events are sometimes the same, if they were emitted
in a very short interval of time. This becomes an issue when the events are
dispatched in separate dataframes, which then sometimes need to be joined back
together in client code. At this point, the ordering between events can be lost,
leading to wrong results when exact ordering is required.

In order to fix that, pick the next representable floating value when there is
an ambiguity. This ensures that the delta added is the smallest possible, which
is around the 16th least significant digit.

trace/base: Optimizing DataFrame memory footprint

Under the hood pandas represents numeric values as NumPy ndarrays and
stores them in a continuous block of memory. Values of the same column
are represented using the same type and thus number of bytes.

Many types in pandas have multiple subtypes that can use fewer bytes to
represent each value. For example, the float type has the float16,
float32, and float64 subtypes.

Use the function pd.to_numeric() to downcast numeric types to use for
each value the minumum number of bytes which is still enough to
represent the maximum value for a given column.

Use also "Categoricals" introduced in Pandas since version 0.15.
The category type uses integer values under the hood to represent the
values in a column, rather than the raw values. Use category to
efficiently compress the representation of string values by replacing
64bit string pointers with an index using less bits.

Credits goes to:

   Using pandas with Large Data Sets
   https://www.dataquest.io/blog/pandas-big-data/



where these changes are proposed and discussed in details.

The proposed change applied to a 473M example trace gives the following
results:

                   | Events |  Memory (MB)  |  Compression
                   |  count | Before  After |      percent
  -----------------+--------+ --------------+-------------
   clock_disable   |  18368 |   3.34   0.88 |    73.652695
   clock_enable    |  19149 |   3.43   0.92 |    73.177843
   clock_set_rate  |  42099 |   7.63   2.01 |    73.656619
   cpu_idle        | 272726 |  28.87  12.74 |    55.871147
   sched_switch    | 315951 |  82.86  23.26 |    71.928554

Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>

base.py: handle any duplicated indices when creating dataframe

Since we handle this at the dataframe cration this call is redundant
now.

Signed-off-by: Qais Yousef <qais.yousef@arm.com>

When running Lisa tests on fastmodels duplicated timestamps happen
frequently and cause some failures down the line.

See https://github.com/ARM-software/lisa/pull/864

 for more details.

The duplicated timestamps are not bad themselves and a side effect of
simulated time not having good enough resolution.

Handling this duplication at the dataframe creation seems to be the most
generic and future proof solution.

The change breaks the test_idle.py because it has a duplicate entry in
its test vector and checks for an exact match. Since we know now that
duplicates will be removed when creating the dataframe, we modify the
test to remove the duplicate value.

Signed-off-by: Qais Yousef <qais.yousef@arm.com>

To ensure that we don't mess up the ordering of the events when fixing
up the duplicate indices, sort the values first by 'Time' (which is the
index) and then '__line' columns.

If '__line' column doesn't exist then we carry on anyways assuming we
can't do much preserving the original behavior of the function.

Signed-off-by: Qais Yousef <qais.yousef@arm.com>

Hex conversion downtuning

It doesn't depend on any internal data, and this lets us test it
without creating a Base instance.

Conversion of hexadecimal-containg strings was fixed in

  commit 9eca88cf ("base: Fix string to int conversion")

but was made slightly overzealous. An event field that really is a
string but that is a valid hexadecimal value without leading "0x" is
currently converted to an int.

Turns out that passing base=0 to int() means to let python figure out
the base from the shape of the string, so let's do that instead of
trying to be too smart for our own good.

tests/test_ftrace: Fix test_get_all_freqs_data()

In Python2, iterating over a dict's keys will always give the same
sequence (although the ordering is arbitrary). In Python3 that
ordering is still abitrary, but to make things even better it can
change from one execution to another - see [1].

GenericFTrace.get_all_freqs_data() returns a list which's ordering
depends on the keys of the input dictionnary. That in itself would
need to be changed to a better design, but for the time being we can
fix the test and make Trappy pass reliably by converting the output
of get_all_freqs_data() to a dict().

[1]: https://docs.python.org/2/using/cmdline.html#cmdoption-r

base: Fix string to int conversion

Because it relied on str.isdigit(), Base.string_cast() would
erroneously assume a hex string (e.g. "0x42") wouldn't be castable to
int.

Use a more robust pattern using try/except blocks. This will happily
cope with negative and hex values.

While at it, rename string_cast() to string_cast_int(), since the
former only handles ints and does an ugly empty return when the
specified type isn't int.

trappy: utils: Fix get_duplicates FutureWarning

We've had this one for a few months now:

trappy/utils.py:78: FutureWarning: 'get_duplicates' is deprecated and will be removed in a future release. You can use idx[idx.duplicated()].unique() instead
  dups = index.get_duplicates()

Do what the warning says. Also, bump up the required pandas version to
one that can use the new expression.