Skip to content

base: make parsed_data a generator

Congregate Migrate requested to merge github/fork/JaviMerino/improve_mem into master

With big traces, the parsed_data array becomes huge (up to gigabytes). parsed_data is only used as a parameter for pandas.DataFrame. pandas accepts a generator instead of a list. Therefore instead of creating a huge array for pandas, pass a generator to pandas and let it process the incoming information as it needs to.

Before:

init-before

After:

init-after

As far as I can tell, .0 and data are internal pandas variable and thus outside of our control.

Merge request reports