Data extracted from logfile scraping - example format

[Jim MacArthur <jim macarthur codethink co uk>]

This is some information Dominic might be able to use as he's writing 
analysis tools, but I thought I'd CC the list in case it's of any 
interest to others. It's the current output from my logfile scraper. 
Since custom logfile formats went into BuildStream recently 
(https://gitlab.com/BuildStream/buildstream/commit/03bdc5cd04646f45baace2e0902690329acb757c) 
it's much easier to extract this information. After scraping, the 
following info ends up in a Python structure (dict of dicts). Here's the 
information as a .CSV; it could go out in JSON, or some other format, if 
desired.

The main new columns are 'id' and 'parent'; since some tasks are owned 
by others, 'parent' points to another 'id' which is the super-task which 
that activity is part of.

id,elapsed,wallclock,key,element,action,message,parent,
0,--:--:--.------,18:03:51,,,START,Loading pipeline,None,
1,0.013667,18:03:51,,,SUCCESS,Loading pipeline,0,
2,--:--:--.------,18:03:51,,,START,Resolving pipeline,None,
3,0.020043,18:03:51,,,SUCCESS,Resolving pipeline,2,
4,--:--:--.------,18:03:51,,,START,Resolving cached state,None,
5,0.08466,18:03:51,,,SUCCESS,Resolving cached state,4,
6,--:--:--.------,18:03:51,,,START,Checking sources,None,
7,0.000141,18:03:51,,,SUCCESS,Checking sources,6,
8,--:--:--.------,18:03:51,,,START,Starting build,None,
9,--:--:--.------,18:03:51,20b0640d,fetch:gnu-toolchain/base.bst,START,root/.cache/buildstream/logs/baserock/gnu-toolchain-base/20b0640d-fetch.53.log,None,
10,--:--:--.------,18:03:51,20b0640d,fetch:gnu-toolchain/base.bst,START,Fetching 
remote ref: 
a9a66b162cb158078cb67e1ba94a8947d1076463f2e5e429caa416f311ac0a01 from 
origin: https://ostree.baserock.org/releases/,9,
11,--:--:--.------,18:03:51,b57571d4,fetch:gnu-toolchain/stage1-binutils.bst,START,root/.cache/buildstream/logs/baserock/gnu-toolchain-stage1-binutils/b57571d4-fetch.55.log,None,
12,--:--:--.------,18:03:51,b57571d4,fetch:gnu-toolchain/stage1-binutils.bst,START,Fetching 
git://git.baserock.org/delta/binutils-tarball,11,
13,--:--:--.------,18:03:52,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 5% (200/3384) 276.6 kB,10,
14,--:--:--.------,18:03:53,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 10% (361/3386) 6.5 MB,10,
15,3.410843,18:03:54,b57571d4,fetch:gnu-toolchain/stage1-binutils.bst,SUCCESS,Fetching 
git://git.baserock.org/delta/binutils-tarball,12,
16,3.414241,18:03:54,b57571d4,fetch:gnu-toolchain/stage1-binutils.bst,SUCCESS,root/.cache/buildstream/logs/baserock/gnu-toolchain-stage1-binutils/b57571d4-fetch.55.log,11,
17,--:--:--.------,18:03:54,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 17% (608/3386) 13.7 MB,10,
18,--:--:--.------,18:03:55,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 25% (852/3386) 19.4 MB,10,
19,--:--:--.------,18:03:56,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 28% (974/3386) 30.5 MB,10,
20,--:--:--.------,18:03:57,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 34% (1164/3386) 45.1 MB,10,
21,--:--:--.------,18:03:58,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 42% (1454/3386) 51.1 MB,10,
22,--:--:--.------,18:03:59,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 51% (1739/3386) 56.0 MB,10,
23,--:--:--.------,18:04:00,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 59% (2016/3386) 62.1 MB,10,
24,--:--:--.------,18:04:01,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 66% (2251/3386) 70.5 MB,10,
25,--:--:--.------,18:04:02,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 70% (2396/3386) 79.0 MB,10,
26,--:--:--.------,18:04:03,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 79% (2697/3386) 83.5 MB,10,
27,--:--:--.------,18:04:04,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 89% (3016/3386) 86.6 MB,10,
28,--:--:--.------,18:04:05,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 97% (3302/3386) 89.9 MB,10,
29,--:--:--.------,18:04:06,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Receiving 
objects: 99% (3385/3386) 95.9 MB,10,
30,--:--:--.------,18:04:07,20b0640d,fetch:gnu-toolchain/base.bst,STATUS,Writing 
Objects,10,
31,21.563527,18:04:12,20b0640d,fetch:gnu-toolchain/base.bst,SUCCESS,Fetching 
remote ref: 
a9a66b162cb158078cb67e1ba94a8947d1076463f2e5e429caa416f311ac0a01 from 
origin: https://ostree.baserock.org/releases/,10,
32,21.566584,18:04:12,20b0640d,fetch:gnu-toolchain/base.bst,SUCCESS,root/.cache/buildstream/logs/baserock/gnu-toolchain-base/20b0640d-fetch.53.log,9,
33,--:--:--.------,18:04:12,20b0640d,build:gnu-toolchain/base.bst,START,root/.cache/buildstream/logs/baserock/gnu-toolchain-base/20b0640d-build.80.log,None,
34,--:--:--.------,18:04:12,20b0640d,build:gnu-toolchain/base.bst,START,Staging 
sources,33,
35,1.395921,18:04:14,20b0640d,build:gnu-toolchain/base.bst,SUCCESS,Staging 
sources,34,
36,--:--:--.------,18:04:14,20b0640d,build:gnu-toolchain/base.bst,START,Caching 
Artifact,33,
37,5.882733,18:04:20,20b0640d,build:gnu-toolchain/base.bst,SUCCESS,Caching 
Artifact,36,
38,7.647511,18:04:20,20b0640d,build:gnu-toolchain/base.bst,SUCCESS,root/.cache/buildstream/logs/baserock/gnu-toolchain-base/20b0640d-build.80.log,33,
39,--:--:--.------,18:04:20,b57571d4,build:gnu-toolchain/stage1-binutils.bst,START,root/.cache/buildstream/logs/baserock/gnu-toolchain-stage1-binutils/b57571d4-build.82.log,None,
40,--:--:--.------,18:04:20,b57571d4,build:gnu-toolchain/stage1-binutils.bst,START,Staging 
dependencies,39,
41,0.237297,18:04:20,b57571d4,build:gnu-toolchain/stage1-binutils.bst,SUCCESS,Staging 
dependencies,40,
42,--:--:--.------,18:04:20,b57571d4,build:gnu-toolchain/stage1-binutils.bst,START,Integrating 
sandbox,39,
43,--:--:--.------,18:04:20,b57571d4,build:gnu-toolchain/stage1-binutils.bst,STATUS,Running 
integration command,42,
44,2.590252,18:04:23,b57571d4,build:gnu-toolchain/stage1-binutils.bst,SUCCESS,Integrating 
sandbox,42,
45,--:--:--.------,18:04:23,b57571d4,build:gnu-toolchain/stage1-binutils.bst,START,Staging 
sources,39,
46,0.883648,18:04:24,b57571d4,build:gnu-toolchain/stage1-binutils.bst,SUCCESS,Staging 
sources,45,
47,--:--:--.------,18:04:24,b57571d4,build:gnu-toolchain/stage1-binutils.bst,START,Running 
configure-commands,39,
48,--:--:--.------,18:04:24,b57571d4,build:gnu-toolchain/stage1-binutils.bst,STATUS,Running 
configure-commands,47,
49,1.125368,18:04:25,b57571d4,build:gnu-toolchain/stage1-binutils.bst,SUCCESS,Running 
configure-commands,47,
50,--:--:--.------,18:04:25,b57571d4,build:gnu-toolchain/stage1-binutils.bst,START,Running 
build-commands,39,
51,--:--:--.------,18:04:25,b57571d4,build:gnu-toolchain/stage1-binutils.bst,STATUS,Running 
build-commands,50,
52,26.023581,18:04:51,b57571d4,build:gnu-toolchain/stage1-binutils.bst,SUCCESS,Running 
build-commands,50,
53,--:--:--.------,18:04:51,b57571d4,build:gnu-toolchain/stage1-binutils.bst,START,Running 
install-commands,39,
54,--:--:--.------,18:04:51,b57571d4,build:gnu-toolchain/stage1-binutils.bst,STATUS,Running 
install-commands,53,
55,0.926899,18:04:52,b57571d4,build:gnu-toolchain/stage1-binutils.bst,SUCCESS,Running 
install-commands,53,
56,--:--:--.------,18:04:52,b57571d4,build:gnu-toolchain/stage1-binutils.bst,START,Running 
strip-commands,39,
57,--:--:--.------,18:04:52,b57571d4,build:gnu-toolchain/stage1-binutils.bst,STATUS,Running 
strip-commands,56,
58,0.65732,18:04:53,b57571d4,build:gnu-toolchain/stage1-binutils.bst,SUCCESS,Running 
strip-commands,56,
59,--:--:--.------,18:04:53,b57571d4,build:gnu-toolchain/stage1-binutils.bst,START,Caching 
Artifact,39,
60,4.564453,18:04:57,b57571d4,build:gnu-toolchain/stage1-binutils.bst,SUCCESS,Caching 
Artifact,59,
61,37.668725,18:04:58,b57571d4,build:gnu-toolchain/stage1-binutils.bst,SUCCESS,root/.cache/buildstream/logs/baserock/gnu-toolchain-stage1-binutils/b57571d4-build.82.log,39,
62,66.89696,18:04:58,,,SUCCESS,Build Complete,8,

At the moment I've got a quick demo which condenses that into the 
following info:

Resolving pipeline: 0.02s
Resolving cached state: 0.08s
Checking sources: 0.00s
Starting build: 66.90s
root/.cache/buildstream/logs/baserock/gnu-toolchain-base/20b0640d-fetch.53.log: 
21.57s
      Fetching remote ref: 
a9a66b162cb158078cb67e1ba94a8947d1076463f2e5e429caa416f311ac0a01 from 
origin: https://ostree.baserock.org/releases/: 21.56s
root/.cache/buildstream/logs/baserock/gnu-toolchain-stage1-binutils/b57571d4-fetch.55.log: 
3.41s
      Fetching git://git.baserock.org/delta/binutils-tarball: 3.41s
root/.cache/buildstream/logs/baserock/gnu-toolchain-base/20b0640d-build.80.log: 
7.65s
      Staging sources: 1.40s
      Caching Artifact: 5.88s
root/.cache/buildstream/logs/baserock/gnu-toolchain-stage1-binutils/b57571d4-build.82.log: 
37.67s
      Staging dependencies: 0.24s
      Integrating sandbox: 2.59s
      Staging sources: 0.88s
      Running configure-commands: 1.13s
      Running build-commands: 26.02s
      Running install-commands: 0.93s
      Running strip-commands: 0.66s
      Caching Artifact: 4.56s

But we may want to slice it up in different ways.