[BuildStream] Weekly progress report - with an important request

[James Ennis <james ennis codethink co uk>]

Hello,

I've appended a line of results to the tables that I posted to the list 
last week, and also attached the notebook.

IMPORTANT REQUEST:
It looks as if the landing of source cache [0] has caused a rather large 
slow down for the (fresh) 'build' of the import project. This only seems 
to have affected the scheduler, as `bst show` times (pre-scheduler) have 
remained unchanged, I'd imagine this is down to the addition of two 
extra queues.

After some discussion between myself and Ben today, to avoid this 
happening in the future perhaps we should, as MR authors, benchmark our 
changes locally and comment these on the MR (for larger MRs, I do not 
think this is required for one liners or doc changes...). For now, Ben 
has created a simple benchmarking tool, here: 
https://gitlab.com/BenjaminSchubert/bst-benchmarks

Simply clone the repo and run:

    tox -e benchmark -- --python-version py36 BRANCH_TO_COMPARE

NOTE: BRANCH_TO_COMPARE must be pushed to the remote beforehand, if you 
only have your branch locally, this will not work.
NOTE2: py35 or py37 for Python 3.5 and 3.7, respectively

This script will default to cloning the debian-like project [1] and 
benchmarking `bst show` (with and without yaml cache) and `bst build` of 
base-files/base-files.bst, which results in a pipeline of ~6000 (import) 
elements. Note that a 'build' of ~6000 elements takes a while, so this 
will probably need to be run over a coffee break/lunch.

See the options (`./benchmark.py --help`) if you wish to benchmark 
another project or benchmark with more builders etc.

Now, this is something that we, as developers, could start trying to do 
immediately, but obviously there is no strict requirement (for now) to 
do this. Therefore, in the near future, I intend to kick off a mailing 
list discussion regarding the potential benchmarking of a project in 
BuildStream's CI to gather people's thoughts are on the matter.

See the linked ML post ([2]) if you're struggling to open the notebook 
(the graphs have some fun interactive features).

Results below:

Averages of the show and build times of base-files/base-files.bst [1]:

| Date       |Show (s)| Show with cache (s)| Build (s)| Show once built 
(s)|
|------------|--------|--------------------|----------|--------------------|
| 21/01-27/01| 46.69  | 27.40              | 378.58   | 33.65            
  |
| 28/01-03/02| 46.36  | 27.35              | 383.83   | 33.49            
  |
| 04-02-10/02| 45.72  | 26.53              | 268.70   | 32.68            
  |
| 11/02-17/02| 43.07  | 24.04              | 228.94   | 30.13            
  |
| 18/02-24/02| 40.87  | 22.35              | 228.72   | 28.56            
  |
| 25/02-03/03| 39.33  | 19.85              | 240.00   | 26.44            
  |
| 04/03-10/03| 38.84  | 19.03              | 238.12   | 25.86            
  |
| 11/03-17/03| 38.00  | 18.31              | 301.37   | 25.29            
  |

Averages of the maximum memory usage for the show and build of 
base-files/base-files.bst:

| Date       |Show (M)| Show with cache (M)| Build (M)| Show once built 
(M)|
|------------|--------|--------------------|----------|--------------------|
| 21/01-27/01| 1084   | 1108               | 1109     | 1110             
  |
| 28/01-03/02| 1083   | 1108               | 1109     | 1110             
  |
| 04-02-10/02| 556    | 581                | 582      | 582              
  |
| 11/02-17/02| 403    | 425                | 425      | 426              
  |
| 18/02-24/02| 410    | 424                | 425      | 425              
  |
| 25/02-03/03| 493    | 514                | 515      | 619              
  |
| 04/03-10/03| 519    | 542                | 543      | 695              
  |
| 11/03-17/03| 521    | 544                | 545      | 697              
  |


Thanks,
James

Attachment: weekly_results.ipynb
Description: Text document