Hello,This week's weekly performance update will be much more verbose than previous posts as I'd like to take the time to talk about the changes we've made to the project we have been using to generate the benchmarks data and I will also try to mention some of the interesting trends/observations.
So, we have made a tweak to the Debian-like [0] project. The master branch of this repo has remained unchanged, ~75,000 *local* import elements whose dependency relationship reflects that of the source and binary packages in Debian-stretch. However, there now exists the branch: "jennis/use_remote_file" [1], which uses "remote" imports (with file urls) of uniquely generated files (each of size 4.0K and located in the project's files/ directory). These files are generated by the ./prepare_project.sh script on that branch.
Note that we are still testing a subset of this project - base-files/base-files.bst is the target which produces a pipeline of ~6000 elements.
The reason we back-generated all the data for an altered project was because Raoul's source-cache MR (!1124) [2] reminded us that local imports were not being fetched and staged into the local cache (pre-!1124), so we saw a spike in build times from when this landed. Therefore, in order to more accurately simulate a 'real-world' project, we altered the Debian-like project to use 'remote' file imports.
In addition to this, we are now also measuring the build times and peak memory usage for 4, 8 and 12 builders.
-----All results were obtained on the same hardware (a Codethink dev machine configured to be a gitlab runner - not being used for anything else), and the results files are being appended to by a CI job. Specs as follows:
* Linux (Debian stable) * x86_64 * 16G RAM * 500Gb SSD * Intel i7-3770 * 8 cores @ 3.40 GHz -----Now, bearing in mind all the results have been completely regenerated, here are the times and max-rss (peak memory usage) results for a `bst show` of base-files/base-files.bst in the Debian-like project (jennis/use_remote_file branch) [1] (Note that I've set the column width of the table to 72 in the hope that this will wrap nicely for everyone):
┌─────┬─────────────────┬─────────────────┬───────────────────────────┐ │ │ Show │ YAML cache show │ Show once built │ ├─────┼───────┬─────────┼───────┬─────────┼─────────────────┬─────────┤ │Dates│Time │Memory │Time │Memory │Show once built │Memory │ │ │(s) │(M) │(s) │(M) │(s) │(M) │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │01/14│49.23 │1089 │27.17 │1115 │32.82 │1117 │ │01/20│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │01/21│48.85 │1089 │27.35 │1115 │32.84 │1117 │ │01/27│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │01/28│49.50 │1088 │27.17 │1115 │32.76 │1117 │ │02/03│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │02/04│48.08 │561 │26.68 │588 │31.90 │589 │ │02/10│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │02/11│45.46 │408 │23.75 │435 │29.50 │437 │ │02/17│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │02/18│43.17 │417 │21.65 │443 │27.28 │445 │ │02/24│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │02/25│41.75 │493 │19.43 │519 │25.69 │625 │ │03/03│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │03/04│40.96 │517 │18.81 │543 │25.20 │696 │ │03/10│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │03/11│40.44 │519 │18.06 │545 │24.50 │698 │ │03/17│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │03/18│38.87 │521 │16.74 │547 │23.19 │696 │ │03/24│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │03/25│22.63 │422 │13.40 │433 │17.36 │563 │ │03/31│ │ │ │ │ │ │ └─────┴───────┴─────────┴───────┴─────────┴─────────────────┴─────────┘ # Interesting observations * On the whole, times and peak memory usage are decreasing.* Our peak memory has not been consistently decreasing. This is due to the fact that we have made a couple of time vs memory trade offs, favouring a reduction in `show` time. The most notable culprit here being !1164 [4]: Altering the behavior of the garbage collector - in favor of a faster show. - This is very noticeable in the attachment 'all_show_memory_results.png', there is a spike in memory near the beginning of March. - This corresponds to a dip in the time taken in the 'all_show_times_results.png' attachment.
* Please do not be alarmed by the fact that the most recent `show` is much higher than the most recent `show with YAML cache`, even though we no longer support the YAML cache, YAML New World Order (!1257) [4] landed mid-week last week, so the average has been calculated with results from pre-!1257.
* Our peak memory usage for `bst show` once the artifact cache is hot is much higher than for all other shows.
-----Here are the times and max-rss results for a `bst build` of base-files/base-files.bst in the Debian-like project [1]
┌─────┬─────────────────┬─────────────────┬───────────────────────────┐ │ │ 4 Builders │ 8 Builders │ 12 Builders │ ├─────┼───────┬─────────┼───────┬─────────┼─────────────────┬─────────┤ │Date │Time │Memory │Time │Memory │Show once built │Memory │ │ │(s) │(M) │(s) │(M) │(s) │(M) │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │01/14│1383.67│1119 │1078.76│1119 │735.41 │1119 │ │01/20│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │01/21│561.12 │1117 │602.20 │1117 │680.10 │1117 │ │01/27│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │01/28│560.7 │1117 │601.57 │1117 │678.03 │1117 │ │02/03│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │02/04│373.29 │590 │425.09 │589 │500.71 │590 │ │02/10│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │02/11│308.75 │436 │357.69 │436 │412.90 │436 │ │02/17│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │02/18│292.78 │445 │350.70 │445 │398.30 │445 │ │02/24│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │02/25│342.28 │521 │387.28 │521 │436.51 │521 │ │03/03│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │03/04│354.64 │544 │400.00 │544 │441.61 │544 │ │03/10│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │03/11│352.95 │546 │395.50 │546 │442.52 │546 │ │03/17│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │03/18│348.53 │549 │383.93 │549 │431.18 │549 │ │03/24│ │ │ │ │ │ │ ├─────┼───────┼─────────┼───────┼─────────┼─────────────────┼─────────┤ │03/25│286.67 │435 │302.77 │435 │332.87 │435 │ │03/31│ │ │ │ │ │ │ └─────┴───────┴─────────┴───────┴─────────┴─────────────────┴─────────┘ # Interesting observations* Both peak memory usage and time have decreased dramatically since mid-Jan
* From late Feb/early March, our build times increased again by ~60 seconds and our peak memory by ~100M... Upon investigation, we found that !1164 [4]: Altering the behavior of the garbage collector - in favor of a faster show. was responsible for this. - There is an obvious spike which shows this in the 'all_build_time_results.png' attachment. - This also has a corresponding spike in memory usage, see the 'all_build_memory_results.png' attachment. - So the discussion needs to be had whether this commit is worth reverting.
* The most notable of observations is in mid-January, where our build times were very high, but, 4 builders were slower than 8 builders which were slower than 12 builders, as one might expect. Then, *all* of the build times were heavily reduced by Tristan's change to the scheduler: !1067 - Fix element processing order [4]. Now, rather interestingly, all times are significantly lower but 12 builders are consistently slower than 8 builders which are consistently slower than 4 builders. So the behavior has flipped, and is perhaps now behaving non-intuitively.
- See the all_build_time_results.png attachment for clarity.
- This suggests that if I tried 24 builders, for example, this patch
would have likely made my build slower.
-----As usual, the notebook displaying the results for last week's data has been attached. Please see my previous email [5] if you're unable to open this. If you're comparing this to previous weeks, just bear in mind that the data has been regenerated. I've also attached a compressed tarball that contains all of the results. This might be of more interest.
I hope that you found this brief overview of these benchmarks helpful.It's great to see that, in general, we seem to be heavily improving BuildStream's performance!
Thanks, James [0] https://gitlab.com/jennis/debian-stretch-bst[1] https://gitlab.com/jennis/debian-stretch-bst/tree/jennis/use_remote_file
[2] https://gitlab.com/BuildStream/buildstream/merge_requests/1124 [3] https://gitlab.com/BuildStream/buildstream/merge_requests/1257 [4] https://gitlab.com/BuildStream/buildstream/merge_requests/1067[5] https://mail.gnome.org/archives/buildstream-list/2019-February/msg00046.html
Attachment:
all_build_memory_results.png
Description: PNG image
Attachment:
all_build_time_results.png
Description: PNG image
Attachment:
all_show_memory_results.png
Description: PNG image
Attachment:
all_show_time_results.png
Description: PNG image
Attachment:
all_results.tar.gz
Description: GNU Zip compressed data
Attachment:
weekly_results.ipynb
Description: Text document
