Re: [BuildStream] Adding benchmarking to buildstream CI

[Lachlan Mackenzie <lachlan mackenzie codethink co uk>]

[snip]

On 2019-07-11 12:21, Darius Makovsky wrote:
> On 10/07/2019 17:18, Lachlan Mackenzie via buildstream-list wrote:
> > [...]
> > The solution will not add anything to the buildstream pipeline, but 
> > will
> > utilize a "benchmarking bot"
>
> Where would this be hosted and would this incur additional cost?

This is tbd, but it is a broader question that will be considered with 
marge bot.

> > [...]
> > which will need to be set by a user when a particular MR branch needs 
> > to
> > be
> > benchmarked.
>
> I think, if this is adopted, we should discuss which branches need to
> auto-labelled.

Some control process will be required that takes into account review 
process and procedure. It is expected that benchmarking will only take 
place when a functionally complete and agreed solution is in place (not 
just based on satisfactory pipeline completion). If benchmarking shows 
significant performance issues it is possible that the buildstream 
solution would need to be reconsidered and benchmarking run again, but 
only once review has completed.

> > [...]
> >    - The bot would trigger a curl build request to benchmarking gitlab 
> > CI
> > to
> >      carry out benchmarking run on the MR branch which has the oldest
> > tagged
> >      "benchmarking" request whose buildstream pipeline has completed
> >      successfully.
>
> Would there be any way to manually influence the queue? Will this
> benchmarking then block merges until it completes (whether that's a
> policy or enforced)?

Explicitly - this will not block Buildstream merges, it is a non gate 
indicative test only. If it shows issues it will not prevent merge, but 
might prompt further discussion only.

> > [...]
> > It is expected that developers will pick up generated discussion 
> > points
> > and
> > review results via the benchmarking gitlab ci, rather than adding them
> > to the
> > discussion verbosely.
>
> It might make more sense to just send them to the MR directly so devs
> only have to look there.

These will be available to all people who have interest in the MR and 
hence will be added as part of the discussion. Any point added can be 
closed down without any blocking behavior holding things back.