Re: Source distribution, WAS: Re: Proposal for Remote Execution

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Thu, 2018-06-14 at 23:14 +0200, Sander Striker wrote:
[...]
> > If the client side only solution for source mirroring works well
> > enough
> > without too much fuss then we don't really need the `bst mirror`
> > service.
> >
> > The problem however is that it's a pain point for the GNOME build
> > experience right now, we need it solved in one way or another.
>
> There are some very specific things in the GNOME build setup
> apparently :)  Can we aim to tackle a concrete example?

I didn't want to get into detail but sure: GitLab.

The thing is when you try to run builds on GitLab, you get temporary
"caches", when these caches get zapped or replaced, we need to download
the sources again.

Compare this to two scenarios:

  o Using flatpak-builder to build the GNOME SDK

    Here we *would* have the same problem, but we suffer it only every
    time we setup a new dedicated SDK builder machine, which persists
    its own source cache.

  o Using JHBuild as a developer

    Here we have to ensure access to a lot of things which were not
    necessary to build when using JHBuild, since JHBuild just builds
    against your host and expects your host to take care of installing
    most dependencies.

So the specific problem of upstreams suddenly going missing is
currently sorely felt by the GNOME build use cases.

> > I mostly want to keep the door open for `bst mirror` to materialize
> > if
> > it does, because it has the potential of being more convenient to
> > BuildStream users, and a lot of the code needed to get it done is
> > already in BuildStream.
>
> On the convenience I'm personally not sold.  Essentially this is a
> setup step [of actual mirror(s)] and a configuration in project.conf
> for the owner of the project.  For the majority of the users it will
> be completely transparent.

I think we're not talking about the initial hurdle of setting it up
(although that can also be a bonus), its rather the constant
development pipeline overhead for getting something new to be mirrored,
while not mirroring new versions of things which you don't need in
permanence.

This extra maintenance of explicitly saying "Now we need to mirror
libfoo" sometimes even manifests itself as a separate thing which
either needs a review process, or requires a different set of
permissions, and we should be able to automate what gets mirrored based
on what you are using.

Project based repository mirroring (let's call it), could however also
be achieved by using a script which reads a BuildStream project and
informs the third party what to mirror (possibly a script that is
triggered by the mirroring solution itself at the time it decides it's
time to start mirroring things).

Cheers,
    -Tristan