Re: Source distribution, WAS: Re: Proposal for Remote Execution

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Thu, 2018-06-07 at 14:36 +0100, Sander Striker wrote:
[...]
> > We could call it something different than "cache".  And the design
> for persistence is ours to decide.  That said I don't think we disagree.
>  
> > As you highlight above, even if we did go the extra mile to ensure that
> > the Source Cache is persistent for the sources we will need, there are
> > still places where we expect the original format to be available, which
> > mirrors should ensure (like workspaces).
>
>  
> > Basically, I think that "trusting the intermediate source cache
> > designed for remote execution to consequently solve source mirroring"
> > is the wrong way to think about this,
>
> I would paraphrase it slightly differently, as "considering to alter the
> design of the source caching to be able to trust it for a subset of
> source mirroring".
>  
> > we are probably looking at two
> > potentially useful, but separate things:
> >
> >   o Sharing the SourceCache
> >
> >     - This is still just a "cache"
> >     - This contains only the unpacked/staged sources
> >     - Sharing this means I dont need the whole git locally,
> >       if a shared source cache already has exactly this
> >
> >   o Source Mirroring
> >
> >     - I don't worry about third party sites going missing
> >     - I can reproduce this forever, until I delete my mirror(s)
> >     - I probably want the original sources in their original formats
>
> I think that is a reasonable summary.  I am still wondering if a generic
> mirroring solution is needed as a core part of BuildStream.  Solving
> this for git is a known problem, and if you need to set up an endpoint
> on the mirror anyway...  Solving this for tarballs, is also a known
> (e.g. an immutable caching proxy).  For setting up Subversion mirrors
> the same applies.
> Ultimately it's a trade off of whether we want to have this in scope,
> considering complexity and maintenance 
>  
> > I don't think that Sharing a SourceCache is really going to solve the
> > problem of Source Mirroring,
>
> Right, it has the potential to solve a subset of problems that source mirroring
> was initially set to solve.  Specifically, setting up geographically close mirrors for
> either latency or scalability.  Basically what you break down in Source Cache
> and Source Mirror above.

Right so, we had a bit of a chat on IRC and I just wanted to touch base
on list about this...

Basically:

  o The SourceCache by itself will not allow us to really check the box
    that "Source Mirroring is supported" (while it does satisfy a
    critical subset of those requirements).

  o The client side of source mirroring is still interesting, as it
    assumes external mirroring solutions exist and just allows
    BuildStream to use them.

  o I'd rather not start making changes for SourceCache to be a robust
    mirror for the subset of features it can provide, until after we
    have client side source mirroring solved.

    This means, the SourceCache can remain a cache, and we probably
    wont want to change it once we have real mirroring already covered.

    Interestingly, SourceCache can *still* act as a shared cache for
    sources, and possibly optimize the builds in this way.


So, let's not give SourceCache this mirroring burden just yet, lets
finish at least the client side of mirroring first and then re-assess
if we still want that, and why.


On a related topic; the last part of this is `bst mirror`, where does
this fit in ?

We need client side mirroring to work with some suitable third party
F/OSS mirroring software project first, and have some user facing
instructions of how BuildStream users can use it... if we cannot get
there, that is when we really *need* `bst mirror`.

If the client side only solution for source mirroring works well enough
without too much fuss then we don't really need the `bst mirror`
service.

The problem however is that it's a pain point for the GNOME build
experience right now, we need it solved in one way or another.

I mostly want to keep the door open for `bst mirror` to materialize if
it does, because it has the potential of being more convenient to
BuildStream users, and a lot of the code needed to get it done is
already in BuildStream.

Cheers,
    -Tristan