Re: Discussion on source mirroring

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Thu, 2018-03-15 at 16:30 +0000, Jonathan Maw wrote:
> I've been giving some thought on source mirroring, recently, after 
> reading the discussion at 
> https://gitlab.com/BuildStream/buildstream/issues/179.
>
> Source mirroring will be valuable for us because:
> * The canonical upstream may disappear without warning
> * The canonical upstream may be slow to access due to limited 
> infrastructure or geographical distance.

Much discussion has been had, replying back to the top of the thread.

In the midst of plotting out a longer term roadmap, I've spent some
time this afternoon spec'ing this out in two separate issues, such that
they are compatible and do not interfere with eachother (i.e. both an
automated and convenient approach marries nicely with interoperability
with external solutions).


  Support for downloading sources from mirrors
  https://gitlab.com/BuildStream/buildstream/issues/328


  Support for mirroring of upstream sources
  https://gitlab.com/BuildStream/buildstream/issues/330


To simplify things, I should note, quoting from issue 328 above:

  The following draft makes the assumption that a given source alias
  can should be mirrored as a single unit; which makes the
  implementation a bit simpler, while imposing that a project.conf
  declare aliases in such a way that matches the chosen mirroring 
  solution.

  This is to say, if you have a lot of git repositories in the 
  same upstream location, but only wish to mirror a portion of those
  git repositories, you must use separate source aliases to refer to
  them in your project.

Perhaps it's not worded perfectly, but I think you get the picture, we
make things a little bit simpler by allowing granularity only at the
"alias" level, but never on a per source basis, with a tradeoff that
you must specify mirrors in your project accordingly, grouping things
by which sources you want to mirror together.

Besides this point, the data structures of a "mirror" as well as the
original proposal for `bst mirror` and `Source.mirror()` has evolved a
little bit so as to properly account for cases where an upstream has
modified it's sources in an incompatible way (such as tarball
modifications, or git history rewrites).

The drafts in those issues may not be perfect and may need some minor
changes, some of which can only be discovered in implementation.

Comments on the issues, or here on the mailinglist, are always welcome.

Cheers,
    -Tristan