[BuildStream] Proposal: Change how subprojects interact with parent project remotes and add extra configuration

[James Ennis <james ennis codethink co uk>]

Hi all,

TL;DR
Currently, BuildStream master will pull/push subproject artifacts
from/to the parent project remote(s) (remotes which have been declared
in a project's project.conf).

This behaviour is different to what was originally intended as the
default behaviour for subprojects and, when introduced, it was 
introduced
without being tested/documented/announced.

I propose that we revert this "new" behaviour and add configuration
which enables it and I would like to hear people's thoughts.


## Context to the problem
After having briefly discussed this with Jürg, it seems that the
*intended* default behaviour was that project-specific remotes should
indeed be project-specific. Although, due to a previous bug in
master, this was not the case. This was fixed by 1ee4a4ba [0] which was
introduced as part of !1313 [1]. This commit *alone* reinforced the
intended behaviour - meaning that subproject artifacts would not
be pulled/pushed from/to a parent project's remote(s).

However, within the same MR (!1313 [1]), the second commit (24c0de16
[2]) ensured that the subproject would inherit the parent's caches which
resulted in a change of default behaviour.

This MR as a whole closed #401 [3], which meant that a project's remote
cache could contain *all* artifacts, including subproject artifacts.
However, this was a behaviour change which did not come with any
testing, documentation or an announcement.


## The problem
From the discussion on #401 [3], it is clear that there are users who
require BuildStream to behave differently for their respective
use-cases.

We need to be flexible and provide the appropriate configuration options
to ensure that BuildStream can satisfy the different use-cases.

## The proposal
I'm going to propose what I think the default behaviour should be, and
what configuration I think we should add to support the different
use-cases.

I suspect that people's opinions of what the default behaviour should be
will differ, so I've brought this to the list so that we can all agree
on the final implementation.

After having discussed this with Jürg, we have come up with three
possible variants in behaviour:

1. Use per-project cache servers as configured
    - This would be the default behaviour
2. Use both top-level and subproject cache servers (preferring
   top-level).
    - This is how master currently behaves, but we intend to make this a
      configurable option.
3. Only use top-level (and user) cache servers. Completely ignore
   subproject cache servers.
    - This would be another configurable option.

It's understandable why users will want (2) or (3) so we definitely
should be providing configuration which allows this, but I think that
(1) follows the principle of least surprise, because it exhibits
behaviour which would match the pull/push behaviour when building
the projects separately, and thus should be the default.

To implement this, I suggest that we revert 24c0de16 [2] and add (2) and
(3) as possible *junction-level* configuration options.


## Aside
Just as an aside; I have a commit which reverts the current
behaviour and tests the originally intended behaviour (see [4]).

Please let me know your thoughts.
Thanks,
James


[0] 
https://gitlab.com/BuildStream/buildstream/merge_requests/1113/diffs?commit_id=1ee4a4ba3a91e27515967d5e2daaedf664f10c78
[1] https://gitlab.com/BuildStream/buildstream/merge_requests/1113
[2] 
https://gitlab.com/BuildStream/buildstream/merge_requests/1113/diffs?commit_id=24c0de16faec2b8b9bd6a03504ce951dc49afbe2
[3] https://gitlab.com/BuildStream/buildstream/issues/401
[4] 
https://gitlab.com/BuildStream/buildstream/commit/f32750e83720607a66158cd498cfca936fd5cbb6

Thanks,
James