Re: [BuildStream] Proposal: Change how subprojects interact with parent project remotes and add extra configuration

[James Ennis <james ennis codethink co uk>]

Hello list,

Just a note to all that this work has now been completed, see:
https://gitlab.com/BuildStream/buildstream/merge_requests/1403.

Subproject elements will no longer be cached in parent project remotes
by *default*.

If you are using master and you wish to revert back to the old
behaviour, you can add a `cache-junction-elements: True` config option
to your junction element.

Additionally, the third use-case of "I don't want anything to do with
whatever remotes my junction has declared" has also been implemented.
Similarly, add `ignore-junction-remotes: True` as a config option to
your junction.

This has been documented [0] and is in NEWS [1]

Thanks,
James

[0] 
https://buildstream.gitlab.io/buildstream/elements/junction.html#overview
[1] 
https://gitlab.com/BuildStream/buildstream/commit/b03eab16b624cda9bbcc72303ea814a6ac374dc4

On 2019-06-26 12:07, James Ennis via buildstream-list wrote:
> Hi all,
>
> TL;DR
> Currently, BuildStream master will pull/push subproject artifacts
> from/to the parent project remote(s) (remotes which have been declared
> in a project's project.conf).
>
> This behaviour is different to what was originally intended as the
> default behaviour for subprojects and, when introduced, it was 
> introduced
> without being tested/documented/announced.
>
> I propose that we revert this "new" behaviour and add configuration
> which enables it and I would like to hear people's thoughts.
>
>
> ## Context to the problem
> After having briefly discussed this with Jürg, it seems that the
> *intended* default behaviour was that project-specific remotes should
> indeed be project-specific. Although, due to a previous bug in
> master, this was not the case. This was fixed by 1ee4a4ba [0] which was
> introduced as part of !1313 [1]. This commit *alone* reinforced the
> intended behaviour - meaning that subproject artifacts would not
> be pulled/pushed from/to a parent project's remote(s).
>
> However, within the same MR (!1313 [1]), the second commit (24c0de16
> [2]) ensured that the subproject would inherit the parent's caches 
> which
> resulted in a change of default behaviour.
>
> This MR as a whole closed #401 [3], which meant that a project's remote
> cache could contain *all* artifacts, including subproject artifacts.
> However, this was a behaviour change which did not come with any
> testing, documentation or an announcement.
>
>
> ## The problem
> From the discussion on #401 [3], it is clear that there are users who
> require BuildStream to behave differently for their respective
> use-cases.
>
> We need to be flexible and provide the appropriate configuration 
> options
> to ensure that BuildStream can satisfy the different use-cases.
>
> ## The proposal
> I'm going to propose what I think the default behaviour should be, and
> what configuration I think we should add to support the different
> use-cases.
>
> I suspect that people's opinions of what the default behaviour should 
> be
> will differ, so I've brought this to the list so that we can all agree
> on the final implementation.
>
> After having discussed this with Jürg, we have come up with three
> possible variants in behaviour:
>
> 1. Use per-project cache servers as configured
>     - This would be the default behaviour
> 2. Use both top-level and subproject cache servers (preferring
>    top-level).
>     - This is how master currently behaves, but we intend to make this 
> a
>       configurable option.
> 3. Only use top-level (and user) cache servers. Completely ignore
>    subproject cache servers.
>     - This would be another configurable option.
>
> It's understandable why users will want (2) or (3) so we definitely
> should be providing configuration which allows this, but I think that
> (1) follows the principle of least surprise, because it exhibits
> behaviour which would match the pull/push behaviour when building
> the projects separately, and thus should be the default.
>
> To implement this, I suggest that we revert 24c0de16 [2] and add (2) 
> and
> (3) as possible *junction-level* configuration options.
>
>
> ## Aside
> Just as an aside; I have a commit which reverts the current
> behaviour and tests the originally intended behaviour (see [4]).
>
> Please let me know your thoughts.
> Thanks,
> James
>
>
> [0]
> https://gitlab.com/BuildStream/buildstream/merge_requests/1113/diffs?commit_id=1ee4a4ba3a91e27515967d5e2daaedf664f10c78
> [1] https://gitlab.com/BuildStream/buildstream/merge_requests/1113
> [2]
> https://gitlab.com/BuildStream/buildstream/merge_requests/1113/diffs?commit_id=24c0de16faec2b8b9bd6a03504ce951dc49afbe2
> [3] https://gitlab.com/BuildStream/buildstream/issues/401
> [4]
> https://gitlab.com/BuildStream/buildstream/commit/f32750e83720607a66158cd498cfca936fd5cbb6
>
> Thanks,
> James
> _______________________________________________
> buildstream-list mailing list
> buildstream-list gnome org
> https://mail.gnome.org/mailman/listinfo/buildstream-list