[BuildStream] Modifying/excluding elements from junctions

[Abderrahim Kitouni <akitouni gnome org>]

Hi all,

this is to discuss possible solutions (and bst features that could
help) on one problem we have using freedesktop-sdk from
gnome-build-meta.

So freedesktop-sdk includes a few gnome projects (glib, gtk and
friends) and some freedesktop projects that depend on them (like
gstreamer). We would like to use the freedesktop projects as-is, but
want to be in control of the gnome projects (use the latest git commit
in the nightly pipeline, use an exact tarball for releases).

Before discussing which buildstream features we need for the solution,
I'd like to first talk about the semantics of what's a solution to
this problem. I'm using the terminology above (freedesktop-sdk,
gnome-build-meta, freedesktop, gnome) to keep things manageable, but
this applies to any project in the same situation. There are three
possible ways to do it:

- overlap: just stage the gnome-build-meta version on top of the
freedesktop-sdk version. (This is what we currently do for glib)
  con:
  - can keep files from the freedesktop-sdk version
  - involves adding a dummy dependency to ensure the gnome-build-meta
version is always staged on top of the freedesktop-sdk version.

- replace: remove the freedesktop-sdk version of the element and
replace it with the new version, without rebuilding reverse
dependencies. (we do this for e.g. gstreamer plugins)
  pro:
  - uses artifact from junction as-is (no need to rebuild, have the
exact same binaries)
  con:
  - can lead to subtle failures if the replacing element isn't exactly
ABI compatible with the replaced element.

- rebuild: replace the freedesktop-sdk element with another one,
rebuilding all reverse dependencies. (we do this for a few other
elements)
  pro:
  - everything is built consistently (e.g. ABI, or changes in headers
are applied)
  con:
  - need to rebuild everything (we end up with things that aren't
exactly the same as the junction).

all of the above are currently possible, although they involve jumping
through some hoops: using filter elements, and manually keeping
dependencies in sync for "replace"; patching (or copying and modifying
element from) the junction for "rebuild".

The first question is which of these workflow do we want to support?
Then, there are a few other things to consider regarding how to
actually implement it (what syntax to use? should it be restricted to
junctions? etc.).

Valentin David has a MR implementing the "rebuild" workflow [1]. The
discussion there is also worth reading.

For my part, I'd like to have the "replace" workflow, with a syntax to
allow staging an element instead of another so whenever an element
depends on both the replacing and the replaced element, only the
replaced element should be staged. This could also be used to use a
stack element from the junction, but exclude one of its dependencies.

What do you think?