[BuildStream] How to synchronize nested junctions

[Chandan Singh <chandan chandansingh net>]

Hi,

tl;dr: nested junctions are hard, and we need a way to make experiences
involving them nicer.

Problem
=======
I will try to explain the problem by using a hypothetical example, but I would
like to mention that this is a real problem that we are seeing in some of our
internal projects.

Imagine you have three BuildStream projects:

  1. Base SDK, which includes things like compilers, basic shell utilities etc.
  2. Core libraries, which includes things like infrastructure libraries that
     are shared by many teams, but are not part of the base SDK for various
     reasons.
  3. User-land applications, which are generally at the leaf-levels of the
     global dependency graph. Such applications may need internal libraries
     of their own but that shouldn't be too important for the purposes of this
     email.

So, we will have the following edges in our dependency graph:

[core-libs]      --> [base-sdk]
[user-land apps] --> [base-sdk]
[user-land apps] --> [core-libs]

Since both [user-land apps] and [core-libs] depend on [base-sdk], we have a
problem in the sense that they have to be kept in sync. Otherwise, the
resulting output may not be compatible. This is hard because [core-libs] and
[user-land apps] live in entirely different repositories, and one of the reasons
for us to adopt BuildStream was that it won't require us to handle such
problems by keeping multiple repositories in sync.

Also, imagine that there are more than one projects at the level of
[user-land-apps] so this problem now becomes much harder.

Proposal
========
If you agree that this is a problem then let's talk about how to fix it.

In the [user-land apps] project, if we were able of saying that BuildStream
should use whatever version of [base-sdk] that's used by [core-libs], we would
not have this problem. If someone doesn't need compatibility for such nested
junctions, then nothing changes for them. But, for people who want/need to
ensure such consistency, they would now have a way of depending on elements
that are in a nested junction. And doing it this way won't require an
external process to keep the two junctions in sync.

Pseudo-junctions/Symbolic junctions
-----------------------------------
This isn't fully thought-out just yet, but I wanted to float this idea to see
how others like it. Also, if I have overlooked some obvious solutions, please
let me know.

If we somehow allow creating a `base-sdk.bst` pseudo-junction in
[user-land apps] project that is equivalent of saying
`core-libs.bst:base-sdk.bst`, that would solve this issue.

I have called these pseudo-junctions because they are not really junctions
themselves but pointers to other junctions. We will have to ensure to not call
them pseudo-junctions in the final version :)

Some potential ways of achieving this could be:

  1. Allow junction elements to refer to nested junctions by supporting
     an additional config option corresponding to which nested element it
     should treat as the target of the symbolic link.

  2. Support nested junction lookups completely, i.e. allowing one to say
     `core-libs.bst:base-sdk.bst:libfoo.bst`. If we do this, it would also be
     nice to add some syntactic sugar to avoid typing
     `core-libs.bst:base-sdk.bst:` each time one needs to lookup a nested
     element.

I am not fully sold on either approaches just yet, so I'd be quite interested
to hear what others think.

Cheers,
Chandan