Re: [BuildStream] How to synchronize nested junctions

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi,

It's nice to be ironing out these corner cases of junctions :)


On Thu, 2019-04-11 at 10:07 +0200, Jürg Billeter wrote:
> Hi Tristan,
>
> On Thu, 2019-04-11 at 16:07 +0900, Tristan Van Berkom wrote:
> > On Wed, 2019-04-10 at 17:57 +0200, Jürg Billeter wrote:
> > > The current behavior is to use the ref (and subproject options) from
> > > the junction in the top-level project. If two subprojects have possibly
> > > conflicting junctions of the same name and the top-level project
> > > doesn't have a junction of the same name, BuildStream reports an error.
> > >
> > > (To be precise, the junction used for conflict resolution doesn't have
> > > to be in the top-level project, a common ancestor of the conflicting
> > > projects is sufficient.)
> >
> > I think this is a bit reversed and I suggest changing this behavior so
> > that it is a configurable warning.
> >
> > Rationale: My expectation when depending on a subproject is that I will
> > obtain exactly the artifacts from that project as I would, if I were to
> > build that project explicitly myself. 
> >
> > I think that overriding a detail of my subproject implicitly in this
> > way, especially in a way the user might not expect or be aware of, is a
> > bad thing.
>
> I agree. The reason why the higher level wins right now is that there
> is a need to support conflict resolution if two subprojects at equal
> level have conflicting junctions, and that decision has to fall on the
> higher level.

Understood.

> However, the ideal situation would be to avoid conflicts in the first
> place and in that case it indeed makes sense to inherit ref/options
> from the subproject's junction.

I think that the new semantic Chandan and you are are discussing is the
ultimate solution for avoiding conflicts.

> > If I am project (A) and I depend on both (B) and (C), I might not be
> > aware that (B) one day also decides to depend on (C) - in this instance
> > I am no longer depending on (B)'s definition of (B), I am in fact
> > redefining what (B) is by overriding it's ref to (C).
> >
> > I propose that (following the above example) we change this to be a
> > configurable warning [...]
>
> I generally agree with this approach. However, we might also want to
> add junction config to explicitly suppress this warning/error for
> particular cases (somewhat similar to the overlap whitelist). I.e., we
> should be able to configure in the junction for (B) (B.bst in (A)) that
> junction C.bst in (B) should be overridden by junction C.bst in (A).

I am rather ambivalent on this point, it can be nice to have... but as
you say...

> (Theoretically, we could consider generalizing this to allow overriding
> arbitrary elements in subprojects, however, I'm not sure about that.)

Indeed, and this is a bigger problem I think we need to consider
separately.

I am currently trying to build a GNOME VM image using gnome-build-meta
on top of freedesktop-sdk, and what I find is that GTK+ is built in
both projects, but differently.

I am not yet sure what the right solution to this will be, I think we
will need one, but I think we need more discovery before committing to
a given solution.

> As the number of such junction conflicts is expected to be small, an
> option would be to always consider junction conflicts fatal, i.e.,
> require the explicit override suggested above instead of supporting
> non-fatal warnings. Don't know what would work better for the majority
> of projects.

Always fatal could be okay, it turns this aspect into a breaking change
rather than an added warning, but it may be alright.

Again rather ambivalent as to whether it should be always fatal...
however it does inform my response to your later comment, which would
be more robust (as you put it) if we force them to be always fatal.

> > * If the ref and project options for (B)->(C) differ from the ref and
> >   project options for (A)->(C) in any way, then a warning is issued
> >   at build time, explaining that the ref and options declared
> >   for the (A)->(C) junction will override the ref and options declared
> >   for the (B)->(C) junction.
>
> Do you think it will be common to have a structure where junctions
> potentially conflict but the ref/options will always be in sync and
> thus, they don't actually conflict? I'm still in favor of the current
> approach (in this aspect) of considering structural conflicts without
> actually comparing ref/options. This seems simpler and more robust to
> me, otherwise you may get conflicts when updating subprojects.

I think I understand what you mean, but I still doubt that avoiding
comparisons of junction configurations is viable...

What I am originally saying is that:

  If the ref and options do not match, I want an error (or at least a
  warning).

What you are saying is that we do not at this time compare refs or
options, and you are pointing out that it would be more fragile, I
agree inasmuch as we need the comparison algorithm to always be in sync
with existing possible junction configuration options.

Now, if we add Chandan's symbolic junction into the mix; we can say
that if you depend on (B) and (C), you *must* use a symbolic junction
to (C) instead of trying to declare the ref/options yourself in (A).

If I understand correctly, this approach allows us to use a fatal error
which forces project (A) to change their concrete junction to (C) into
a symbolic junction to (C) (using the configuration of (B)'s junction
instead).

I say *change* here because I am considering that things were working
fine for project (A) which depended on both (B) and (C) - *until*  the
moment which project (A) updated it's ref to a version of project (B)
which started to also depend on project (C).

Up until here... I agree that if we make the errors fatal, we can avoid
comparing actual junction configurations, by ensuring that there is
only ever one junction from one project to another which actually
declares options and refs.


However:

What happens with diamond dependency shapes; where (B) and (C)
independently depend on project (D), and project (A) depends on both
(B) and (C) ?

As I see it, (B) and (C) both need "concrete" junctions to (D), since
they are also both independent projects which serve their own purposes
and need to be built on their own.

But project (A) needs a way to ensure that it is using compatible
versions of both projects (B) and (C)... i.e. it must have a way to
keep those in sync.

If for instance... project (A) had a symbolic junction to (D), by
explicitly mirroring the (B)->(D) junction... it would still be
unacceptable to blindly use the (B)->(D) junction ref and configuration
to override the (C)->(D) junction.


The diamond configuration seems to reinforce that we really do need to
have a way to explicitly override junction configurations in
subprojects, but I still see overriding subproject junction
configurations as a brute force last resort; ideally we need a way for
project (A) to have a compatible versions of (B) and (C) which both
depend on (D).

I think at least in this case, we really do need to compare junction
configurations in order to at least provide a sensible warning that (B)
and (C) are out of sync.

Cheers,
    -Tristan