[BuildStream] Re-using top level elements

[William Salmon <will salmon codethink co uk>]

Hi BuildStream mailing list

## Context

Whilst the BuildStream community are still finessing junctions for 
dependencies they generally work well and there are some nice open 
examples, eg, gnome-build-meta[1] using Freedesktop-SDK[2].

However for reusing some classes of element I have not found good 
examples of how to use junctions:

  * Reuse "top level" elements (see below) through junctions

  * Reuse elements that you need to tweak [3] through junctions

This email will focus on the first but the ML thread for the second is 
also important.

## Top level element

To me these are any element that manipulates previous elements, eg 
script elements, although I don't want to tie these down too tightly to 
any one use case. I think the problem is reasonably generic as many 
classes of project may find this useful.

Script elements can be used to take existing elements and tweak them, 
the act of tweaking may want to be defined once and shared via a 
junction but the element subject to the tweak may come from the project 
containing the junction.

Other elements as well as script elements behave like this but they are 
less common.

## Prior art

BuildStream allows projects to include yaml across junctions, projects 
like CRASH do this [4] to make use of predefined variables in there 
junctions [2].

Plugins can help with this but most plugins still have some parameters 
so while they can help I do not think they solve this problem.

## Extending

Including yaml can be extended to element like yaml, a example of such 
can be found in the example app [5] and example pi support project [6].

Includes seem to be the most canonical/bst way to share top level elements.

## General notes

Because the two projects make assumptions about each other, the base 
project can only be used by projects that respect those assumptions.

A form of API between projects using junctions already exists for 
projects that interact with junctions, eg. the file names of bst files 
in the junction are used in the consuming project, therefore they must 
match.

But for template yaml the junction-ed template makes assumptions about 
the consumed project, eg, the name of the junction that included it, if 
it needs to refer to other elements in the junction-ed project.

## Quirks of the approach

For our attempt to use them we had a setup as follows;

For our PoC we assumed that all consuming projects, eg. [5], have a 
junction called bsp.bst that is used to consume the support projects, 
eg. [6].

The "top level" elements that need to combine elements from both 
projects must refer to the elements form the junction-ed project via the 
junction, as noted above.

This leads to templates in the base project for use by other consuming 
projects, where the template has to refer to other elements in the 
original project by a junction name. This means that the template can 
not be used in the original project. The original project does not have 
a junction to itself.

The need for a reference consumer project to test the templates is not 
ideal.

## Call for feedback

How are other members of our community achieving similar things?

Are you interested in our examples? We would welcome general feedback on 
them given that they are "Proof of Concepts" focused on achieving shared 
"Top level" elements.

Do people think we have missed some tricks to make this easier? Do the 
community think we could make some of this easier? Is there appetite for 
something like a junction element that refers to the project that it is 
in, so template style elements can be tested more easily? or is there a 
alternative trick here?

Regards
Will Salmon

[1] https://gitlab.gnome.org/GNOME/gnome-build-meta
[2] https://gitlab.com/freedesktop-sdk/freedesktop-sdk
[3] https://mail.gnome.org/archives/buildstream-list/2020-May/msg00013.html
[4] 
https://gitlab.com/celduin/crash/jetbot-system-bst/-/blob/master/project.conf#L51
[5] https://gitlab.com/celduin/bsps/example-app
[6] https://gitlab.com/celduin/bsps/pi-3b-plus-bsp