Re: [BuildStream] Re-using top level elements

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi William,

First I'm going to say that, while I have some context from private
conversations to have a clue, I find the term "reusing elements" used
here confusing.

While artifacts and plugins are things to be reused and shared among
projects, elements themselves are not, they are a project's mechanism
of declaring an 'Element' (instance of a plugin).

I would not recommend trying to use includes to accomplish any
templating, and I would not recommend using includes to define
dependency lists (if using a cross junction include to define a
dependency works at all, I would be quite surprised).


What I think you are driving at, in my own words, is that basically:

  * There are some constructs which can be difficult or require
    a lot of setup to express

  * Sometimes these constructs require multiple elements, for instance:
    - Build some special extra tools which allow one to deploy an image
    - Compose the filesystem data which needs to be deployed into that
      image
    - Write a final script element, like x86image as an example, which
      uses these tools to create the image

  * It would be nice to be able to share these constructs across
    projects in some way, such that the more complex multi-element
    construct can be expressed with a simple declaration, safely.

  * Especially, if I have a low level project which does provide
    some artifacts which allow building the output, I should be
    able to provide that construct to a project which junctions
    my project.

    Providing this from a lower level project appears to be something
    safe, because I even provided builds of the required artifacts to
    accomplish this task.

Am I pretty much interpreting this correctly ?


Without diving into how to accomplish this, I can say that I am largely
in agreement with this, we should be able to improve the sharing of
more complex constructs across projects, that would be great.

My initial thoughts on this are that we have to revisit the concept of
making it easier to express more complex constructs in our YAML format,
which originally came up during the element testing thread[0], but was
later expanded on in the subsequent BeaverCon in Manchester, where we
had a long workgroup session to discuss testing and validating of
artifacts.

To [re]summarize the relevant bits of those conversations:

  * I originally had a simple draft of how we might express multiple
    elements in a single YAML file: see my 'sequence' example in [1].

    This would have allowed some hard coded ways for BuildStream to
    also infer something about how elements in a single YAML file would
    depend on eachother.

  * In the workgroup session, Jürg suggested that instead of having
    a rigid, built-in concept like I had suggested, it would be
    interesting to introduce a new kind of plugin which would allow
    instantiating elements in different ways.

    This approach would be quite powerful, as it would allow for
    plugins to express more complex constructs, and also infer some
    values and configurations about the elements it instantiates,
    including their dependency relationships.


> From what I understand, this approach would (almost) solve the problems
you are trying to solve, by allowing your base project to declare such
a plugin which allows higher level projects to express your construct
with much less YAML.

I say "almost" because there is another issue which you point out,
which is the implied knowledge about a given element in the project
which your plugin would want it's instances to automatically depend on.

To bring back the x86image example, imagine a world where:

  * Freedesktop SDK provided the x86image plugin, but this plugin
    would define the whole construct, exposing some parameters of
    how to apply `compose` configurations to the input, and
    some parameters about filesystem image size, etc.

  * Freedesktop SDK also provided artifacts allowing x86image
    scripts to work (e.g. syslinux, coreutils, mkfs etc).

  * Projects could then depend on Freedesktop SDK for the purpose
    of rolling out bootable x86 images, regardless of whether they
    even use Freedesktop SDK in the payload they are distributing.

  * Now what if the plugin provided by Freedesktop SDK could be
    "bound" to Freedesktop SDK in some way, such that it could
    implicitly depend on elements from that project ?

    This would allow the plugin to automatically depend on artifacts
    from Freedesktop SDK for the purpose of preparing a sysroot in
    which the commands it performs will succeed.

This last point would also be quite powerful, as you point out, there
is no way for a plugin to have knowledge about an element or even how
that element can be addressed, because the plugin does not have a
junction to the project from where it was provided from (the concept of
a project where a plugin was provided from doesn't even exist).

I think that pursuing this line of development (i.e. both "plugins
which allow the expression of multiple elements", and "binding these
plugins to a specific project") would payoff a lot, but take it with a
grain of salt; I think it's just in brainstorming stages at this point
and needs more discussion.

Cheers,
    -Tristan

[0]: https://mail.gnome.org/archives/buildstream-list/2018-July/msg00025.html
[1]: https://mail.gnome.org/archives/buildstream-list/2018-July/msg00033.html


On Wed, 2020-05-20 at 11:25 +0100, William Salmon wrote:
> Hi BuildStream mailing list
>
> ## Context
>
> Whilst the BuildStream community are still finessing junctions for 
> dependencies they generally work well and there are some nice open 
> examples, eg, gnome-build-meta[1] using Freedesktop-SDK[2].
>
> However for reusing some classes of element I have not found good 
> examples of how to use junctions:
>
>    * Reuse "top level" elements (see below) through junctions
>
>    * Reuse elements that you need to tweak [3] through junctions
>
> This email will focus on the first but the ML thread for the second is 
> also important.
>
> ## Top level element
>
> To me these are any element that manipulates previous elements, eg 
> script elements, although I don't want to tie these down too tightly to 
> any one use case. I think the problem is reasonably generic as many 
> classes of project may find this useful.
>
> Script elements can be used to take existing elements and tweak them, 
> the act of tweaking may want to be defined once and shared via a 
> junction but the element subject to the tweak may come from the project 
> containing the junction.
>
> Other elements as well as script elements behave like this but they are 
> less common.
>
> ## Prior art
>
> BuildStream allows projects to include yaml across junctions, projects 
> like CRASH do this [4] to make use of predefined variables in there 
> junctions [2].
>
> Plugins can help with this but most plugins still have some parameters 
> so while they can help I do not think they solve this problem.
>
> ## Extending
>
> Including yaml can be extended to element like yaml, a example of such 
> can be found in the example app [5] and example pi support project [6].
>
> Includes seem to be the most canonical/bst way to share top level elements.
>
> ## General notes
>
> Because the two projects make assumptions about each other, the base 
> project can only be used by projects that respect those assumptions.
>
> A form of API between projects using junctions already exists for 
> projects that interact with junctions, eg. the file names of bst files 
> in the junction are used in the consuming project, therefore they must 
> match.
>
> But for template yaml the junction-ed template makes assumptions about 
> the consumed project, eg, the name of the junction that included it, if 
> it needs to refer to other elements in the junction-ed project.
>
> ## Quirks of the approach
>
> For our attempt to use them we had a setup as follows;
>
> For our PoC we assumed that all consuming projects, eg. [5], have a 
> junction called bsp.bst that is used to consume the support projects, 
> eg. [6].
>
> The "top level" elements that need to combine elements from both 
> projects must refer to the elements form the junction-ed project via the 
> junction, as noted above.
>
> This leads to templates in the base project for use by other consuming 
> projects, where the template has to refer to other elements in the 
> original project by a junction name. This means that the template can 
> not be used in the original project. The original project does not have 
> a junction to itself.
>
> The need for a reference consumer project to test the templates is not 
> ideal.
>
> ## Call for feedback
>
> How are other members of our community achieving similar things?
>
> Are you interested in our examples? We would welcome general feedback on 
> them given that they are "Proof of Concepts" focused on achieving shared 
> "Top level" elements.
>
> Do people think we have missed some tricks to make this easier? Do the 
> community think we could make some of this easier? Is there appetite for 
> something like a junction element that refers to the project that it is 
> in, so template style elements can be tested more easily? or is there a 
> alternative trick here?
>
> Regards
> Will Salmon
>
> [1] https://gitlab.gnome.org/GNOME/gnome-build-meta
> [2] https://gitlab.com/freedesktop-sdk/freedesktop-sdk
> [3] https://mail.gnome.org/archives/buildstream-list/2020-May/msg00013.html
> [4] 
> https://gitlab.com/celduin/crash/jetbot-system-bst/-/blob/master/project.conf#L51
> [5] https://gitlab.com/celduin/bsps/example-app
> [6] https://gitlab.com/celduin/bsps/pi-3b-plus-bsp
> _______________________________________________
> buildstream-list mailing list
> buildstream-list gnome org
> https://mail.gnome.org/mailman/listinfo/buildstream-list