Re: [BuildStream] Specifying default dependencies for element kinds

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Fri, 2019-06-07 at 19:49 +0100, Chandan Singh wrote:
> Hi,
>
> Some of us have been recently discussing about how to handle default
> dependencies for certain element kinds in BuildStream, and I thought this
> might be good topic for the mailing list.

Thanks for raising this Chandan,

I think that this is essentially:

   "It could be infer additional YAML configurations from the plugin" 

And I wonder if this extends beyond only the build dependencies on the
build systems required.

> Problem
> =======
> Let me briefly explain the problem first.
>
> Most build elements in BuildStream expect some commands/executables to already
> be present in the sandbox. For example, `kind: make` expects to be able to
> find `make` in the build sandbox. Usually these dependencies are provided
> by the base platform, which often imports a tarball or something like that.
>
> This works fine for the most part, but when a project has element plugins
> that require toolchain that is built by the same project (or a subproject),
> the `.bst` files start to become repetitive and overly verbose.
>
> For example, if a project builds its own Python, let's say in `python.bst`,
> each element of `kind: pip` will have a build-dependency in `python.bst`.
> Here is a more generic example:
>
>     kind: build-tool
>     sources: []
>     build-depends:
>       - build-tool.bst
>       ...
>       - app-dependencies.bst

Right, personally I don't find this verbosity particularly problematic,
but if there is a very elegant and nice way to automate these deps to
be implied by their plugin that could be nicer indeed :)

E.g. we currently have a directory with elements which provide these
dependencies:

   https://gitlab.com/freedesktop-sdk/freedesktop-sdk/tree/master/elements/public-stacks

And then we have all the elements which use these explicitly build
depend on them:

   https://gitlab.com/freedesktop-sdk/freedesktop-sdk/blob/master/elements/components/alsa-lib.bst
   https://gitlab.com/freedesktop-sdk/freedesktop-sdk/blob/master/elements/components/alsa-plugins.bst
   https://gitlab.com/freedesktop-sdk/freedesktop-sdk/blob/master/elements/components/aom.bst
   https://gitlab.com/freedesktop-sdk/freedesktop-sdk/blob/master/elements/components/appstream-glib.bst
   ...

As I said, this seems to be nice as it is explicitly listed, but
perhaps it can be even better :)


That said, I wonder if there is more that can and should be inferred by
the commonality found in how a given build system produces output. For
instance, building a program or library with a given build system,
might produce additional outputs that other build systems do not, and
possibly in different locations - an example of this is how build
systems which use libtool might produce the `.la` files that might be
required when building but not when running.

We can see here for instance that freedesktop-sdk extends the default
devel split rules as we appear to have missed some things (like the
`cmake` equivalent of m4 macros in the aclocal directory):

    https://gitlab.com/freedesktop-sdk/freedesktop-sdk/blob/master/project.conf#L252

Again, it's not really that much of an eyesore, but perhaps this could
be done more elegantly somehow ?

In general, I think that I favor inferring more YAML configuration from
the plugin, and I wonder if ideally that should be deep down in the
YAML layer somehow - i.e. not a feature tied to such a specific
construct as "dependencies", but perhaps a hook to allow more
participation in YAML composition ?

That said, one of the obvious problems with this as a feature in
BuildStream is that BuildStream cannot have intimate knowledge of the
project that it is building - i.e. BuildStream cannot itself know how
you decide to provide the tooling required for autotools to work in the
sandbox - so - the project would have to be the one to somehow make a
global statement that: "autotools elements depend on autotool.bst"


> Ideas
> =====
>
> Includes
> --------
> Without changing BuildStream, the closest we can achieve today is having
> include files that define both kind and the required dependencies. That's one
> option but I personally don't like that idea very much because it doesn't
> provide an easy way to manage this information in a central location.
>
> So, I was thinking that it might be nice to be able to define such "default"
> dependencies for element plugins at the project level.
>
> Defaults in project configuration
> ---------------------------------
> This may not be a fully formed solution just yet but I was imagining something
> like the following in project.conf:
>
>     build-depends:
>         pip:
>           # Implicit default dependency for all elements of kind `pip`
>           - python.bst
>
>         build-tool:
>           # Implicit default dependency for all elements of kind `build-tool`
>           - build-tool.bst
>
> There's no reason to treat build dependencies differently, so I'd imagine
> `runtime-depends` (also `depends`) to work in the same way. Perhaps we might
> also want to allow define default build dependencies for the entire project as
> well. That can be useful for things like base platform, which almost everything
> else needs.
>
> If we go with this idea, it might also make sense to support similar
> directives that we already support for variables and configuration options. [1]

Interesting, I am quite liking this approach... and I wonder, does this
not already work all by itself ?

Or are the plugin specific YAML overrides for elements limited to the
"config", "variables" and "environment" sections ?

I.e. I would want it to be done somewhere like here:


It also works well with downstream projects, for instance consider:

  o The base project which bootstraps gcc and builds the build systems
    from the ground up, offers an include file which extends the build
    elements.

  o Downstream projects which junction this "root" project include
    those extensions to the build element configurations in their
    project.conf

This way the information can be maintained in one place and need not be
repeated in an error prone way across multiple junctioning projects.

Cheers,
    -Tristan