[BuildStream] Specifying default dependencies for element kinds

[Chandan Singh <chandan chandansingh net>]

Hi,

Some of us have been recently discussing about how to handle default
dependencies for certain element kinds in BuildStream, and I thought this
might be good topic for the mailing list.

Problem
=======
Let me briefly explain the problem first.

Most build elements in BuildStream expect some commands/executables to already
be present in the sandbox. For example, `kind: make` expects to be able to
find `make` in the build sandbox. Usually these dependencies are provided
by the base platform, which often imports a tarball or something like that.

This works fine for the most part, but when a project has element plugins
that require toolchain that is built by the same project (or a subproject),
the `.bst` files start to become repetitive and overly verbose.

For example, if a project builds its own Python, let's say in `python.bst`,
each element of `kind: pip` will have a build-dependency in `python.bst`.
Here is a more generic example:

    kind: build-tool
    sources: []
    build-depends:
      - build-tool.bst
      ...
      - app-dependencies.bst


Ideas
=====

Includes
--------
Without changing BuildStream, the closest we can achieve today is having
include files that define both kind and the required dependencies. That's one
option but I personally don't like that idea very much because it doesn't
provide an easy way to manage this information in a central location.

So, I was thinking that it might be nice to be able to define such "default"
dependencies for element plugins at the project level.

Defaults in project configuration
---------------------------------
This may not be a fully formed solution just yet but I was imagining something
like the following in project.conf:

    build-depends:
        pip:
          # Implicit default dependency for all elements of kind `pip`
          - python.bst

        build-tool:
          # Implicit default dependency for all elements of kind `build-tool`
          - build-tool.bst

There's no reason to treat build dependencies differently, so I'd imagine
`runtime-depends` (also `depends`) to work in the same way. Perhaps we might
also want to allow define default build dependencies for the entire project as
well. That can be useful for things like base platform, which almost everything
else needs.

If we go with this idea, it might also make sense to support similar
directives that we already support for variables and configuration options. [1]

---

Let me know how you think about this. I'd also be very interested to hear if
others have better ways to approach this problem.

Cheers,
Chandan

[1]: https://buildstream.gitlab.io/buildstream/format_intro.html#directives