Proposing new include directive

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi all,

The original BuildStream format had an "include" semantic built into
it, which I removed because it did not have enough real supporting use
cases at the time.

Originally I thought it might be interesting for the purpose of
composition of complex things, such as carefully constructing the
compiler tuning configuration flags one might need for defining support
for various different CPU/board configurations, without having to
redundantly declare a lot of things.


Problem statement
~~~~~~~~~~~~~~~~~
This might still be useful for the above mentioned purpose of defining
compiler tunings, but now we have a much more pressing use case for
this, which is outlined in:

    https://gitlab.com/BuildStream/buildstream/issues/331

Essentially, we need a way to "inherit" certain things from projects we
depend on, because project.conf is becoming burdensome for multiple
reasons:

  o gnome-build-meta depends on (or will shortly depend on) the
    freedesktop-sdk project, which is a good example of project
    separation. However we can already see that there is a lot of
    redundant project.conf declarations which will have to be included
    in both projects - and parallel maintenance of these things will
    result in the details falling out of sync.

    It would make much more sense here if `gnome-build-meta` were to
    instead "extend" the configurations declared in `freedesktop-sdk`

  o We are also trying to bake up an elegant solution for building
    things like simple Flatpak applications, these might in turn
    depend on `gnome-build-meta`, or another SDK like a KDE one,
    and in this case the requirements in `project.conf` are much
    too onerous for smaller and simpler projects.


The include directive proposal
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
After shooting around some potential ideas to solve the issues
described above, Jürg and I have concluded that bringing back the
include directives in some form would be an elegant way to solve this,
mostly because implicit inheritance models don't really make sense when
you get down to the dirty details.

This should add a minimal amount of code to the actual core, I expect
most of the leg work here to consist of elaborate test cases and
documentation to cover this.


  The include directive
  ~~~~~~~~~~~~~~~~~~~~~
  We reserve parenthesized words expressed as dictionary keys in
  the BuildStream YAML format, and currently have the following
  directives:

    (?) Conditionals
    (!) Assertsions
    (<) List prepend
    (>) List append
    (=) List override

  This proposal adds:

    (@) Include

  Wherever the (@) directive is encountered as a dictionary key, it's
  value is expected to be either:
    - An include string
    - A list of include strings

  It's nice to support the string-or-list in this case I think; while
  it is important to support multiple includes in the same dictionary
  and perform the includes in a loop, I suspect that in many cases we
  will only be using a single include.


  The include string
  ~~~~~~~~~~~~~~~~~~
  The include string itself represents a project relative filename for
  inline composition in the including dictionary.

  This however has the exception that it may be prefixed with a
  "junction path" from the including project.

  Examples:

    # Include a dictionary defined in the same project
    foo:
      some-key: Pony
      (@): common/something.inc

    # Include a dictionary defined in the project referred to
    # by the "junction.bst" junction
    foo:
      some-key: Pony
      (@): junction.bst:common/something.inc

    # Junction paths can be recursive
    foo:
      some-key: Pony
      (@): junction.bst:base.bst:common/something.inc


  Note about junction paths
  ~~~~~~~~~~~~~~~~~~~~~~~~~
  Junction paths are "already a thing", however they are presently only
  used for display purposes.

  That said, since `project.refs` work has recently landed, it has
  already become important to be able to address elements using these
  junction paths - some minor work needs to be done in order to support
  things such as:

     bst track junction.bst:element.bst


  How include composition works
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  The composition of an included YAML file into the including
  dictionary works in the opposite direction of the (?) conditional
  directives.

  With (?) directives, the conditional YAML fragment *overrides*
  what is declared in the dictionary declaring the conditional, one
  can say that the composited YAML fragment is "on top"

  With the (@) include directive, composition happens the other way
  around, such that it first replaces the including dictionary, and
  the declaring dictionary is composited over the included one.

  One can say then that an included YAML fragment is composited
  "underneath" the including dictionary.


  When include directives are processed
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  The (@) directives can be processed before any conditionals
  have been processed on the including side. But must be processed
  after any conditionals have been processed in the target included
  YAML file.

  Currently the Loader and the Project object which is responsible
  for reading the project options, do an early resolution of
  conditional statements - these two aspects are necessarily processed
  separately.

    For the project
    ~~~~~~~~~~~~~~~

      o A few special attributes, such as project name and required
        format version, must be loaded first

      o Now includes can be processed.

      o Now we load the actual options

        o These options might have been extended by way of includes
          (this might be helpful for deriving supported machine
          architecture options from sub projects, such that support for
          these need not be added in every project).

          Whether we can reasonably support this extension of options
          can be discovered during implementation.

      o Then conditionals are processed.

      o Now the rest of the loading is allowed to proceed.


    For element declarations (.bst files)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

      o Includes can be processed immediately

      o Now we process conditional statements

      o Now we continue with the remainder of the loading process

    NOTE: It's important that conditionals be resolved completely
          within the context of loading a file to be included
          somewhere, such that we never end up including raw,
          unresolved conditional statements in a depending project.

    For files being included
    ~~~~~~~~~~~~~~~~~~~~~~~~

      o First any includes declared in *that* included file are
        processed.

      o Now conditional statements are processed in context of the
        owning project.

      o Now the YAML fragment is ready to be composited


  A note about list composition directives
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  List composition directives (<), (>) and (=) are handled directly as
  a consequence of dictionary composition.

  It should be possible for included files to also contain unhandled
  list composition directives which can be handled in later passes of
  composition, this is mostly a solved problem since we already have
  support for list composition directives inside conditional
  directives.


I will close this proposal here, and leave an additional example use
case in post scriptum.

Any thoughts and comments on this proposal are invited :)

Cheers,
    -Tristan


PS: Anticipated use case example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Based on the freedesktop-sdk and gnome-build-meta use cases, which
represent 3 levels already (because freedesktop-sdk is already composed
of 2 interdependant BuildStream projects), this is how I expect things
to be declared once we have include directive support:


  base-sdk-bootstrap
  ~~~~~~~~~~~~~~~~~~
  Here we declare things which make sense in the base runtime

    include/project.inc
    ~~~~~~~~~~~~~~~~~~~

      # base variable declarations
      variables:
        libdir: ...

      # base environments
      environment:
        CPPFLAGS: "-O2 -D_FORTIFY_SOURCE=2"

      # base shell configuration, lets do something bash specific
      shell:
        command: [ 'bash', '--noprofile', '--norc', '-i' ]

    project.conf
    ~~~~~~~~~~~~

      # Include the shared include file
      #
      (@): include/project.inc

      # Local project specific configurations
      #
      name: base-sdk-bootstrap
      aliases:
        flathub: https://flathub.org/
      element-path: elements
      ...

  base-sdk
  ~~~~~~~~
  Here we declare things which make sense in the freedesktop SDK

    include/project.inc
    ~~~~~~~~~~~~~~~~~~~

      # Inherit the options from base-sdk-bootstrap
      #
      (@): bootstrap-junction.bst:include/project.inc

      # Lets extend this with some knowledge that is relevant
      # for the type of system we're building here...

      # This system probably knows about XDG stuff, lets
      # share host access with the XDG runtime dir so that
      # we have access to it from `bst shell` environments
      shell:
        environment:
          XDG_RUNTIME_DIR: '$XDG_RUNTIME_DIR'

        host-files:
        - '${XDG_RUNTIME_DIR}'

    project.conf
    ~~~~~~~~~~~~

      # Include the shared include file
      #
      (@): include/project.inc

      # Local project specific configurations
      #
      name: base-sdk
      ...

  gnome-build-meta
  ~~~~~~~~~~~~~~~~
  No need for a full example here I think.

  Here we can again extend project.inc separately, before including
  it directly in gnome-build-meta's project.conf.

  In this way project.inc from gnome-build-meta has grown enough
  features such that it can simply be included in a project.conf
  of a flatpak generating project, and that simple project need not
  replicate all of the knowledge known by the supporting SDK projects.