Project options and other format enhancements (and dropping "variants")

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi all,

So it's pretty clear by now that we're going to need a solution for
parallel orthogonal project options and that variants by itself is not
a solution for this, or, that other variant-like constraint based
solutions which cater to parallel configurations are just overly
complex (both to implement and probably also to understand as a
format).

This is a bit of a pressing matter because changing the format needs to
be done before we can make a first stable release of BuildStream, which
I'd like to do very soon.

As discussed in this thread[0], it makes little sense to keep variants
around if we're going to have another way of doing conditionals.

So, I've been chewing on this for a while now and thinking what other
things we can solve at the same time, what approach gives us something
great even if we cant have the variant thing which I liked, and, this
email contains a draft of the changes I'm thinking of making while
removing variants.


Comments and discussion invited, are there some better ideas ? is there
any horrible flaws in the plan ?


Cheers,
    -Tristan


[0]: https://mail.gnome.org/archives/buildstream-list/2017-August/msg00004.html



Considerations
==============

  * Need multiple parallel options, this is why variants
    is not suitable by itself

  * When we have multiple options in parallel; this also means
    that elements may write conditional code against more than
    one option.

      o If multiple conditional statements appear on an element,
        the order of their resolution must be constant (this
        may mean a list of conditions)

      o Conditional statement semantics are hopefully more fun
        than just:

          if (condition) { yaml fragment applies }

        It will be interesting to have `else` blocks and also
        I could imagine that `switch/case/default` will be
        useful.

      o Nested conditional statements are also desirable

  * Users need to be able to specify project options at
    BuildStream invocation time - this can be a multiple
    command line option (ala `--option foo=bar --option frob=baz`)
    and also perhaps configurable with a local user configuration
    (user could specify preferred options on a per project basis).

  * When nesting projects, one will need to expose the same interface
    to calling project elements as we offer to the user (so
    nesting recursive pipeling elements will need to be able to
    configure the projects they build).

  * If we're going to add these options, we should use this as an
    opportunity to drop other things from the buildstream codebase
    which could be achieved with this.

      o The target and host arch conditionals in the format and
        corresponding buildstream CLI options should be removed
        in the case that the project is free to expose this.

  * We should address other shortcommings in the format at the same
    time, one which comes to mind is how we composite arrays.

      o Currently the core codebase decides at array composition time
        whether a yaml fragment's array is going to be appended to or
        replaces an existing array.

        This is backwards; the user writing the yaml elements should
        be able to express that their array should replace the target
        array, or be appended or even prepended to the existing array
        (this would also greatly aleviate the need for things like
        pre-build-commands and post-install-commands and the like, at
        the same time it would make it possible to augment existing
        split rule public data).

      o In the same vein as above, there is currently no way to *unset*
        a value in a target dictionary.


I thought about the above for a while now. At first I was considering
something similar to what we've been doing and add some well known
keywords to the yaml at well known locations (e.g.; the 'conditions' list
found at the toplevel scope of any element.bst and also in the project.conf).

But instead of doing that, I think it would be nicer to have something
deeper in the loading engine which applies across the board, and I think
this fits in nicely with the changes I want to make for array composing
and the like.


Format Draft
============
Here is a basic outline of what I'm thinking of. For the sake of this
email lets call these things "project options" or "options".


   Option Declaration
   ------------------
   A project declares which options are valid for the
   project in the project.conf.

   These options should have some metadata which can be used
   to declare the defaults, assert valid values of the options,
   and also a description string which the CLI can use to communicate
   the meaning of project options to buildstream users (not all users
   building a project wrote the project.conf).


   Format Enhancements
   -------------------
   I would propose we add some special tokens which can be used at any level
   of a buildstream element.bst file, or also in some specific parts of the
   project.conf (since project.conf is declaring options, we cannot conditionalize
   that part)

   Below are my ideas for the '>>', '<<', '==', '??' and '!!' operators.


   The '>>', '<<' and '==' operators
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   This would be a semantic that is inherently understood by the
   code we use to composite dictionaries together.

   Use it to specify how an array is applied to an array which
   might already exist in the target dictionary.

   Appending:

       config:
         build-commands:
           '>>':
           - echo "Build Complete !"

   Prepending:

       config:
         build-commands:
           '<<':
           - echo "Commencing build..."

   Replacing:

       config:
         build-commands:
           '==':
           - echo "Commencing build..."

   OR (the default would be to replace):

       config:
         build-commands:
         - echo "Commencing build..."


   The '??' operator
   ~~~~~~~~~~~~~~~~~
   This would be the main entry point to conditional statements and
   would be inherently understood at every level of the element format and
   also in some select parts of the project.conf.

   These conditionals would be resolved before anything else (without
   resolving '<<', '>>' or '==' either) at load time, as a preprocessing
   step on the yaml loaded dictionaries which will later be composited
   and loaded by element plugins.

   The '??' token can appear either as a value in any location; or as
   a key in a dictionary; the resolved value of the conditional expression(s)
   will take the original place of the '??' dictionary after being resolved,
   in the case of the '??' appearing in a dictionary; the resolved value
   is composited into the dictionary and the original '??' key is deleted.

   A condition can also choose to resolve to None, which is to say the
   conditional did not produce any YAML to augment (a simple if statement
   without any else clause), in this case the value is simply left
   out. If  the '??' was specified as the value of a dictionary key, the
   dictionary key is also deleted.

   A single '??' initiates a condition but is not the condition itself,
   the condition itself is the dictionary which is the single value
   in this single key dictionary.

   Further, the value of a '??' can be a list of conditions; these will
   be resolved in the specified order; each time compositing against
   the resulting composition of the former condition.


   The '??' expression format
   ~~~~~~~~~~~~~~~~~~~~~~~~~~
   So at first I was thinking what this would look like as a pure
   YAML format, but it looks like it will be way too verbose for
   expressing simple comparisons.

   Example:


       variables:
       '??':
         condition:
           kind: ifeq
           args:
             option: debug
             value: on
         then:
           conf-extra: --enable-debug
         else:
           conf-extra: --disable-debug

   Later I thought maybe we do our own parsing of strings like
   `ifeq(option, value)`, but that also becomes a little unwieldy, hard
   to maintain and extend to support compound expressions.

   So what I'm leaning towards now is to create a simple expression
   format based on S-Expressions, this way the same expression above would
   just look like:


       '??':
         condition: (ifeq "debug" "on")
         then:
           conf-extra: --enable-debug
         else:
           conf-extra: --disable-debug


   This is especially nice once you want to do anything a bit more
   complex, the following would be a lot more verbose to express if
   it were in YAML:


       '??':
         condition: |

           (and (ifeq "logging" "off") (ifeq "debug" "on"))

         then:
           ... value ...
         else:
           ... value ...


   The S-Expressions are fairly easy to parse and there is a python
   library for that (http://sexpdata.readthedocs.io/en/latest/).

   This is fairly simple to implement with a recursive test algorithm
   which delegates the arguments of every nested expression to a function
   that is associated to the keyword (like '(ifeq ...)' and '(and ...)'
   above, we can easily add '(or ...)' and '(not ...)' etc).


   To elaborate a bit further on how values resolved from condition
   blocks are composited in a loaded YAML file when being resolved,
   I've added some examples. These currently assume the S-Expression
   approach to interrogating project options.


   Conditional of a simple string value
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


       variables:
         debug-flags:
           '??':
             condition: (ifeq "debug" "on")
             value: --enable-debug
             else: --disable-debug


   Conditional inside a dictionary
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


       variables:
         log-flags: --enable-logging
         some-other-key: The Flying Pony
         '??':
           condition: (ifeq "debug" "on")
           value:
             debug-flags: --enable-debug
             debug-message: |

               You can apply more than one key at a time
               if your condition is at the dictionary level

           else:
             debug-flags: --disable-debug


   Conditional composition from the toplevel
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


       '??':
         condition: (ifeq "debug" "on")
         value:
           variables:
             debug-flags: --enable-debug
         else:
           variables:
             debug-flags: --disable-debug


   Conditional value in list
   ~~~~~~~~~~~~~~~~~~~~~~~~~
   Example: Only include libdebug.bst if the debug
   project option is "on"

   In this case if debug is not "on" then the '??' element
   is simply excluded, as the condition does not resolve to
   any value.


       depends:
       - libfoo.bst
       - libbar.bst
       - '??':
           condition: (ifeq "debug" "on")
           value: libdebug.bst


   Condition list at the toplevel
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   Example: here we have one condition list at the toplevel
   of our element and buildstream will composite the resolved
   value of each condition against the element yaml.

   This allows prioritization of conditions.


       '??':
       - condition: (ifeq "logging" "on")
         value:
           variables:
             log-flags: --enable-logging
           dependencies:
             '>>':
             - libs/logging.bst
         else:
           variables:
             log-flags: --disable-logging

       - condition: (ifeq "debug" "on")
         value:
           variables:
             log-flags: --enable-logging
             debug-flags: --enable-debug
           dependencies:
             '>>':
             - libs/logging.bst
         else:
           variables:
             debug-flags: --disable-debug


   Condition list (as value)
   ~~~~~~~~~~~~~~~~~~~~~~~~~
   The value of the '??' key can be a dictionary (a condition)
   or a list (a sequence of conditions).

   The conditions are executed one after the other, the last
   condition which does not resolve to 'None' is applied (if
   all conditions resolve to 'None', then the key in the dictionary
   is left unchanged by this statement.


       variables:
         log-level:
           '??':
           - condition: (ifeq "logging" "on")
             value: log
           - condition: (ifeq "debug" "on")
             value: debug


   Nested conditions
   ~~~~~~~~~~~~~~~~~
   Example: in this case we ignore the debug option entirely if
   logging itself is not enabled.


       '??':
         condition: (ifeq "logging" "on")
         value:
           variables:
             log-flags: --enable-logging
             '??':
               condition: (ifeq "debug" "on")
               value:
                 debug-flags: --enable-debug
           dependencies:
             '>>':
             - libs/logging.bst



   The '!!' operator
   ~~~~~~~~~~~~~~~~~
   This would simply be an assertion and can be used in place of any
   value, or in any dictionary.

   The follwing:


       '!!': Requested pony is not supported on rainbow platform.


   Would raise a LoadError and cause BuildStream to exit with
   the given assertion failure message, along with the yaml filename,
   column and line number which triggered the assertion.

   In combination with the above conditionals, this would provide
   the user a means to express unsupported configurations either
   at the project or element level.

   E.g.:


       '??':
         condition: (and (ifeq "logging" "off") (ifeq "debug" "on"))
         value:
           '!!': |

             Unable to log additional debugging information
             if we dont have any logging module to log it to.