[BuildStream] Some ideas on standardised element tests

[Thomas Coldrick <thomas coldrick codethink co uk>]

Hi,

I'd like to propose some kind of start to standardised element tests to 
match
our standardised source tests. The question of this was raised by Ben 
Schubert
in his mail here [0] on migrating plugins and testing.

Firstly some motivation: obviously adding more tests is beneficial - 
that goes
without saying. However, with !1607 [1] we would run the standardised 
tests for
downstream plugins in the BuildStream CI, which means the more standard 
tests
for plugins, the more sure we can be that a change won't be causing a 
breakage
_somewhere_.

================================================================================

Now in order to standardise element tests, I think we need to classify the
types of plugins that are around at the moment. To my mind, all the element
plugins I have seen fall into four somewhat rough categories. These are:

* BuildElements
  - These are elements which just run the standard commands for a given 
build
    system
  - Generally run on some source code and produce an artifact
  - Popular examples include the `cmake` and `autotools` plugins
* Compose-like Elements
  - These elements take the artifacts of their components and combine them,
    producing an artifact
  - The composition does not need to be simple, examples other than 
`compose
    could be `flatpak_image`, `flatpak_repo`, `x86image` and any other 
element
    that produces an "image-like artifact"
* Metadata Collection Elements
  - Elements that scrape the metadata of their dependencies in order to get
    some information about the build or the components into the finished
    artifact
  - Generally produce an artifact
  - Examples are `collect_integration` and `collect_manifest`.
* Test elements
  - Elements that perform some kind of test on either the dependencies or
    actually inside the sandbox.
  - Don't produce artifacts, as a general rule
  - Example could be `check_forbidden` in freedesktop-sdk

Of these 4 basic types, I think that only BuildElements and Compose-like
elements are able to be tested in a standardised way. There are some odd
elements that don't fit any category, like `stack` or `junction`, but I 
suspect
these are limited to core plugins.

For BuildElements, there are a few standard tests that I can think of. 
Firstly,
we already almost perform standardised tests on elements with those for
resolving variables, which have a secondary purpose of checking defaults 
don't
change inadvertently. In short I think we could run:

* Bash syntax checks on{configure,build,install}-commands
* Variable loading checks (as we already do)
* Build + checkout test
  - Standardising this may be slightly nontrivial, we'd have to essentially
    make sure the test project (I think a "hello world") installs 
something into
    the same location for each one
  - An element wanting to be standardised would have to provide it's 
own test
    project.
* Build + run test
  - Basically the same as build + checkout
  - I'd propose that all projects build a `/usr/bin/hello` binary, that 
prints
    "Hello World!"

Note that these would necessarily require a project with a base image that
contains all of the buildsystems they use. I have another proposal for this
that is better suited to a separate email.

Compose-like elements could have similarly standardised tests for resolving
variables and build + checkout, but don't really have a standardised 
"run" in
in the same way. I suppose there may be a subset that subclass from a
`ScriptElement`, like the `x86image` plugin, which could also have bash 
syntax
checked.

===============================================================================

This is only a high-level idea of what kind of standardised tests could 
be run
on elements, at least as a start, and is by no means comprehensive. There's
also a very real possiblity that my analysis of "plugin categories" is 
missing
something.

As for implementation itself, I would have thought we could do something 
like
the `register_repo_kind()` function, only taking arguments for the elements,
for example a URL to the "Hello World" project, the type of said project 
source
and so on. We can then have a hook to transform this data into an 
element. We'd
also need config on the "category" of element that we're dealing with, 
or maybe
a more granular configuration of which tests *can* be run.

The main purpose of this is to get the ball rolling on some kind of
standardised element tests, so if there are more ideas for possible tests or
a better picture of how to implement this then please add away.

Cheers,

Tom


[0]: 
https://mail.gnome.org/archives/buildstream-list/2019-August/msg00017.html

[1]: https://gitlab.com/buildstream/buildstream/merge_requests/1607