[BuildStream] Clean separation of BuildStream 1 & 2

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi all,

As we have decided to break API and start work on BuildStream 2 (as
announced in this blog[0]), we need to make some minor changes to
ensure clean separation.

My goal is basically to ensure that both versions are parallel
installable and we don't leave any margin of error for users to
experience any bad paper cuts if they do decide to try and help us
test BuildStream 2.

This email is a descriptive summary of what the specific changes are,
and why they should happen.

Note that while there is a lot of consideration that went into this and
there are a lot of details, the actual changes are really not many and
not large, and I will be happy to undertake the actual work.

I do however suggest that we block the first development snapshot of
BuildStream 2 (probably numbered 1.91.0) on making these changes first.

If you have any concerns or objections to the changes listed here,
please feel free to raise then here :)

Cheers,
    -Tristan


Forking of bst-external
~~~~~~~~~~~~~~~~~~~~~~~
As Javier raised in the last IRC team meeting[1], bst-external has
ended up being packaged which means it now must retain a stable API.

The bst-external plugins have been mostly compatible with both versions
of BuildStream but we knew this wouldn't last forever, indeed now that
we have landed the YAML new world order[2], some of the bst-external
plugins are broken again as a result (plugins no longer allowed to
treat the node handed to Plugin.configure() as regular dictionaries).

At this time I think we should fork bst-external under the name
bst-experimental, this name will more clearly reflect the instability
of the plugins package - and we should make it clear in the README that
this is a location for sharing plugins which dont offer API guarantees
in the time that they become mature and find a more permanent place to
live.

It is possible that some of the plugins in bst-external are ready to be
stable, but using the bst-experimental repo right away has some
advantages:

* Allows us to act immediately without discussing ultimate final
  destinations for the existing plugins.

* Allows us to start developing a story/policy for how plugins can
  graduate to another more blessed/stable repository (e.g. something
  like bst-plugins-good).


Public python API namespace
~~~~~~~~~~~~~~~~~~~~~~~~~~~
In order for the two python modules to be parallel installable, and
also in order to ensure that plugins do not accidentally import the
wrong BuildStream API, I suggest that we rename the main `buildstream`
module to `buildstream2`.

This implies some minor changes in BuildStream master, and it implies
that any plugins targetting BuildStream 2 will have some small changes
to make in their imports, e.g.:

   from buildstream2 import Element


Identify API version in Plugin base class
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In order to ensure that the BuildStream core does not load any plugins
which are not written for the intended API surface, I propose that we
add a new private field describing the API version to the base Plugin
class.

Currently the plugin entrypoints consist of a "setup" function which
takes no arguments and returns a Plugin derived class, and this
approach allows us to add the needed security without requiring any
additional changes to existing plugins.

When loading a Plugin, the BuildStream core will now check that the
loaded Plugin's API version matches the one which is expected by the
BuildStream core - this will allow us to fail early in the case that
the user is running BuildStream 2 and specifies a plugin which is
written for BuildStream 1 (and vise versa).

It is possible that we might do this with an `isinstance()` check
instead - but including the API version will allow us to create a more
user friendly error message when the user accidentally tries to load an
incompatible plugin in their project.


Add "version" to project.conf, in addition to "format-version"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The new "version" parameter in project.conf will allow us to bail out
early if the user tries to run BuildStream 2 on a project written for
BuildStream 1, and vise versa - but it will also allow the user to
specify the version more intuitively.

This is a fairly straight forward change I've been wanting to do for a
long time regardless of the API break.

Currently the project.conf "format-version" is a numeric version which
is different from BuildStream release versions. This granularity of
"format-version" is still useful for early adopters who want to use a
bleeding edge version of BuildStream before a new feature release
exists to depend on, but is also not very intuitive.

It will be more intuitive (and easier to use I think) to specify a
version which matches the numeric version reported by `bst --version`.

To express what I mean by this, here are some examples:

  # Require BuildStream 1, only the features included in the initial 1.0 release
  version: 1.0

  # Require BuildStream 1, requires features included in the 1.2 release
  version: 1.2

  # Require BuildStream 2, only the features included in the initial 2.0 release
  #
  # This will be called "2.0" immediately in master, even if we are a
  # long way until the first 2.x release.
  #
  version: 2.0


Technically this adds API, I am rather ambivalent as to whether we
actually support the new "version" semantic in BuildStream 1, but at
minimum we should enhance the next `1.2.x` release to include a
sensible error message if the "version" is specified.

The "version" configuration should be required and mandatory for any
project.conf in master and any later versions.


Rename default user configuration file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In order to ensure that we do not confuse configuration files which
support different features and formats, we should just rename the
default location where we look for configuration files.

In master, we will now search for ${XDG_CONFIG_HOME}/buildstream2.conf
instead of ${XDG_CONFIG_HOME}/buildstream.conf

I don't think we need to do more than this for the config file, while
it is possible for the user to specify the wrong config file on the
command line, I think that the existing errors will be informative
enough that the user would realize they are using the wrong config file
(and I don't like the alternative of forcing the user to specify a
"version" in the content of the config file itself).


Optionality of the main entrypoint name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I think that we all mostly prefer to type `bst` on the command line and
would not like to have to write `bst2`, for this reason I do not want
to force the main entry point script to change.

Instead, I would like to add an option to the `setup.py` script to
allow it to either be named `bst` or `bst2`.

If you regularly use BuildStream now for an active project which uses
BuildStream 1 (like freedesktop-sdk or gnome-build-meta), but would
like to help us out in testing the new BuildStream 2, then you will
have the option to install it as `bst2` and can run either one without
any fuss reinstalling.


[0]: https://blogs.gnome.org/tvb/2019/03/04/buildstream-news-and-2-0-planning/
[1]: https://wiki.gnome.org/Projects/BuildStream/Monthly-Meeting/20190319/
[2]: https://gitlab.com/BuildStream/buildstream/merge_requests/1257/