Re: [BuildStream] Clean separation of BuildStream 1 & 2

[Benjamin Schubert <contact benschubert me>]

Hey,

Sorry I'm a bit late to the party, as I realize a MR is already up for this.

TLDR: I think we should avoid renaming everything, as it will bring

more pain to us as maintainers and developers of the project, and to users,

who will need more changes and tweaks than would be necessary when updating.

I think we can keep using standard tools to handle versioning, which will

prevent headaches for testers and maintainers of projects, as those tools are

well known and documented.

More details inline.

On Sun, Mar 31, 2019 at 12:50 PM Tristan Van Berkom via
buildstream-list <buildstream-list gnome org> wrote:
>
> Hi all,
>
> As we have decided to break API and start work on BuildStream 2 (as
> announced in this blog[0]), we need to make some minor changes to
> ensure clean separation.
>
> My goal is basically to ensure that both versions are parallel
> installable and we don't leave any margin of error for users to
> experience any bad paper cuts if they do decide to try and help us
> test BuildStream 2.
>
This is already a solved problem in the python ecosystem by the mean of virtual

environments. If someone wants to test BuildStream 2, the only way I see they

could do it would be to install it from pip/source. I don't see BuildStream 2

packaged in any distribution before it being stable.

That means that the user will need to know at least the very basics of pip, and

thus requiring a virtual environment for the to install it in is, I think

reasonable, as the official python guidelines

(https://packaging.python.org/guides/installing-using-pip-and-virtualenv/) are

to use one in order not to break anything.

Moreover, we also publish nightly docker images in addition to those for stable

releases and development snapshot, making it a one liner to test any new

changes completely isolated, in a setup we more or less guarantee will work.

Given the two above points, I don't think ensuring both can be installed in the

same environment is neither needed nor something we should encourage.

I know this has already been agreed on, but I will still note

something around it:

there is actually no need to do this if, we use alpha and beta releases on pypi.

In that case, we could have releases graduate at the same time, and would

prevent any problem from happening. We could also have dependency requirements

on the plugins stating which version(s) of BuildStream the plugin is

compatible with, which would prevent any accident.

> It is possible that some of the plugins in bst-external are ready to be
> stable, but using the bst-experimental repo right away has some
> advantages:
>
> * Allows us to act immediately without discussing ultimate final
>   destinations for the existing plugins.
>
> * Allows us to start developing a story/policy for how plugins can
>   graduate to another more blessed/stable repository (e.g. something
>   like bst-plugins-good).
>
>
> Public python API namespace
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> In order for the two python modules to be parallel installable, and
> also in order to ensure that plugins do not accidentally import the
> wrong BuildStream API, I suggest that we rename the main `buildstream`
> module to `buildstream2`.
>
> This implies some minor changes in BuildStream master, and it implies
> that any plugins targetting BuildStream 2 will have some small changes
> to make in their imports, e.g.:
>
>    from buildstream2 import Element
>
Why do we even want both to be parallel installable? In order to test,

we can use virtualenvs/docker as stated before. Once released, projects

should move to the new version, which doesn't break anything on projects

(only on plugins), and thus we wouldn't even need both to be installable

in parallel.

>
> Identify API version in Plugin base class
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> In order to ensure that the BuildStream core does not load any plugins
> which are not written for the intended API surface, I propose that we
> add a new private field describing the API version to the base Plugin
> class.
>
> Currently the plugin entrypoints consist of a "setup" function which
> takes no arguments and returns a Plugin derived class, and this
> approach allows us to add the needed security without requiring any
> additional changes to existing plugins.
>
> When loading a Plugin, the BuildStream core will now check that the
> loaded Plugin's API version matches the one which is expected by the
> BuildStream core - this will allow us to fail early in the case that
> the user is running BuildStream 2 and specifies a plugin which is
> written for BuildStream 1 (and vise versa).
>
> It is possible that we might do this with an `isinstance()` check
> instead - but including the API version will allow us to create a more
> user friendly error message when the user accidentally tries to load an
> incompatible plugin in their project.
>
>
Wouldn't we be reinventing the wheel here? Python supports lower and upper

version bounds for packages to be installed together. Can't we just have

"bst_external" have a buildstream>1.0<1.9 requirements and fix the problem

that way? This seems to add a lot of overhead and I'm not sure I can see the

gains. Did we get people asking for this?

Should we then also add "artifact-version", because we might not want

artifacts not from the correct version nor people building with another version

since they would not get the advantages of the artifacts?

What I'm not sure about here is, do we need it? Is there anything in that space

that changed? If not, we should probably not add it.


> Rename default user configuration file
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> In order to ensure that we do not confuse configuration files which
> support different features and formats, we should just rename the
> default location where we look for configuration files.
>
> In master, we will now search for ${XDG_CONFIG_HOME}/buildstream2.conf
> instead of ${XDG_CONFIG_HOME}/buildstream.conf
>
> I don't think we need to do more than this for the config file, while
> it is possible for the user to specify the wrong config file on the
> command line, I think that the existing errors will be informative
> enough that the user would realize they are using the wrong config file
> (and I don't like the alternative of forcing the user to specify a
> "version" in the content of the config file itself).
>
>
That's the point (after renaming the python project buildstream2), I

think I disagree

the most with. My concern here is, imagine a user, on fedora, using BuildStream

to do some gnome development. I do a system-upgrade, get a new version of

BuildStream and suddenly all my caches are gone! What happened? I got a new

version. Since the format of our configuration is _stable_, we should simply

continue using the same name, and "newer" addition should just get

ignored as they are now.

> Optionality of the main entrypoint name
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> I think that we all mostly prefer to type `bst` on the command line and
> would not like to have to write `bst2`, for this reason I do not want
> to force the main entry point script to change.
>
> Instead, I would like to add an option to the `setup.py` script to
> allow it to either be named `bst` or `bst2`.
>
> If you regularly use BuildStream now for an active project which uses
> BuildStream 1 (like freedesktop-sdk or gnome-build-meta), but would
> like to help us out in testing the new BuildStream 2, then you will
> have the option to install it as `bst2` and can run either one without
> any fuss reinstalling.
>
>
Or, you could run BuildStream v2 isolated in another venv and never have any

problem. Anyways, what would happen if BuildStream 2 had to get a version of

a dependency not compatible with BuildStream 1? That scheme would break

immediately.

Cheers,

Benjamin