Re: [BuildStream] [Summary] Plugin fragmentation / Treating Plugins as Sources

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi William,

I really appreciate your mail, I think you are touching at the heart of
where some disagreements are stemming from in ways I was not able to
express clearly. More than this, I think you have recognized a possible
deep disagreement that I could not even fathom as a possibility.

I'll prefix this message with: I feel immensely strongly about creating
reliable and successful software, especially in a FOSS project where I
think we should hold ourselves to the highest standards, we are
developing for an ecosystem and basically targeting the whole world,
starting with upstreams and also product companies which create
products derived from upstreams. This means there are moving parts
which don't have any control over eachother, and for our tool to be
successful we need to be very, very reliable.

It is okay that people might have lower standards or lower expectations
of what we can achieve with BuildStream, I don't mean to judge, I just
wanted to prefix this message so people understand where I am coming
from.


I'll take this advantage to describe some of our original project goals
below, and how I've went to lengths to make sure things always work,
everywhere, after answering a couple questions below...

On Wed, 2019-04-24 at 12:49 +0100, William Salmon via buildstream-list wrote:
> On 19/04/2019 06:35, Tristan Van Berkom via buildstream-list wrote: 
[...]
> > The above scenario leaves maintainer of project (A) with no recourse, 
> > they must go to the length of porting either API stable plugin package 
> > FOO or BAR to use the same API of library BAZ. 
[...]
> Tristan 
>
> I don't understand why this is a issue peculiar to python. 

This is an issue that is particular to python because python libraries
do not even make any promise to be API compatible.

In fact that is not really true, historically there was, but a series
of unfortunate events (I guess) lead to the creation of virtualenv
which later became venv. Still there are upstreams which maintain
stable API responsibly in python, but in the current landscape we don't
really know this anymore, as everyone is now buying into the
"vendoring" approach (it is hard to draw a line between which third
party python dependencies intend to be stable and which do not, further
it can be considered acceptable for those intents to change with the
current vendoring).

Conversely on the host, especially when we're talking about the kind of
extensively scriptable CLI tools that we are likely to use to download
source code, a great deal of effort is made to ensure those tools
remain compatible and scripts which use them don't break from one
version to another, or from one host to another.

> If both project A and project B use sub-process to call a external
> program that should be on the path but different versions of that
> program have slightly different cli options or behaviours then you
> could get in to the same problem. 

The situation is indeed not perfect (my biggest worry is about
differing versions of `patch` for instance), and as this subject has
arose before, I am very keen on supporting projects declaring a sysroot
which provides the tools their plugins use, so that BuildStream can run
those in a sandbox in the same way we run builds, but with networking
enabled. This approach also allows projects to declare a sysroot which
has the tools *they* need, without imposing one on downstream
junctioning projects.

This idea was discussed in more than one gathering, and I think is a
much more robust approach than trying to say that all dependencies are
in pure python and we use venvs for plugin isolation (if that is even
feasible, which I think we're leaning towards not feasible).

> I don't think it is unreasonable to have some plugin's be
> incompatible with each other, we should strive to avoid this but do
> we really want to make this our responsibility to make this
> impossible?

It has always been our responsibility.

I'd like to point out that the *very first* code written in BuildStream
is the plugin loading mechanism which ensures that plugins are isolated
to a project (back then we were calling it a "pipeline"), for instance,
see this commit from november 2016:

    https://gitlab.com/BuildStream/buildstream/commit/1671daf03bd6fad7afba603188d782a332b57a9b

The whole reason we have PluginContext attached to the project and use
PluginBase (http://pluginbase.pocoo.org/) to load plugins, is that we
knew that we were going to implement junctions and need this separation
and freedom.

We knew that we were breaking the mono-repo trend for build metadata as
a clear goal.

For that to work, and to be able to build an open community around this
idea that "projects can depend on projects", we absolutely needed to
ensure plugins are neatly isolated to the projects which pull them in.

Consider the following scenario that we set out to support:

  * I maintain project a project called "Middleware"

  * I have downstream projects, different flavors of "Desktop", etc

  * I depend on an upstream project called "Base" which does the
    runtimes and bootstrapping

  * I have absolutely no control over how my upstreams and downstreams
    are maintained, or what versions or forks of plugins they choose
    to use in their projects.

The idea here is that at any level of the stack of separately
maintained projects, one can always choose when to buy into a new
version of a project they depend on, and it should be more or less
hassle free. Conversely, it means that I can modify my own project with
perfect freedom without forcing changes on my downstreams.

Compare this approach with yocto for example. With yocto you have these
"layers", but if you want to get the latest version of your upstream,
you need to carefully ensure your `.bbappend` files apply to recipes
correctly, the relationship you have with your upstream is painfully
intimate, such that any small change in the upstream requires work in
your rebase to adapt to.

We wanted to eliminate this hassle as much as possible, and say that it
is at least possible for an upstream that is maintained responsibly to
declare what elements in that project are stable API intended to be
cross-project depended on - in this case you are really just depending
on the artifact outputs of an upstream project.

This also plays into project options: We don't want brute force
overrides of how upstream elements are built to be possible, rather we
want the downstream to make a feature request to the upstream, and
cause the upstream to responsibly maintain an option; this is a
tradeoff of where we sacrifice some flexibility, and in return we gain
a workflow that is much more stable and reliable.

> If you have very complex plugin requirements that involve the need
> for different versions of host tools for different projects/plugins
> then as far as i can see neither venv's or the other proposals on
> this thread will help?
>
>
> My summary of this thread would be: 
>
>  * Tristan seems to wants all plugins to be compatible, this is a
> noble goal. 

Thank you for saying this, I want to stress that this is not a new
goal, it is in fact inherently a part of what BuildStream is for; to be
the reliable and robust choice for everyone.

I'll take it a step further and add to that:

* After installing BuildStream, you should just be able to build any
  project.

  In an ecosystem where many projects are built with BuildStream,
  any project maintainer should just be able to say: "download
  my project and build it with BuildStream", and it should really
  "just work".

  Of course, that ecosystem of projects will never materialize if we
  don't prove that we are reliable enough to support it up front,
  and in this delicate stage, any bad experience that a user encounters
  will hurt our reputation very much (the nature of reputations is that
  they are very hard to build, and very easy to destroy).

* A project that you write today should be equally buildable in
  10 years time and produce exactly the same results in 10 years
  time using the new version of BuildStream.

  We are breaking this for BuildStream 2, but really I hope that
  when the dust settles we will have the rock solid reliable tool
  that we set out to create.

>  * But in order to achieve it, we seem to be inventing very complex
> ways to manage this that seem to need a lot of work to create and
> maintain. 
>
>  * The plugin situation is not the best as it stands, but the
> suggested solutions are very expensive. 

If we continue with the status quo that BuildStream plugins are files,
and they do not have external python library dependencies, then it is
really not that expensive.

I don't really understand why this should be such a big issue, I
understand the *desire* to have that freedom, but I don't see how
having that freedom buys us much given what plugins actually do,
especially if any reliability and stability needs to be sacrificed on
the alter of developer freedom and convenience (that doesn't add up to
me at all).

> If we remove the requirement to have all plugins be compatible with
> each other, could we not come up with some much more simple
> improvements? 
>
> I see the need for this thread to explore the cost of having all
> plugins made compatible. As we can not make a good decision as to if
> the gain is worth the cost until we under stand the cost. But
> Tristan's response to Adams email makes it sound like it was decided
> that all plugins must be compatible, when i am not sure this is the
> case. 

Breaking API for BuildStream 2 was a decision we made together, I don't
think this decision of breaking API is an excuse to redefine what we
set out to achieve in the first place.

> I think Angelos' summary is helpful in pointing out how much extra
> work there will be in creating a `gold plated` solution but if we are
> to maintain the solution are we also going to have our CI enforce
> this? and how much more cognitive load are we going to add to
> developers and maintainers? 
>
> We have agreed that bst 2 will brake compatibility with bst 1.2 but
> have we agreed that all plugins must be compatible with each other?

See above comment, really I think that agreement is implicit when
starting to contribute to the project in the first place, as it is
clearly at the core of our goals, it is even reflected in the very
first commits to the project.

Breaking API is also a *temporary* thing, it is not an API that is to
be broken in permanence. When we seal 2.0 I hope we can develop 2.x
responsibly and stably and maintain it for a good 5 years at least, or
even 10 before having (or even wanting) to break API again (maybe we
will have have to break by default if python 3 is ever phased out and
obsoleted, in which case a symbolic break may be required if only for
the sake of plugins).


In closing, William has touched upon what I think is the driving
undertone of this whole thread, and recognized some differences of
opinion which I could not recognize.

I believe that we can still make BuildStream the rock solid reliable
tool that everyone can trust will always "just work" (frankly I feel
that it is not worth our collective and expensive time to develop
anything short of this, and I constantly make the assumption that
everyone here agrees on this), if we keep our eyes on the prize here,
we can definitely make it happen.

Cheers,
    -Tristan