Re: [BuildStream] [Proposal] Plugin fragmentation / Treating Plugins as Sources

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi William,

On Tue, 2019-04-16 at 16:24 +0100, William Salmon via buildstream-list wrote:
> I didnt mean to sen this to the list yet but i dont think its too far 
> off so please just ignore the first line
>
> On 16/04/2019 16:17, William Salmon via buildstream-list wrote:
> > Could i get some feed back on this please. I have tried to keep it 
> > much more focused than last time but it seems to have ended up being a 
> > bit of a ramble again...
> >
> > On 11/04/2019 13:35, Tristan Van Berkom via buildstream-list wrote:
> > > Hi all,
> > >
> > > I'm afraid this is going to be a long and beefy proposal, I have
> > > included a couple of TLDRs inline for those who want to gloss over
> > > this.
> >
> > Indeed, it is very long. I am very interested in this topic and have 
> > tried to engage with this thread several times. But I have struggled 
> > to keep track of the thread and I must confess to not really 
> > understanding the details of your proposal. It is not only myself that 
> > dose not have the capacity for this email. Other reply seems to have 
> > said a similar thing.

So my initial proposal was a fully detailed action plan with some TLDRs
for those not interested in details, I wanted to demonstrate exactly
what would be implemented with the `git` origin and show that I had a
clear implementation plan which was easy enough to execute.

My initial response to Chandan was a bit off; I.e. I did not understand
his intention that every plugin would live in a *separate* venv
namespace (which I had considered before but wrote off completely as it
seems insanely complicated and feasibility improbable, however others
seem to think it's worth a shot so I'm not against it).

That reply may have complicated the beginning of the thread, sorry for
that.

I think that if you pick up the thread from Adam Coldrick's replies it
might be an easier read, as it speaks more of the pros and cons of
either treating plugins as source files which we can download, or
treating them as python packages which we install into virtual
environments.

    https://mail.gnome.org/archives/buildstream-list/2019-April/msg00037.html
    https://mail.gnome.org/archives/buildstream-list/2019-April/msg00039.html
    ...

Also, remember that whether BuildStream core *itself* is API stable or
not is not very relevant, as soon as BuildStream 2 is stable, or even
once it starts being used in the wild, people will use plugins.

Plugins will be either API stable themselves, or they wont be, and
there is no restriction we place on users about which plugin to use.

So this discussion revolves largely around how a future will play out
when the user wants to use some plugins which are stable and some other
plugins which are not, and how to ensure that we can safely use these
plugins in a single interpretor environment, because we need to load
all plugins from each (junctioned) project into the same environment
and same process.

> > Before this thread got too technical I would like to have had a 
> > opportunity to have joined the architectural discussion.
> >
> > From my perspective, as a user of may open source projects, I do not 
> > expect my OS to give me cutting edge versions of programs and I really 
> > don't expect it to have multiple versions of the same software so I am 
> > really confused as to why build stream is trying so hard to go so far 
> > above and bond when we have plenty of big challenges already.
> >
> > As a user of buildstream at work and occasionally at home I have been 
> > running bst 1.2 and bst master side by side for different projects for 
> > months in separate venv's.
> >
> > I have found this very easy to set up, and use. But I am very familiar 
> > with python, (maybe not by this groups standards but certainly by our 
> > I have target users standers)
> >
> > When I contrast this to other opensource projects, even those with 
> > sponsorship and a supportive user base that I have used personally, 
> > were trying out master close to the release  date requires compiling 
> > them for a hour or two i think the venv model works really well.
> >
> > For people who really want to have a system wide bst 1.2 and master 
> > then there are other solutions like docker or vm's that also work well.

Ok so, I'll say it up front so that there is no misunderstanding:

   I think it is incredibly irresponsible towards end users to even
   fathom a situation where you are creating software which *requires*
   end users to manage a venv just in order to install your software.

   I realize that I am calling out basically the majority of the python
   community by saying that venvs are just a poor excuse to be
   irresponsible about API stability, and using the venv excuse is just
   externalizing the problem upon users, instead of just guaranteeing a
   seamless install and use experience.

With that off my chest, I should say that this is rather orthogonal to
the discussion at hand - because even if we forced users to manage
which version of BuildStream worked with which version of bst-external
and which project at which version of project, in separate venvs for
each project that a user uses BuildStream for - there still arises the
issue of junctions.

For a more concise explanation about why having users manage venvs for
different, API unstable plugin collections is not only irresponsible
towards users but cannot even practically work (because junctions), see
this post:

  https://mail.gnome.org/archives/buildstream-list/2019-April/msg00026.html

And search for "I consider the `pip` origin to be dangerous", start
reading from there.

Cheers,
    -Tristan