Re: [BuildStream] [Proposal] Plugin fragmentation / Treating Plugins as Sources

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Sander,

On Mon, 2019-04-29 at 19:03 +0200, Sander Striker wrote:
> Hi,
>
> TL;DR I think we can move plugins out, still allowing users a decent
> user experience, by having project maintainers leverage existing VCS
> technology.  And evaluating next steps after that.

[...]
> It looks like I fumbled my sentence above.  It should have said: "I
> would like to refer back to Tristan's proposal when it comes to the
> detailed description of what the documentation would look like, for
> example the break down of plugin domains."
>
> The following were meant as incremental steps - I'm not sure if that
> point was lost based on the reply.

Aha, I read this 'incremental' as in, lets discuss the motives and
reasoning before getting into implementation details of a solution.

So yeah, sorry if this was completely lost on me, I read the following
points as possible avenues and not as incremental steps.


I have put some detailed replies below but let's start with a TLDR:

  - Short answer: When viewed as incremental steps, I could be happy
   
with this order of events, but will definitely want to implement
   
something along the lines of this proposal around (3).

    Without (3), I think this is quite a regression, in the sense that
    it is much less convenient than BuildStream 1 if we have to use
    git submodules for all plugins instead of having them available
    automatically with a BuildStream installation.

  - I worry a bit about churning plugin locations many times instead of
    just once. This would mean depending projects working with unstable
    BuildStream 2 will have to churn more as a result and will cause
    more pain.

    That said, it's not a huge concern for me if that's what it takes
    to move forward.

  - I think the "use git submodules" story was better before we created
    the "pip" plugin origin, because then we clearly dictated the terms
    of use (unstable plugins are only ever project-bound and never
    installation-bound, and installation-bound plugins were distributed
    only with the core).

    Based on how people have been using unstable plugins via the "pip"
    origin instead of safely loading them in per-project isolation,
    I worry that we are leaving too much opportunity for users to cut
    themselves on sharp edges, so to speak.

    That said, I can try to transition freedesktop-sdk/gnome-build-meta
    to use bst-external via local origin / git submodules and see what
    happens there.


[...]
> > Once we split up the plugins into multiple domains, it means:
> >
> >   * Users need to have the knowledge of plugins are required
> >     communicated to them out of band, making the experience generally
> >     more painful for the typical user who wants to start contributing
> >     to a given project.
> >
> >     While there is still names encoded in project.conf, the knowledge
> >     of how to obtain these packages and where they come from is
> >     still a nuisance and needs to be carried - this is much reduced
> >     in the current situation where our upstream plugins live directly
> >     in core, and is not immensely bad if we have all blessed plugins
> >     in a single repo.
>
> This will work with local origin in combination with git submodules,
> svn externals, or mercurial subrepositories - whatever VCS the
> project has chosen.

I should point out that users don't really have VCS freedom, or am I
able to use `svn externals` in order to use a plugin repository that is
maintained in git ?

Just pointing out that we cannot presume that maintainers of projects
and plugin repos are the same group (even if they often are), and I
don't really think this is an issue because I really think in 2019, the
VCS game is over and everyone starting new projects are using git
anyway.

> Regardless of ones like or dislike of this functionality, it is
> common, and it does work.  And if a project maintainer tells her
> users to git clone --recurse-submodules or to run git submodule
> update --init, the project consumers will use that, and be able to
> work beyond that.

Right, that's sort of what we tried but I don't think this is really
working out how it was intended (this might have been a side effect of
desiring to test out the workflow with the newly introduced
installation-bound `pip` origin plugin loading technique, though).

I have to worry that by not making project-bound plugin loading more
convenient than a "use git submodules" story, we are inviting users to
get themselves into difficult situations where multiple junctioned
projects depend on a common, API unstable set of plugins.

> >   * We multiply the tedious work that our downstream distro maintainers
> >     are generously doing for us, some of which have already complained
> >     about this move (and also highlighted that we will be much less
> >     useful by not providing an experience where almost anything at
> >     least on linux can be done "out of the box" with a single install).
>
> The same answer applies here I think.  I don't see why a distro
> maintainer would have to do anything.

This is a fair point and now I am no longer certain what is needed.

I think that the plugins which are installation bound need to be
packaged in general, that is what makes the most sense to me.

The core upstream plugins we have are maintained in an API stable
fashion, so they are the best candidate for system level installation
rather than project bound installation.

Now we could reach a story where our core plugins are fragmented and
not packaged, and we recommend all project maintainers to access these
plugins via local plugin origin / git submodules, but I feel that is
rather a step down from having them conveniently available with
BuildStream right ?

That said, if this is the story we recommend "for now" it would make my
proposal a more obvious step forward to take at any given time in the
future, it would just provide more convenience over the submodule
story.

> BTW, it would be useful if said distro maintainers would raise their
> points here.  Are we talking about 1?  2? 16?  Which ones?  It's very
> hard to understand whether to interpret complain as "grumbled" or as
> "are up in arms".

So far I have the mail of Matthieu to go on which I linked in the
original post:

   https://mail.gnome.org/archives/buildstream-list/2019-February/msg00059.html

Which even on it's own, I don't think we can write off as an outlier.

Of course there is the option of doing more of this on our side, but
really we should be aiming for BuildStream to be something that can be
maintained long term by a single person in their spare time (once all
is said and done and the dust starts to settle).
 
> > Further, it doesn't fully solve the problem of arguing which
> > plugins go
> > into which package. I might still find myself arguing in favor of
> > scenarios where the user needs to install the least separate packages
> > in order to work on a given project. I think these debates are not
> > worth having if we just have a technical solution which makes it
> > painless.
>
> I'm not sure what you're getting at.  Maybe I didn't portray the
> granularity of fragmentation correctly here?  I honest see splitting
> each plugin out either into its own repo or into a domain repo as
> incremental substeps of step 2.  I don't think anyone is arguing
> about how to fragment, or if someone is, I missed it(?)

I kind of have to find a way for existing projects to build with the
least hassle as possible, and there is a push towards removing ostree
before anything else:

   https://gitlab.com/BuildStream/buildstream/issues/977
   https://gitlab.com/BuildStream/bst-plugins-ostree/merge_requests/1
   https://gitlab.com/BuildStream/bst-plugins-experimental/merge_requests/1

I expect that once we do start talking about which plugins go where, I
will find myself arguing that the essential ostree plugin is a must for
a set of blessed plugins (nothing can really build without it as the
base import, perhaps docker source is a viable alternative but
currently ostree is the default for base imports).

[...]
> I personally fail to see why we need to wait.   I find the "people
> hate submodules" not enough of a technical argument.  That projects
> take a decision to advise users to install plugins on their system,
> is their choice.  If they think they can manage their user's base
> plugins the best way this way, then allow them to do so.  If at some
> later point there is a better way, 

I think this is an unfinished sentence... maybe you are alluding to the
possibility of implementing something more convenient at some point.

In any case yes I can agree with this, however I do have the opinion
that it is our responsibility to make the safest mode of operation the
most convenient one, so I am very keen on evolving the plugin origins
so that project-bound plugins are more obvious and easy to use than via
submodules.

I would be quite unhappy if we reach a 2.0 release where upstream core
plugins need to be loaded via git submodules, it seems quite a step
down from the convenient experience we have in 1.x with core plugins
coupled with the BuildStream installation itself.

Cheers,
    -Tristan