Re: [BuildStream] [Proposal] Plugin fragmentation / Treating Plugins as Sources

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Fri, 2019-04-26 at 17:02 +0200, Sander Striker via buildstream-list
wrote:
> Hi,
>
> We seem to be spending a lot of energy on this topic, in particular
> on a certain aspect.

Hi Sander,

Thanks for trying to bring some clarity here.

> I'm going to propose we get a bit more incremental here, leaving out
> a lot of detail as this thread contains too much to quote from.  I
> would like to refer back to Tristan's proposal when it comes to the
> detailed description of what the documentation would look like, the
> break down of plugin domains:
>
> 1)  Move _all_ plugins that are not hardwired in (import, filter,
> junction, ...) out to _one_ external repository.  The documentation
> would reflect a single location for all the plugins.  Ensure that
> manually retrieving these plugins and loading them through the plugin
> local or pip origin works.  

This would work for me, although it is less optimal compared to just
not moving any of our upstream plugins outside of the core at all.

It creates a situation where the user only needs to ever install one
additional package, and project specific plugins can be loaded from
pip/local origins.

Plugins can still be installed on hosts which don't have the respective
dependencies, so even if we start supporting windows/osx/linux specific
plugins in the same repository, they should be able to safely bail out
at preflight() time with a sensible error.

I think the only real issue with this is pylint, we would probably want
to fix it upstream so that we can selectively lint files depending on
whether they are expected to pass linting on the given host.

With that said, why move plugins outside of BuildStream at all if we
are going to just keep the blessed plugins in a single repo ?

> 2)  Split the plugins into domain specific repositories.  The
> documentation will reflect the different categories, and where to
> retrieve the plugins.    Ensure that manually retrieving these
> plugins and loading them through the plugin local or pip origin
> works.

This doesn't work for me.

Once we split up the plugins into multiple domains, it means:

  * Users need to have the knowledge of plugins are required
    communicated to them out of band, making the experience generally
    more painful for the typical user who wants to start contributing
    to a given project.

    While there is still names encoded in project.conf, the knowledge
    of how to obtain these packages and where they come from is
    still a nuisance and needs to be carried - this is much reduced
    in the current situation where our upstream plugins live directly
    in core, and is not immensely bad if we have all blessed plugins
    in a single repo.

  * We multiply the tedious work that our downstream distro maintainers
    are generously doing for us, some of which have already complained
    about this move (and also highlighted that we will be much less
    useful by not providing an experience where almost anything at
    least on linux can be done "out of the box" with a single install).

Further, it doesn't fully solve the problem of arguing which plugins go
into which package. I might still find myself arguing in favor of
scenarios where the user needs to install the least separate packages
in order to work on a given project. I think these debates are not
worth having if we just have a technical solution which makes it
painless.

> 3)  Assess if we need to do anything more at this point.  Does the
> existing plugin origins solve loading plugins well enough for the
> time being?  Do we need to evolve the way we load plugins (by adding
> another origin)?

Exactly, I cannot get behind (2), which is why I am proposing (and
volunteering to do the work) to evolve the way we load plugins in such
a way that the full knowledge of where plugins come from are encoded
directly into project data, and BuildStream can obtain them on behalf
of the user who invoked BuildStream.

This would solve all my problems with the fragmentation of plugin
repositories we are aiming for.

This change would however have further benefits, i.e. if we have this
level of convenience, why stop at domain specific categories ?

We could fragment plugins maximally instead, and have a situation that
is more flexible for projects/users (users can select more specifically
which plugins they want on a per-plugin basis), and more freedom on the
development side (we can evolve plugins separately, without needing to
keep a repository of plugins in sync by distributing them together).

Cheers,
    -Tristan