TL;DR I think we can move plugins out, still allowing users a decent user experience, by having project maintainers leverage existing VCS technology. And evaluating next steps after that.
On Fri, 2019-04-26 at 17:02 +0200, Sander Striker via buildstream-list
wrote:
> Hi,
>
> We seem to be spending a lot of energy on this topic, in particular
> on a certain aspect.
Hi Sander,
Thanks for trying to bring some clarity here.
I'm not sure it's helping yet :)
> I'm going to propose we get a bit more incremental here, leaving out
> a lot of detail as this thread contains too much to quote from. I
> would like to refer back to Tristan's proposal when it comes to the
> detailed description of what the documentation would look like, the
> break down of plugin domains:
It looks like I fumbled my sentence above. It should have said: "I would like to refer back to Tristan's proposal when it comes to the detailed description of what the documentation would look like, for example the break down of plugin domains."
The following were meant as incremental steps - I'm not sure if that point was lost based on the reply.
> 1) Move _all_ plugins that are not hardwired in (import, filter,
> junction, ...) out to _one_ external repository. The documentation
> would reflect a single location for all the plugins. Ensure that
> manually retrieving these plugins and loading them through the plugin
> local or pip origin works.
This would work for me, although it is less optimal compared to just
not moving any of our upstream plugins outside of the core at all.
It creates a situation where the user only needs to ever install one
additional package, and project specific plugins can be loaded from
pip/local origins.
Plugins can still be installed on hosts which don't have the respective
dependencies, so even if we start supporting windows/osx/linux specific
plugins in the same repository, they should be able to safely bail out
at preflight() time with a sensible error.
I think the only real issue with this is pylint, we would probably want
to fix it upstream so that we can selectively lint files depending on
whether they are expected to pass linting on the given host.
With that said, why move plugins outside of BuildStream at all if we
are going to just keep the blessed plugins in a single repo ?
That would be the status quo. I could have made the incremental steps start with "0) Do what we do today", but that didn't seem useful.
> 2) Split the plugins into domain specific repositories. The
> documentation will reflect the different categories, and where to
> retrieve the plugins. Ensure that manually retrieving these
> plugins and loading them through the plugin local or pip origin
> works.
This doesn't work for me.
Once we split up the plugins into multiple domains, it means:
* Users need to have the knowledge of plugins are required
communicated to them out of band, making the experience generally
more painful for the typical user who wants to start contributing
to a given project.
While there is still names encoded in project.conf, the knowledge
of how to obtain these packages and where they come from is
still a nuisance and needs to be carried - this is much reduced
in the current situation where our upstream plugins live directly
in core, and is not immensely bad if we have all blessed plugins
in a single repo.
This will work with local origin in combination with git submodules, svn externals, or mercurial subrepositories - whatever VCS the project has chosen.
Regardless of ones like or dislike of this functionality, it is common, and it does work. And if a project maintainer tells her users to git clone --recurse-submodules or to run git submodule update --init, the project consumers will use that, and be able to work beyond that.
* We multiply the tedious work that our downstream distro maintainers
are generously doing for us, some of which have already complained
about this move (and also highlighted that we will be much less
useful by not providing an experience where almost anything at
least on linux can be done "out of the box" with a single install).
The same answer applies here I think. I don't see why a distro maintainer would have to do anything.
BTW, it would be useful if said distro maintainers would raise their points here. Are we talking about 1? 2? 16? Which ones? It's very hard to understand whether to interpret complain as "grumbled" or as "are up in arms".
Further, it doesn't fully solve the problem of arguing which plugins go
into which package. I might still find myself arguing in favor of
scenarios where the user needs to install the least separate packages
in order to work on a given project. I think these debates are not
worth having if we just have a technical solution which makes it
painless.
I'm not sure what you're getting at. Maybe I didn't portray the granularity of fragmentation correctly here? I honest see splitting each plugin out either into its own repo or into a domain repo as incremental substeps of step 2. I don't think anyone is arguing about how to fragment, or if someone is, I missed it(?)
> 3) Assess if we need to do anything more at this point. Does the
> existing plugin origins solve loading plugins well enough for the
> time being? Do we need to evolve the way we load plugins (by adding
> another origin)?
Exactly, I cannot get behind (2), which is why I am proposing (and
volunteering to do the work) to evolve the way we load plugins in such
a way that the full knowledge of where plugins come from are encoded
directly into project data, and BuildStream can obtain them on behalf
of the user who invoked BuildStream.
That isn't very incremental. The user can still have a decent experience without BuildStream doing the work, but leaving that to the VCS. What the user would miss out on is the plugins strict origins being captured in the project metadata. Which we could get to at a later point, just like the more domain specific problem of provenance in artifacts has been deferred.
This would solve all my problems with the fragmentation of plugin
repositories we are aiming for.
I personally fail to see why we need to wait. I find the "people hate submodules" not enough of a technical argument. That projects take a decision to advise users to install plugins on their system, is their choice. If they think they can manage their user's base plugins the best way this way, then allow them to do so. If at some later point there is a better way,
[...more on fragmentation...]
Cheers,
-Tristan
Cheers,
Sander
