Re: [BuildStream] [Proposal] Plugin fragmentation / Treating Plugins as Sources

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Adam,

On Mon, 2019-04-15 at 10:16 +0100, Adam Coldrick via buildstream-list wrote:
> Hi,
>
> Horrible Python hacks are one of my favourite subjects :) .
>
> TL;DR/Summary:
>
> Its possible to override Python's import mechanisms to achieve this
> (multiple versions of a library used in one process). A working (in
> Python 2.7, but trivially portable to 3.x) example of this is [0].
>
> I don't have much knowledge about PluginBase so I don't know if it
> provides similar utility without forcing plugin authors to use the same
> hack, or even how possible it would be to extend it to do so.
>
> I think it is worth the effort of at least looking into it; whilst the
> proposed `bst plugin fetch` idea solves one part of the problem solved
> by pip, I think forcing plugin authors to be limited to the standard
> library is too restrictive.

Again, I think the idea is "neat", but I should reiterate that external
imports is really very limited to a very small amount of plugins who
actually import external libraries.

There is no reason at all for an element to import anything external,
and only a small subset of plugins actually import external libraries.

It appears to be a lot of work, including a lot of host side venv
maintenance for all the edge cases, like when you upgrade your host
python interpreter as Mathieu points out in the sibling post, for
really not that much value add.

Then again, if we can really get it done and it really works well, we
could try to transition Source plugins to use python-only dependencies
as much as possible instead of calling out to host tools, for any cases
where you really dont need any backing host dependencies. Not sure how
feasible this is, though.

[...]
> I don't remember seeing a strong recommendation to use submodules when
> I was first experimenting with BuildStream a few months ago, though its
> entirely possible I didn't read the docs well enough. Even so, I don't
> really like having to set up submodules in my project just to provide
> plugins from an upstream source (the plugins might not be stable or
> marked as "blessed" by being in the main repo but they are still part
> of the BuildStream GitLab group and look upstream to me).

Yes, that was the accident which lead to `bst-external` becoming
defacto stable by virtue of it being packaged, indeed.

The messaging was not good enough around this.

> <snip>
> >   A) If we ensure that there is only `bst-plugins-good` plus whatever
> >      local plugins a project uses, then we don't have this problem.
> >
> >      But then there is not much point in moving the plugins outside
> >      of BuildStream at all.
> >
> >      I am really fine with this, but due to the (completely
> > irrational)
> >      desire I've seen around wanting to strip out the important
> > ostree
> >      plugin from the core even in advance of other plugins, I worry
> >      that trying to maintain the core plugins in a single repo is
> > only
> >      going to cause internal friction and arguments about what should
> >      be included or not.
>
> I agree with this, I'm not sure of the point of just moving all the
> plugins into a plugins repo.
>
> >   B) If we fragment plugins maximally, as stated above, we need
> >      BuildStream to automate the operation of obtaining them.
>
> +1. However, I'm interested to know what you see as the benefits of
> maximal fragmentation, rather than having repositories/plugin
> "libraries" which provide groups of related plugins?

Say I use API stable versions of a lot of the upstream plugins
maintained in the same repo... but now I want to use a modified/forked
plugin for *one* of the plugins from that same repo.

Basically maximal fragmentation allows more flexibility from both the
maintenance and user experiences.

Also, I just mostly think it's messy to have something in between.

I.e. it's nice to have all the core plugins in one repo where they are
easy to find, or it's nice to have a single index in our documentation
linking to the docs of each individual plugins that someone can use.

I wouldnt like to get into a scenario where some plugins happen to live
in the same repo, and others are fragmented, also because I worry this
again leads to pointless debates about what plugins belong in which
repo.

[...]
> I understand that the plugins are currently expected to indicate
> missing dependencies rather than blindly assume they're present, but in
> my experience of getting started with BuildStream the part where I
> needed to install a bunch of tools to make plugins actually work was
> one of the most frustrating.
>
> I think its worth minimising this as much as possible, and its
> certainly possible to automate installation of Python dependencies. In
> general, I think "you need to install the following host tools for this
> plugin to use" is more expected than "you need to install the following
> host tools and also this set of Python dependencies".

Right, see my reply above, maybe if this works well we could try to
move in that direction.

Still I don't know how realistic it is, could we satisfy all the
dependencies of Source plugins with python implementations everywhere
(or even mostly) ?

> <snip>
> > > If we split up the current Core plugins into separate repositories,
> > > I'd have
> > > expected that each plugin will declare its own dependencies in its
> > > setup.py.
> >
> > Most dependencies cannot be installed via python anyway, requiring a
> > setup.py and pythonic package structure for all plugins just for the
> > sake of a few outliers seems to be pretty overkill.
>
> Is there really much pain associated with writing a setup.py and
> structuring things as a Python package? Especially if splitting plugins
> into sets of related plugins rather than 1 plugin == 1 repo.

I'm not hugely concerned with this honestly.

I do feel it is unnecessary, especially I do not like that plugin
packages need to explicitly declare their entry points in their
setup.py for the existing `pip` origin.

I also don't like that `pip` origin plugins are shared in the
environment/host due to them being installed as python packages,
instead of being installed in a BuildStream owned directory (but this
problem would go away if this venv proposal works).

Then again... later on the implication is that people should publish
them on PyPI, I think that is a bit much just for the sake of
maintaining a plugin.

[...]
> > We can certainly dictate that if you want to maintain a BuildStream
> > plugin via the `git` origin, that you must host it in a git
> > repository.
>
> This is true, but that is effectively dictating to someone how they
> should manage their source code if they want to produce a plugin that
> can actually be easily used by BuildStream users.

I don't think that is any less reasonable than dictating that they must
create a "python package", in any case with a python package, it either
needs to be published on PyPI (even more pain), or accessed with git,
as far as I can see.

[...]
> > BuildStream doesnt require either of the above: A BuildStream plugin
> > is
> > a python *file* which BuildStream carefully loads into an isolated
> > namespace (such that multiple, differing plugins of the same name can
> > exist in the same interpretor).
>
> A Python package is a directory containing Python modules. It is
> entirely possible to distribute a single Python module on PyPI. A good
> example is the bottle[1] framework. I don't think that adding a small
> setup.py alongside a single-file plugin is much overhead really.

Writing up a setup.py is one thing, but actually making releases on
PyPI seems like a lot more work, I expect most plugin maintainers would
just tell users to access their plugin via git, using the syntax
supported by pip.

Cheers,
    -Tristan