Re: [BuildStream] [Proposal] Plugin fragmentation / Treating Plugins as Sources

[Adam Coldrick <adam coldrick codethink co uk>]

Hi Tristan,

On Mon, 2019-04-15 at 20:45 +0900, Tristan Van Berkom wrote:
<snip>
> Again, I think the idea is "neat", but I should reiterate that
> external
> imports is really very limited to a very small amount of plugins who
> actually import external libraries.
>
> There is no reason at all for an element to import anything external,
> and only a small subset of plugins actually import external
> libraries.

This is the current situation, but it feels quite restrictive to
enforce that requirement on anyone considering writing a plugin in the
future. I don't really have an idea in mind of what kind of plugin may
want to do that, and maybe this assumption is invalid and nobody will
ever want external dependencies in a new plugin (I'd be interested to
hear what authors of plugins such as those in bst-external think of
this, since I don't have a ton of context really).

> It appears to be a lot of work, including a lot of host side venv
> maintenance for all the edge cases, like when you upgrade your host
> python interpreter as Mathieu points out in the sibling post, for
> really not that much value add.
>
> Then again, if we can really get it done and it really works well, we
> could try to transition Source plugins to use python-only
> dependencies
> as much as possible instead of calling out to host tools, for any
> cases
> where you really dont need any backing host dependencies. Not sure
> how
> feasible this is, though.

It does seem like it could be quite a lot of work, but I think that
this touches on the extent of the value that it would add. Anything
that reduces the number of things that someone installing BuildStream
using pip rather than a distro package seems good to me.

<snip>
> Say I use API stable versions of a lot of the upstream plugins
> maintained in the same repo... but now I want to use a
> modified/forked
> plugin for *one* of the plugins from that same repo.
>
> Basically maximal fragmentation allows more flexibility from both the
> maintenance and user experiences.
>
> Also, I just mostly think it's messy to have something in between.
>
> I.e. it's nice to have all the core plugins in one repo where they
> are
> easy to find, or it's nice to have a single index in our
> documentation
> linking to the docs of each individual plugins that someone can use.
>
> I wouldnt like to get into a scenario where some plugins happen to
> live
> in the same repo, and others are fragmented, also because I worry
> this
> again leads to pointless debates about what plugins belong in which
> repo.

That makes sense, and leads me to prefer the maximally fragmented
option. 

<snip>
> Still I don't know how realistic it is, could we satisfy all the
> dependencies of Source plugins with python implementations everywhere
> (or even mostly) ?

I expect certainly some (bzr for example) can be converted like this,
but I don't know whether *all* of them could (I didn't find a Python
lzip library for .tar.lz files when quickly looking around for
example).

<snip>
> I also don't like that `pip` origin plugins are shared in the
> environment/host due to them being installed as python packages,
> instead of being installed in a BuildStream owned directory (but this
> problem would go away if this venv proposal works).

Yeah, this combination of minimal user overhead (in dependency
management, or just installing the plugins themselves) and also control
for BuildStream is why I like the venv proposal.

<snip>
> [...]
> > > We can certainly dictate that if you want to maintain a
> > > BuildStream
> > > plugin via the `git` origin, that you must host it in a git
> > > repository.
> >
> > This is true, but that is effectively dictating to someone how they
> > should manage their source code if they want to produce a plugin
> > that
> > can actually be easily used by BuildStream users.
>
> I don't think that is any less reasonable than dictating that they
> must
> create a "python package", in any case with a python package, it
> either
> needs to be published on PyPI (even more pain), or accessed with git,
> as far as I can see.

I agree PyPI might be a bit arduous to require of everyone that wants
to write a plugin. `pip` does support installing directly from various
different VCS types, rather than just git though, see [0].

> [...]
> > > BuildStream doesnt require either of the above: A BuildStream
> > > plugin
> > > is
> > > a python *file* which BuildStream carefully loads into an
> > > isolated
> > > namespace (such that multiple, differing plugins of the same name
> > > can
> > > exist in the same interpretor).
> >
> > A Python package is a directory containing Python modules. It is
> > entirely possible to distribute a single Python module on PyPI. A
> > good
> > example is the bottle[1] framework. I don't think that adding a
> > small
> > setup.py alongside a single-file plugin is much overhead really.
>
> Writing up a setup.py is one thing, but actually making releases on
> PyPI seems like a lot more work, I expect most plugin maintainers
> would
> just tell users to access their plugin via git, using the syntax
> supported by pip.

If plugins are being written somewhere with CI, its pretty easy to
automate the PyPI release part as part of a CI pipeline run when a tag
is pushed or similar, but I understand this concern.

Plugin authors may prefer to tell users to access them using whatever
VCS they're using, but I feel like it would be nice to allow them the
option of using PyPI if that is their preferred method of distributing
their Python projects.

[0]: https://pip.pypa.io/en/latest/reference/pip_install/#vcs-support

Thanks,

Adam