Re: [BuildStream] [Proposal] Plugin fragmentation / Treating Plugins as Sources

[Chandan Singh <chandan chandansingh net>]

Hi Tristan,

Thanks for writing this up. As you say, it's a bit beefy so I haven't
had the time to digest all of it. But, here are a few comments based
on what I have gathered so far.

On Thu, Apr 11, 2019 at 1:35 PM Tristan Van Berkom via
buildstream-list <buildstream-list gnome org> wrote:
>       This proposal attempts to reconcile the above by stating that:
>       * Plugins should be split up into domain specific repositories
>       * Plugins should not be packaged at all

I think this is probably a stronger statement than we need. I'd have
thought it'd be something like "Plugins don't need to be packaged". Or
are you suggesting to remove the `pip` origin for plugins entirely?

>       * Plugins are source code, and BuildStream will automatically
>         obtain them in the same way that we track and fetch any other
>         source code, so that it is a painless experience for the user.

Yes, but that's not quite it. Plugins also have dependencies - both on
host tools and on Python packages. For example, the `git`/`bzr`
plugins depend on the corresponding host tools. And, plugins like
Docker source have dependency on python packages like `requests`.
Treating them as _just_ source code will put the burden of getting
their dependencies on users. Note that this can be somewhat mitigated
if plugins are packaged.

> Unlike other origins, I hope we would need to support the dictionary
> which defines which format version number of each plugin is required,
> this information I think is redundant if we already control the version
> which will be used via a git 'ref' of the plugin repository.

I like the idea of something like a `git` origin, but I have a
counter-proposal :) When I was thinking about it a while back, I was
actually leaning towards a `venv` origin. I think this might be better
for the following reasons:

  * We don't "bless" a VCS and people are free to use to whatever VCS
they wish to use.
  * Since we'd be installing them as Python packages, at the very
least we will be handling the python dependencies of the plugins, if
not system base dependencies.

The `venv` origin would be very similar to `pip` origin except that
instead of expecting the plugins to be installed already, it will
install them in a virtual environment that is controlled by
BuildStream. Since we control this virtual environment, users won't
have to know about it so it doesn't add any extra burden on them.

This won't really put any extra burden on the plugin maintainers,
except that they will have to ensure that the define correct metadata
in setup.py, as they do today. However, they won't need to publish it
anywhere as in the projects, plugin source can be defined something
like: `git+https://gitlab.com/BuildStream/source-plugins/bzr@COMMIT_ID/TAG`.

> For the purposes of tracking, I think it is important to perform
> tracking of plugins completely separately from elements and junctions.
>
> This would mean we would need to introduce a new command:
>
>    bst source track-plugins
>
> or even just:
>
>    bst track-plugins

> From a UX perspective, I think it might be good to have a separate
`bst plugin` command group, similar to `source` and `artifact` groups.
I can imagine `bst plugin list/show` and `bst plugin fetch` commands
also being useful in future.

 Cheers,
Chandan