Re: [BuildStream] Thoughts around plugins (Too long, please read carefully)

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi,

Sorry I started this subthread and then sort of fell off the planet for
a while...

On Thu, 2018-11-22 at 16:27 +0000, Sander Striker via BuildStream-list wrote:
> Hi,
[...]
> > So while this is a very interesting idea, I think the complexity involved
> > is high, and the effort required is astronomical and would scale with the
> > number of plugins which end up being written.  It's much safer, IMO, to
> > provide a kind of lookup site for plugins, but not actually deliver or version
> > them through that.  So, for example, someone might run a putative web search
> > for "buildstream pijul source plugin" and come to a page on buildstream.build
> > which explains what it is and where to get it from, along with a suitable
> > submodule configuration.
>
> Maintaining such an index _well_ is also significant effort, and has the potential for similar scaling 
> properties.

I admit this wasn't a thoroughly thought out plan.

I did mention taking ownership of the plugin distribution story,
similar to how rust & cargo works, but indeed this can mean maintaining
the index ourselves.

When writing this, I was indeed more interested in the aspect of taking
over the functionality of how plugins are obtained, with an eye towards
making life easier for users who would not need to install various
plugin packages separately.

I wonder if there is any need at all to have any official source or
index, or to do anything beyond saying "This is a plugin and it can be
obtained with a Plugin source".

It could result in forcing us to keep the git plugin logic in
BuildStream proper, and having a git driven Source with a specific use
case of downloading plugins; and it might only mean forcing plugin
maintainers to make their plugins available in git repositories.

> > One alternative to submodules could be to include a repo URI and ref in the
> > plugin stanza in the project.conf though, and let BuildStream go ahead and
> > fetch the plugin.  I think it's entirely up to the plugin to be sensible
> > and alter its input into the cache key if it changes incompatibly though.

Yes, I am liking this appraoch...

> The cache keys part is fairly crucial.  I understand Tristan's
> thought here, the only way to be truly sure about artifacts is to
> capture all the inputs in the cachekey.  And here a plugin is an
> input.  That does lead to the problem of instantly invalidating
> caches when a plugin gets updated (or at least double storage
> requirements).
> Where do we put the trust here?  Do we capture the plugin version and
> potentially checksums in the artifact provenance, so that we at least
> know what version of a plugin an artifact was produced with?  Or are
> we not trusting at all, and do we invalidate caches, potentially more
> than desirable?

Interestingly this seems to be an opportunity to place the choice of
trust in the hands of project authors, which may make a lot of sense as
different plugins might have different guarantees about stability and
conforming to the format and artifact versioning.

It's easily conceivable to have a `git` Source for the purpose of
obtaining plugins which optionally ignores the git commit sha in it's
`Plugin.get_unique_key()`, simply returning a constant for the unique
key would have the effect of "trusting the plugin maintainers" to
follow the format and artifact revisioning policies.

Mostly just throwing some ideas out there and don't expect this to
materialize soon, but it could be a good and friendly addition to make
a transition where plugins are moved around more user friendly for the
next cycle...

Cheers,
    -Tristan