Re: [BuildStream] cache key changes

[Jürg Billeter <j bitron ch>]

> On Fri, 2019-07-05 at 15:00 +0100, Darius Makovsky via buildstream-list 
> wrote:
> [...]
> > * `artifact-version` is replaced by `core-version`
>
> I think this is wrong. If this is only a name change, I think it's more
> clear as `artifact-version`, as this is clearly *not* tied to the core
> BuildStream release version and very intentionally so.
>
> If this is a proposal to tie artifact cache keys directly to the core
> release version, then I think this is very wrong, as it would incur
> cache key changes where there is no need for them.

The proposal was only about a name change. I definitely agree that we
shouldn't tie artifact cache keys to the core release version.

Another option for the name would be `core-artifact-version`. That
said, I don't have a strong opinion on the name. If we write a proper
spec/documentation for the cache key format, it's also less of an
issue.

> [...]
>
> So, while it is certainly good to include the plugin name, it might be
> important to include the "plugin origins" configuration from the
> project in the cache keys as well, coupled together, these two pieces
> of information should ideally uniquely identify the plugins in use.

I'm not quite sure how this would work. If we were to simply hash the
origin of each plugin from project.conf, how does that help with
multiple 'local' plugins with the same name? Or what exactly would you
include in the cache key?

An option could be to allow specifying some form of source/reference
URL also for local plugins. Either as (optional) field in project.conf
plugins config or maybe as class variable in the plugin itself. I.e., a
unique identifier for plugins that developers must change when they
fork a plugin.

> If we include the plugin origins, we should probably *keep* the project
> specific key from Project.get_unique_key(). Delegate that contribution
> to the cache key to Project would both be more elegant than trying to
> do it all in Element, and also more optimal, as the Project could hash
> it's own checksum once and only contribute a string to every cache key.

Adding this to the project key would trigger rebuild of all elements
when adding a new plugin. Caching this as Element class variable would
make more sense to me.

Cheers,
Jürg