Re: [BuildStream] Cache keys

[Jürg Billeter <j bitron ch>]

Hi Will,

On Wed, 2020-01-08 at 12:22 +0000, William Salmon via buildstream-list
wrote:
> Some time a plugin changes in a fully braking way and we need to bump 
> the version and change the cache key of every element built with or from 
> the plugin. I have come to believe this to be true for bst unless we 
> make some huge changes.

Element plugins have two version numbers. BST_FORMAT_VERSION and
BST_ARTIFACT_VERSION. Only the latter affects the cache key.

When adding a new optional feature that doesn't impact the artifact of
existing elements, the plugin should increase BST_FORMAT_VERSION but
leave BST_ARTIFACT_VERSION as is. This shouldn't trigger any rebuilds.

If the behavior of a plugin changes, i.e., it's breaking artifact
compatibility/reproducibility with the same `get_unique_key()` output,
BST_ARTIFACT_VERSION should be incremented. We should normally try to
avoid this, of course. We may want to allow plugins to support multiple
artifact versions in the future to allow full backward compatibility
without blocking bug fixes.

Source plugins also have BST_FORMAT_VERSION but there is nothing
comparable to BST_ARTIFACT_VERSION. Source plugins that change behavior
in an incompatible way can add a version number to the output of
`get_unique_key()`, though, if necessary. We could also consider adding
BST_SOURCE_VERSION or similar, if there is a need.

> But often we add a feature to a plugin, (I have just added git-lfs 
> support to the git_tag plugin)
>
> This plugin will only change its behavior if its cache key is
> changed.

This scenario should be covered by BST_FORMAT_VERSION. Is this correct?
Or do you still think we need something else?

Cheers,
Jürg