Re: [BuildStream] cache key changes

[Sander Striker <s striker striker nl>]

Just to throw another [implementation detail] suggestion in, which Chandan mentioned during a hallway conversation last week: how about we define the cache key's as a proto?  That way we have the serialization be very clear, and the components of it can be documented with the spec.

Thoughts?

Cheers,

Sander

On Wed, Jul 10, 2019 at 1:22 PM Jürg Billeter <j bitron ch> wrote:

Hi Gökçen,

On Wed, 2019-07-10 at 12:47 +0100, Gökçen Nurlu via buildstream-list
wrote:
> [...]
> - dependencies = order determined by `Element.dependencies()`
>
> [...]
>
> I think this traversing order is an unnecessary, extra, perhaps
> harmful detail while calculating the cache-key. Ideally, I think it
> should be a 'set' but I don't know a way of encoding them other than
> using a sorted list. This can have to gains:
> - In case we change the traversing order (perf reasons, scheduler
> related things, etc.) cache-keys won't get affected.

We actually want or rather need the cache key to change if we ever
decide to change staging order as staging order can influence the build
(e.g., due to overlaps).

We could also trigger a cache key change by incrementing the core
artifact version if we ever changed staging order, of course. I.e.,
storing dependencies in staging order in the cache key dict is not a
hard requirement. At least with the current approach. If we were to
change staging order to be based on dependency declaration order in the
.bst files, we'd really need an ordered dependency list in the cache
key dict.

> - It MIGHT! pave the way for a more lightweight traversal method,
> indeterministic or not, only to be used for cache-key-calc purposes,
> potentially speeding up anything

This could be interesting. However, until such an optimization
materializes I'd prefer keeping the staging order in the cache key dict
for the following reasons:

 * As described above, staging order must affect the cache key and
   thus, using the staging order directly in the cache key dict is the
   safer option (no risk of changing staging order and forgetting to
   increment the core artifact version).
 * The optimization might actually require a different structure of the
   dependency list, which would anyway require a core artifact version
   increment. E.g., it could make sense to generate an additional hash
   that uses the cache keys of all runtime dependencies of an element.
   We could then use these 'runtime hashes' of direct build
   dependencies in the cache key dict `dependencies`, which would allow
   us to skip recursion.

Cheers,
Jürg

_______________________________________________
buildstream-list mailing list
buildstream-list gnome org
https://mail.gnome.org/mailman/listinfo/buildstream-list