Re: Dual cache key calculation and build planning modes

[Jürg Billeter <j bitron ch>]

Hi Tristan,

thanks for the write-up. It looks like a reasonable plan for handling
dual cache keys / build planning modes. That said, I'm not very fond of
having such a non-predictable build planning mode at all. If there is
no other way to fulfill the requirements, so be it, but maybe the
following idea would be an acceptable alternative.

The underlying reason why weak build planning modes will (mostly) work
is that shared library interfaces are reasonably stable, at least in
the GNOME ecosystem. Instead of using weak cache keys, hoping for the
best, and manually intervening when something breaks, we could supply a
bit more information to BuildStream such that it can avoid rebuilding
reverse dependencies for minor library changes without losing the
predictability we have right now. It could even become the default
mode.

My proposal is to add a secondary ref to sources of library elements.
That secondary ref points to the oldest revision that has the same API
and ABI as the primary ref (and doesn't contain any major bugs that
would break build tools). Whenever such a library is staged as a
dependency of a build process (directly or indirectly), the artifact
built from the secondary ref is used. The artifact built from the
primary ref is used only when composing a system.

This will require building all libraries with differing refs twice, but
only when building from scratch. Artifact sharing should be very
effective with this approach.

I haven't worked out all the details yet but I would appreciate input
on whether it makes sense to continue in this direction or if this is
not suitable to fulfill the requirements. One question is how to handle
bumps of secondary refs for fully automated build systems. Is a manual
action on interface breaks and additions acceptable?

Best regards,
Jürg