Re: Dual cache key calculation and build planning modes

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Thu, 2017-07-06 at 11:53 +0100, Paul Sherwood wrote:
> On 2017-07-06 11:03, Tristan Van Berkom wrote:
> > On Thu, 2017-07-06 at 07:15 +0100, Paul Sherwood wrote:
> > > Hi Tristan,
> > > On 2017-07-05 10:05, Tristan Van Berkom wrote:
> > > > Premise
> > > > ~~~~~~~
> > > > The premise is simply that every element has 2 cache keys
> > > > generated at
> > > > all times.
> > >
> > > I haven't had chance to follow all of the detail here, but isn't
> > > your 
> > > 'weak' key just a hash of a subset of the same factors in for
> > > your 
> > > 'strong' key?
> > >
> > > In which case, would it be simpler and better to have one key
> > > which 
> > > combines both?
> > >
> > > foo.<hash(weak-factors)>.<hash(strong-factors - weak-factors)>
> > >
> > > Then tooling can decide whether it cares about the whole key or
> > > just 
> > > the 
> > > 'weak' part?
> >
> > Interesting idea, but I'm not sure what it would add.
>
> If you have two keys, you always end up with two artifacts? 
> Foo.weak.strong is one artifact that can be consumed for all use-
> cases.

Not really no.

You have one artifact, in it's metadata it stores the two keys
(metadata is separate from files and such).

So with different artifact cache technologies we could approach the
problem of storage differently, but with ostree (which treats
filesystem data in a similar way to git) we would:

  A.) Have a single ref for the artifact's strong key (there can
      only be one ref for an artifact's strong key)

  B.) Have a branch for the artifact's weak key (one can have
      multiple artifacts revisioned under the same weak key,
      the newest one should be the branch tip).

With say, a tarball based artifact cache, you could use an approach
where you always store the tarball as the strong key, and you create a
symlink with the name of the weak key, pointing always to the latest
strong artifact key. 

In either case, being able to lookup the artifact with two different
methods does not necessitate duplicating data.

Cheers,
    -Tristan