Re: [BuildStream] Proposal: bst artifact subcommand group

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi,

On Thu, 2018-10-04 at 03:08 +0200, Sander Striker wrote:
[...]
> > This sounds like it makes sense yes, but let's be careful not to cross
> > subject material and confuse folks.
>
> Fair point.  I'm actually trying to think through the impact of
> $project/$element in the $cachekey :).  It lead us here.  And helped
> me come to the conclusion that $project/$element in the $cachekey are
> not prohibiting build avoidance, assuming we do aliasing as you've
> called it below.

Trying to call it something, aliasing might not be the perfect word but
seems pretty close :)

[...]
> > You are talking about aliasing keys for things which we found already
> > existed, which is not a bad idea, but opens up a separate can of worms,
> > which is how we trace the provenance of an aliased artifact, and how we
> > bake up metadata for it.
> >
> > Add to that can of worms: We also need to take care of aliasing the
> > reverse dependencies instead of rebuilding everything, when a low level
> > element's cache key differs but is aliasable.
> >
> >  * I very much value the ability to rename your elements without it
> >    causing any changes in cache key.
> >
> >    If it could introduce a new cache key that is aliasable, that is
> >    almost the same thing, so I like it.
> >
> >  * Aliasing artifacts in the case that a Source configuration has
> >    changed but resulted in staging identical inputs, is valuable too.
> >
> >  * I think that the chances of having two projects which happen to
> >    build an element with the same source code at the same ref,
> >    completely identical inputs and an identical dependency chain - are
> >    astronomically small. Even more so if those builds are occurring on
> >    the same infra.
>
> I can think of some things people could do:
> - Copy projectA.  Rename copy to projectB.  Build.
> - Copy projectA.  Rename copy to projecC.  Prune elements.  Add new element.
> - Create project D. Copy element W from projectA (and its aliases).
>
> The last one could be a "webkit" like element.  We can't assume that
> everyone is going to junction instead of copy the .bst.  The latter
> is easy to understand and quickly do.
>  
> >    If this were the sole purpose, then artifact aliasing would be far
> >    to much trouble for the near zero benefits.
>
> I think we need to be a bit careful with such statements, they sound
> as if based on data, where in reality they're based on assumptions. 
> As our assumptions are not always aligned, I think that's where we
> then start.
> That said, in this particular instance we can ignore our difference
> based on the agreement on the first two use cases.

Ok, fair point that we should base things on data if available.

That said, we should always be careful to avoid drastic measures until
we really know (whether it is seemingly obvious and thus assumed, or
backed by data) that users are going to benefit.

What we're discussing here is a complicated change, expensive to
execute and expensive to maintain cleanly - we shouldn't do such
drastic changes unless we've at least discussed and explored what
benefits these changes will bring.

> > Overall I do like your idea of aliasing cache keys for build avoidance
> > where possible.
>
> Cool.  I think we can park the aliasing part of the discussion and
> resume the artifact command subgroup part of the thread.  I felt we
> were getting close to consensus on that as well?

I think so, however it seems to me there has been quite some churn in
this thread and other related threads.

Probably best I go over it all and summarize here once more.

Cheers,
    -Tristan