Re: Proposal for Remote Execution

[Jürg Billeter <j bitron ch>]

On Thu, 2018-05-31 at 11:06 +0200, Sander Striker wrote:
> On Wed, May 30, 2018 at 4:42 PM Jürg Billeter <j bitron ch> wrote:
> [...] 
> > > Should assemble still be putting together the artifact directory
> > > structure, or do we want to turn the artifact into a proto and store
> > > that in the ArtifactCache?  I mentioned this on a separate thread and
> > > it's easy to have it be lost: https://mail.gnome.org/archives/buildst
> > > ream-list/2018-April/msg00038.html
> > >
> > > The top level directory then doesn't need to be special cased; we can
> > > decide what to download from the ArtifactCache entry rather than the
> > > directory node in CAS.  In terms of interface it "feels" cleaner.
> >
> > The main disadvantage I see is that the CAS server would need to be
> > aware of these additional proto message types for the purpose of
> > purging (artifact expiry). Any extension of such a message would
> > require a CAS server update.
>
> Can you elaborate a bit?  I was expecting us to have CAS itself be
> content agnostic.  And for cache management to be done by a different
> process that is more content aware.  Purging would require awares of
> the contents of ActionCache and ArtifactCache, as well as CAS, right?

The CAS service API itself is content agnostic, yes. However, I
consider cache management a crucial feature and it can't live in
isolation (e.g., we need to avoid race conditions between CAS
access/uploads and purging).

And yes, purging needs to be aware of the contents of ActionCache and
ArtifactCache as there is no generic CAS 'ref' API. However, I would
still like to keep the complexity of this whole thing as low as we can
and not require knowledge of more and more message types. Otherwise the
risk of incompatibilities / the need to sync updates between CAS and
clients increases, in addition to the code size.

In light of the SourceCache discussion, we might even want to
generalize ArtifactCache (with persistency flag or similar) instead of
adding more single purpose services that all need to be supported by
the cache management.

Jürg