Re: [BuildStream] Partial local CAS

[Jürg Billeter <j bitron ch>]

Hi Jim,

On Mon, 2018-11-26 at 17:59 +0000, Jim MacArthur via BuildStream-list wrote:
> In the simplest case, we could just check whether a build requirement 
> exists on the remote execution storage service first, and if it does, 
> and we are using remote execution, we don't need to download that 
> artifact or source to the `bst` client machine. It's quite probable that 
> someone else sharing the remote execution service will have uploaded 
> that artifact already, or a previous run by ourselves will have done.

We generally still need to download the tree (Directory objects) even
for build-only dependencies as that's required for (virtual) staging.
We could potentially download Directory objects on demand instead of
downloading the whole tree, but I think the most important optimization
is to download blobs only if/when required. I consider always fetching
all Directory objects (via GetTree) to be acceptable, at least in a
first step.

The client needs to verify that the blobs are actually on the CAS
server. Either using FindMissingBlobs() or possibly have the
ReferenceStorage service's GetReference() method return success only if
all referenced objects are available. The latter would require a
ReferenceStorage service being available and might be problematic with
potentially optional subdirectories such as 'buildtree'. The former
requires a bit more work on the client side but should be less
problematic, so I would follow that path.

> The second case is that the remote execution service could be made to 
> find build-requirements itself; if the remote service has access to our 
> artifact cache, or source cache when that appears, then it can query 
> those for missing artifacts or blobs before it answers its own 
> 'findMissingBlobs' request.

I don't think we should go that far at this point. I'd stick to the
first optimization, handling this client side, and not requiring
protocol extensions.

Cheers,
Jürg