Re: [BuildStream] A BuildStream fetcher service

[Raoul Hidalgo Charman <raoul hidalgocharman codethink co uk>]

Hi Sander,

Didn't do reply to all so sorry for the double email!

On 07/06/2019 09:37, Sander Striker wrote:
> [snip]
>
> I think this is drawing the wrong conclusion.  We don't need a remote service.  
> The situation you describe needs a BuildStream instance to fill the SourceCache 
> and it requires the lifetime of objects in the SourceCache to be long enough so 
> that you don't need to go back to the origin.
>
> What to track when is highly specific to the project.  I can imagine leveraging 
> the source control system hooks to trigger targetted tracking and fetching.  I 
> don't see us requiring infrastructure for this in BuildStream core.
> If we have configurable endpoints for SourceCache, and SourceCache's CAS, we 
> should be covered.
>
> For remote execution in the CI case I can see some other optimizations that we 
> previously deferred [1].  We can also be smarter in terms of batching uploading 
> of command, action, etc.  And defer checking for missing blobs from the input 
> tree, proactively Execute, and handle PRECONDITION_FAILED.

Ah if you're envisioning that CI will keep all the sources up to date
without users requesting it, then perhaps a service isn't needed but I
think tracking still requires some thinking.

For this, the clients will need some way of updating their sources refs,
else they will still need to download the source, track and then can see
that the updated source is in the remote source cache, which definitely
isn't particularly efficient. The sources are referenced by their hash
and without the new ref users won't know this, and so I think some
service would still be needed to allow users to find out these updated
refs. Of course there might be a simpler way I'm just missing, but with
the way all plugins have to track (bar local and patch as they don't
have refs), I don't think this is possible without a service.

I think Tristan misunderstood this above issue, as being a problem with
downloading the plugins themselves, rather than the tracking of sources
of a particular plugin type.

It looks like partial local CAS might solve some of the other issues
with regards to having to stage dependency artifacts and sources for RE.

> Cheers,
>
> Sander

Cheers,
Raoul