Re: [BuildStream] A BuildStream fetcher service

[Sander Striker <s striker striker nl>]

Hi,

TL;DR I think for the use case this thread is trying to address, we should consider the client is not tracking at all, but only updating the project that contains pre-tracked refs.

On Mon, Jun 10, 2019 at 12:35 PM Raoul Hidalgo Charman <raoul hidalgocharman codethink co uk> wrote:

Hi Sander,

Didn't do reply to all so sorry for the double email!

No worries.

 

On 07/06/2019 09:37, Sander Striker wrote:
[...]
> For remote execution in the CI case I can see some other optimizations that we
> previously deferred [1].  We can also be smarter in terms of batching uploading
> of command, action, etc.  And defer checking for missing blobs from the input
> tree, proactively Execute, and handle PRECONDITION_FAILED.

Ah if you're envisioning that CI will keep all the sources up to date
without users requesting it, then perhaps a service isn't needed but I
think tracking still requires some thinking.

Correct.

 

For this, the clients will need some way of updating their sources refs,
else they will still need to download the source, track and then can see
that the updated source is in the remote source cache, which definitely
isn't particularly efficient. The sources are referenced by their hash
and without the new ref users won't know this, and so I think some
service would still be needed to allow users to find out these updated
refs. Of course there might be a simpler way I'm just missing, but with
the way all plugins have to track (bar local and patch as they don't
have refs), I don't think this is possible without a service.

We should consider the client is not tracking at all, but only updating the project that contains pre-tracked refs.

I am not advocating to do this now, but I could see the tracking process be smarter.  It probably requires some tweaking in the source plugins, and potentially some other tweaks.  Taking the git source plugin as a example, I could imagine that a track there only finds out what the ref is on the remote, rather than actually fetching.

From there the operations are as before, check for an entry in SourceCache when needing to build, and fall back to fetching otherwise.  This would hold true for the bst fetch as well as for bst build.  With bst workspace open we always need to fetch still.

The benefit here is that you would only hit the source remotes for refs, and can leverage SourceCache whereever a centrally managed BuildStream instance has already done the work for you.

[...]

It looks like partial local CAS might solve some of the other issues
with regards to having to stage dependency artifacts and sources for RE.

That, and maybe other optimizations.

Cheers,

Sander