GNOME.org
 

Re: [BuildStream] A BuildStream fetcher service

Follow up to self.

On Fri, Jun 7, 2019 at 10:37 AM Sander Striker <s striker striker nl> wrote:
Hi,

TL;DR I think we're looking at the wrong problem here.

Expanding a little bit, uploading and downloading sources should not be a factor here, when used in a context where it is ensured sources are predominantly in SourceCache.

On Thu, Jun 6, 2019 at 5:39 PM Raoul Hidalgo Charman via buildstream-list <buildstream-list gnome org> wrote:
Hi everyone,

Now that we're starting to look at remote execution more, it's been
noticeable that when doing remote builds, uploading and downloading
sources can take a significant portion of time. Ideally we want to be
able to have a remote service that can fetch these for us and ensure
they are in remote caches, bypassing the need to download a source and
then upload it to the remote server.

I think this is drawing the wrong conclusion.  We don't need a remote service.  The situation you describe needs a BuildStream instance to fill the SourceCache and it requires the lifetime of objects in the SourceCache to be long enough so that you don't need to go back to the origin.

What to track when is highly specific to the project.  I can imagine leveraging the source control system hooks to trigger targetted tracking and fetching.  I don't see us requiring infrastructure for this in BuildStream core.

I failed to quote and agree with this part of Raoul's post:

The simplest approach would just be an instance of buildstream which
periodically tracks, fetches and pushes a projects elements with no
interaction from users of the remote cache. This would just need to
implement a `bst source push` method, which should be implemented
anyway. 
 +1.
 
If we have configurable endpoints for SourceCache, and SourceCache's CAS, we should be covered.

For remote execution in the CI case I can see some other optimizations that we previously deferred [1].  We can also be smarter in terms of batching uploading of command, action, etc.  And defer checking for missing blobs from the input tree, proactively Execute, and handle PRECONDITION_FAILED.

Cheers,

Sander

 
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]