Re: Proposal for Remote Execution

[Jim MacArthur <jim macarthur codethink co uk>]

On 09/05/18 10:04, Jürg Billeter wrote:
> > > My plan was to have the source plugin use a temporary directory and
> > > then import the contents of that directory (in the future) into CAS
> > > during the fetch job. With this there should be no additional overhead
> > > during the build job. And later FUSE will reduce staging cost for the
> > > build job.
> > >
> > > I don't expect source plugins themselves to require any changes, this
> > > should be handled by BuildStream core code.
> > Indeed, _stage_sources_at handles this temporary directory nicely.
> > However, we won't be able to run any commands in the sandbox, and this
> > includes integration commands used to stage dependencies, so I don't
> > think we will be able to do much with them before the FUSE is
> > implemented - unless you were planning to extract to a temporary
> > directory to run each sandbox command, which seems impractical.
> Right, we need FUSE to make use of this. Without FUSE / remote
> execution, the sandbox root directory should always be a regular
> directory (or rather, a virtual directory with a native filesystem
> backend). The temporary directory for each sandbox command would be too
> expensive.
I instrumented a build of freedesktop-sdk bootstrap, and this has only 
two instances where a sandbox (and hence a virtual directory) is created 
but *not* used to run commands. These were an initial import and a final 
compose job taking 1 second and 19 seconds respectively, out of a 34 
minute total build. I'm looking for other projects to try and get some 
data from at the moment.

It seems to me that the existing filesystem-based system could just run 
through our CAS-FUSE when we implement that. I can't think of any 
particular reason that would be slow, although it'd be worth testing 
when we have it. If that does turn out to be a viable option, doesn't 
that make the CAS-backed virtual directory implementation (and in fact 
the whole virtual directory system) redundant?