GNOME.org
 

Re: [BuildStream] Partial local CAS


On 08/11/2018 22:01, Sander Striker via BuildStream-list wrote:

Hi,
After the exchange in the "Coping with partial artifacts" thread, I realize that we haven't actually had a conversation on list about partial local CAS, and by extension local ArtifactCache.  Let me first explain what I mean with partial local CAS.  Let's define it as a CAS that contains Tree and Directory nodes, but not the [all of] actual file content blobs.
I'll outline the context and importance of this concept. In remote execution builds do not run on the local machine. As such to be able to perform a build, it is important to be able to _describe_ the inputs to a build.  When all of the input files are locally available, this can be done.  However, when the input files are not locally available, should we then incur the cost of fetching them?  Is there another way?
To answer that question let's review how remote execution is supposed to work again in the context of BuildStream.  To build an element: 

1) Compose a merkle tree of all dependencies, and all sources
2) Create a Command and an Action message
3) FindMissingBlobs(command, action, blobs in the merkle tree)
4) Upload the missing blobs
5) Submit the request to the execution service
6) Wait for the request to complete
7) Download the result merkle tree
8) Construct a merkle tree for the Artifact (based on the result)
9) FindMissingBlobs(blobs in the artifact merkle tree)
10) Upload the missing blobs
11) Store a ref to the artifact merkle tree in ArtifactCache
Let's dive in a bit and look where the inefficiencies are in the current implementation.
Step 1 happens during staging.  More specifically in buildelement.py:stage().  We start with the dependencies.  For directories backed by CAS, we don't need to actually stage them on the filesystem.  We can import files between CAS directories by reference (hash), without even needing the files locally.  This isn't currently implemented (_casbaseddirectory.py:import_files), but that should change with CAS-to-CAS import (MR !911). After the depencies are staged, we move on to the sources. Currently this is still fairly clunky, as we are actually staging sources on the filesystem and then importing that into our virtual staging directory (element.py:_stage_sources_at). With SourceCache this should be as efficient as staging dependencies for non-modified elements. 

Step 2 through 11 all happen during _sandboxremote.py:run().
Step 2-4 aren't currently implemented in this fashion, and instead serially call a number of network RPCs.  In _sandboxremote.py:run() a call is made to cascache.push_directory().  This will push up any missing directory nodes, or any missing files. In _sandboxremote:run_remote_command() we are using cascache.push_message(), followed by cascache.verify_digest_pushed().  This results in a Write RPC, followed by a FindMissingBlobs RPC.  For both the Command and the Action.  In short, we could be eliminating a couple of RPCs and thus network roundrips here. I'll skip over step 5-6 as these are not very interesting. Although it should be noted that _sandboxremote.py:run() is ignoring the build logs from the execution response. In step 7, which happens in _sandboxremote.py:process_job_output(), we take a Tree digest that we received from the execution service, and use it in a call to cascache.pull_tree().  This will fetch all of the file blobs that are present in the tree that are not available locally.  It will also store all of the directory nodes that are referenced in the tree, and return the root digest.  This is used to construct the result virtual directory of the sandbox. In step 8 we go back to constructing a file system representation of the artifact, instead of using a CAS backed directory.  This happens in element.py:assemble() through a call to cascache.commit().  This will do a local filesystem import of files, the majority of which we exported in step 7. It will put an entry in the local ArtifactCache. Step 9-11 happen during the push phase.  Here we rely on cascache.push() to ensure that the artifact is made available on the remote CAS server.
Sidenote while we're here: apart from step 9-11 we don't actually make it clear to the scheduler which resources are needed.  As far as it is concerned a remote build job is currently taking up PROCESS tokens.
If you made it all the way here, thank you :).  I think we need to eliminate the unneeded filesystem access first. 
Then we can go further and support partial CAS by:
- erroring when FindMissingBlobs() calls return digests that we don't actually have locally - retrieving just the Tree, rather than all blobs when we process ActionResults
Only when you actually want to use the artifact locally should we fetch the actual file objects.  For instance in case of bst [artifact] checkout.  Or bst shell.  If we are not using the files, there is no real point in downloading all this content, which takes both time and disk space.
Regardless of the current discussion on default behaviour and whether we want finished artifacts to always be present on the `bst` client machine, there clearly are some cases (build requirements) in which we don't always need to download artifacts to the local machine if using remote execution. This is my understanding of the potential optimisations:
In the simplest case, we could just check whether a build requirement exists on the remote execution storage service first, and if it does, and we are using remote execution, we don't need to download that artifact or source to the `bst` client machine. It's quite probable that someone else sharing the remote execution service will have uploaded that artifact already, or a previous run by ourselves will have done.
The second case is that the remote execution service could be made to find build-requirements itself; if the remote service has access to our artifact cache, or source cache when that appears, then it can query those for missing artifacts or blobs before it answers its own 'findMissingBlobs' request.
We can't require that the remote execution service does this, since we have to use only the REAPI, nor can we explicitly tell the service where to find artifacts, but if we are running our own remote execution service, we can give it a list of remote storage to query first.
We also don't have any notion of addressing or location for artifacts AFAIK so the remote service would have to query all of its remotes. 

Does that sound like a reasonable goal?

Jim
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]