[Notes] [Git][BuildStream/buildstream][chandan/bst-checkout-build] 12 co

Chandan Singh pushed to branch chandan/bst-checkout-build at BuildStream / buildstream

Commits:

b199afe6

by Jürg Billeter at 2018-09-25T09:01:51Z

_artifactcache/cascache.py: Add _ensure_blob helper

This adds directory objects to the local repository before downloading
files in the directory. However, artifact references are still stored
only after downloading the complete directory and thus, there won't be
dangling references. This will anyway be required for partial download
support.

7d199322

by Jürg Billeter at 2018-09-25T09:01:51Z

_artifactcache/cascache.py: Increase payload size limit for uploads

gRPC can handle 1 MiB payloads. Increase size limit from 64 KiB to speed
up uploads.`

e2e24015

by Jürg Billeter at 2018-09-25T09:01:51Z

_artifactcache/casserver.py: Harmonize payload size limit

Use 1 MiB as payload size limit on the server side for both individual
downloads and batch uploads.

697d10f2

by Jürg Billeter at 2018-09-25T09:01:51Z

_artifactcache/cascache.py: Use BatchReadBlobs

This uses BatchReadBlobs instead of individual blob download to speed up
artifact pulling, if the server supports it.

Fixes #554.

81c51dbf

by Jürg Billeter at 2018-09-25T09:31:55Z

Merge branch 'juerg/cas-batch' into 'master'

_artifactcache/cascache.py: Use BatchReadBlobs

Closes #554

See merge request BuildStream/buildstream!813

2df7d140

by Jim MacArthur at 2018-09-25T10:36:37Z

sandbox.py: Stop caching get_virtual_directory if get_directory is used

Also update the documentation for get_virtual_directory to make the
implications of using both methods clearer.

62f59382

by Jim MacArthur at 2018-09-25T10:36:37Z

element.py/_assemble(): Call get_virtual_directory later.

8cea7b17

by Jim MacArthur at 2018-09-25T10:58:40Z

Merge branch 'jmac/stop-caching-vdirs' into 'master'

Stop caching virtual directories if get_directory is used.

See merge request BuildStream/buildstream!818

83ab183e

by Tiago Gomes at 2018-09-25T16:37:30Z

CI: test building freedesktop-sdk overnight

Closes #514.

c8594806

by Tiago Gomes at 2018-09-25T17:03:55Z

Merge branch 'tiagogomes/issue-514' into 'master'

CI: test building freedesktop-sdk overnight

Closes #514

See merge request BuildStream/buildstream!826

b8ad3707

by Chandan Singh at 2018-09-25T23:47:34Z

Ensure `--deps=none` option works for `bst checkout`

Currently, `bst checkout --deps none` command always produces empty
output. Fix this issue and add regression test for the same.

As part of this, also rename the `dep` loop variable to `element` in
order to avoid confusion in future.

Fixes #670.

7d88fff1

by Chandan Singh at 2018-09-25T23:47:44Z

Add `--deps build` option to `bst checkout`

As discussed in
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
add `--deps build` option to `bst checkout`. This will allow users to
checkout the all build dependencies of a given element using a single
command.

10 changed files:

.gitlab-ci.yml
buildstream/_artifactcache/cascache.py
buildstream/_artifactcache/casserver.py
buildstream/_frontend/cli.py
buildstream/element.py
buildstream/element_enums.py
buildstream/sandbox/sandbox.py
tests/frontend/buildcheckout.py
+ tests/frontend/project/elements/checkout-deps.bst
+ tests/frontend/project/files/etc-files/etc/buildstream/config

Changes:

.gitlab-ci.yml

@@ -79,6 +79,8 @@ source_dist:
    - cd ../..
    - mkdir -p coverage-linux/
    - cp dist/buildstream/.coverage.* coverage-linux/coverage."${CI_JOB_NAME}"
 +  except:
 +  - schedules
    artifacts:
      paths:
      - coverage-linux/
@@ -127,6 +129,8 @@ tests-unix:
      - cd ../..
      - mkdir -p coverage-unix/
      - cp dist/buildstream/.coverage.* coverage-unix/coverage.unix
 +  except:
 +  - schedules
    artifacts:
      paths:
      - coverage-unix/
@@ -148,10 +152,41 @@ docs:
    - make BST_FORCE_SESSION_REBUILD=1 -C doc
    - cd ../..
    - mv dist/buildstream/doc/build/html public
 +  except:
 +  - schedules
    artifacts:
      paths:
      - public/
 +.overnight-tests: &overnight-tests-template
 +  stage: test
 +  variables:
 +    bst_ext_url: git+https://gitlab.com/BuildStream/bst-external.git
 +    bst_ext_ref: 1d6ab71151b93c8cbc0a91a36ffe9270f3b835f1 # 0.5.1
 +    fd_sdk_ref: 718ea88089644a1ea5b488de0b90c2c565cb75f8 # 18.08.12
 +  before_script:
 +  - (cd dist && ./unpack.sh && cd buildstream && pip3 install .)
 +  - pip3 install --user -e ${bst_ext_url}@${bst_ext_ref}#egg=bst_ext
 +  - git clone https://gitlab.com/freedesktop-sdk/freedesktop-sdk.git
 +  - git -C freedesktop-sdk checkout ${fd_sdk_ref}
 +  only:
 +  - schedules
++
 +overnight-tests:
 +  <<: *overnight-tests-template
 +  script:
 +  - make -C freedesktop-sdk
 +  tags:
 +  - overnight-tests
++
 +overnight-tests-no-cache:
 +  <<: *overnight-tests-template
 +  script:
 +  - sed -i '/artifacts:/,+1 d' freedesktop-sdk/bootstrap/project.conf
 +  - sed -i '/artifacts:/,+1 d' freedesktop-sdk/project.conf
 +  - make -C freedesktop-sdk
 +  tags:
 +  - overnight-tests
  # Check code quality with gitlab's built-in feature.
+ #
@@ -170,6 +205,8 @@ code_quality:
          --volume "$PWD":/code
          --volume /var/run/docker.sock:/var/run/docker.sock
          "registry.gitlab.com/gitlab-org/security-products/codequality:$SP_VERSION" /code
 +  except:
 +  - schedules
    artifacts:
      paths: [gl-code-quality-report.json]
@@ -199,6 +236,8 @@ analysis:
      radon raw -s -j buildstream > analysis/raw.json
      radon raw -s buildstream
 +  except:
 +  - schedules
    artifacts:
      paths:
      - analysis/
@@ -224,6 +263,8 @@ coverage:
    - tests-fedora-28
    - tests-unix
    - source_dist
 +  except:
 +  - schedules
  # Deploy, only for merges which land on master branch.
+ #
@@ -248,3 +289,5 @@ pages:
    # See https://gitlab.com/gitlab-org/gitlab-ce/issues/35141
+   #
    - master
 +  except:
 +  - schedules

buildstream/_artifactcache/cascache.py

@@ -44,6 +44,11 @@ from .._exceptions import ArtifactError
  from . import ArtifactCache
 +# The default limit for gRPC messages is 4 MiB.
 +# Limit payload to 1 MiB to leave sufficient headroom for metadata.
 +_MAX_PAYLOAD_BYTES = 1024 * 1024
++
++
  # A CASCache manages artifacts in a CAS repository as specified in the
  # Remote Execution API.
+ #
@@ -854,6 +859,80 @@ class CASCache(ArtifactCache):
          assert digest.size_bytes == os.fstat(stream.fileno()).st_size
 +    # _ensure_blob():
 +    #
 +    # Fetch and add blob if it's not already local.
 +    #
 +    # Args:
 +    #     remote (Remote): The remote to use.
 +    #     digest (Digest): Digest object for the blob to fetch.
 +    #
 +    # Returns:
 +    #     (str): The path of the object
 +    #
 +    def _ensure_blob(self, remote, digest):
 +        objpath = self.objpath(digest)
 +        if os.path.exists(objpath):
 +            # already in local repository
 +            return objpath
++
 +        with tempfile.NamedTemporaryFile(dir=self.tmpdir) as f:
 +            self._fetch_blob(remote, digest, f)
++
 +            added_digest = self.add_object(path=f.name)
 +            assert added_digest.hash == digest.hash
++
 +        return objpath
++
 +    def _batch_download_complete(self, batch):
 +        for digest, data in batch.send():
 +            with tempfile.NamedTemporaryFile(dir=self.tmpdir) as f:
 +                f.write(data)
 +                f.flush()
++
 +                added_digest = self.add_object(path=f.name)
 +                assert added_digest.hash == digest.hash
++
 +    # Helper function for _fetch_directory().
 +    def _fetch_directory_batch(self, remote, batch, fetch_queue, fetch_next_queue):
 +        self._batch_download_complete(batch)
++
 +        # All previously scheduled directories are now locally available,
 +        # move them to the processing queue.
 +        fetch_queue.extend(fetch_next_queue)
 +        fetch_next_queue.clear()
 +        return _CASBatchRead(remote)
++
 +    # Helper function for _fetch_directory().
 +    def _fetch_directory_node(self, remote, digest, batch, fetch_queue, fetch_next_queue, *, recursive=False):
 +        in_local_cache = os.path.exists(self.objpath(digest))
++
 +        if in_local_cache:
 +            # Skip download, already in local cache.
 +            pass
 +        elif (digest.size_bytes >= remote.max_batch_total_size_bytes or
 +                not remote.batch_read_supported):
 +            # Too large for batch request, download in independent request.
 +            self._ensure_blob(remote, digest)
 +            in_local_cache = True
 +        else:
 +            if not batch.add(digest):
 +                # Not enough space left in batch request.
 +                # Complete pending batch first.
 +                batch = self._fetch_directory_batch(remote, batch, fetch_queue, fetch_next_queue)
 +                batch.add(digest)
++
 +        if recursive:
 +            if in_local_cache:
 +                # Add directory to processing queue.
 +                fetch_queue.append(digest)
 +            else:
 +                # Directory will be available after completing pending batch.
 +                # Add directory to deferred processing queue.
 +                fetch_next_queue.append(digest)
++
 +        return batch
++
      # _fetch_directory():
+     #
      # Fetches remote directory and adds it to content addressable store.
@@ -867,39 +946,32 @@ class CASCache(ArtifactCache):
      #     dir_digest (Digest): Digest object for the directory to fetch.
+     #
      def _fetch_directory(self, remote, dir_digest):
 -        objpath = self.objpath(dir_digest)
 -        if os.path.exists(objpath):
 -            # already in local cache
 -            return
+-
 -        with tempfile.NamedTemporaryFile(dir=self.tmpdir) as out:
 -            self._fetch_blob(remote, dir_digest, out)
+-
 -            directory = remote_execution_pb2.Directory()
 +        fetch_queue = [dir_digest]
 +        fetch_next_queue = []
 +        batch = _CASBatchRead(remote)
 -            with open(out.name, 'rb') as f:
 -                directory.ParseFromString(f.read())
 +        while len(fetch_queue) + len(fetch_next_queue) > 0:
 +            if len(fetch_queue) == 0:
 +                batch = self._fetch_directory_batch(remote, batch, fetch_queue, fetch_next_queue)
 -            for filenode in directory.files:
 -                fileobjpath = self.objpath(filenode.digest)
 -                if os.path.exists(fileobjpath):
 -                    # already in local cache
 -                    continue
 +            dir_digest = fetch_queue.pop(0)
 -                with tempfile.NamedTemporaryFile(dir=self.tmpdir) as f:
 -                    self._fetch_blob(remote, filenode.digest, f)
 +            objpath = self._ensure_blob(remote, dir_digest)
 -                    digest = self.add_object(path=f.name)
 -                    assert digest.hash == filenode.digest.hash
 +            directory = remote_execution_pb2.Directory()
 +            with open(objpath, 'rb') as f:
 +                directory.ParseFromString(f.read())
              for dirnode in directory.directories:
 -                self._fetch_directory(remote, dirnode.digest)
 +                batch = self._fetch_directory_node(remote, dirnode.digest, batch,
 +                                                   fetch_queue, fetch_next_queue, recursive=True)
++
 +            for filenode in directory.files:
 +                batch = self._fetch_directory_node(remote, filenode.digest, batch,
 +                                                   fetch_queue, fetch_next_queue)
 -            # Place directory blob only in final location when we've
 -            # downloaded all referenced blobs to avoid dangling
 -            # references in the repository.
 -            digest = self.add_object(path=out.name)
 -            assert digest.hash == dir_digest.hash
 +        # Fetch final batch
 +        self._fetch_directory_batch(remote, batch, fetch_queue, fetch_next_queue)
      def _fetch_tree(self, remote, digest):
          # download but do not store the Tree object
@@ -914,16 +986,7 @@ class CASCache(ArtifactCache):
              tree.children.extend([tree.root])
              for directory in tree.children:
                  for filenode in directory.files:
 -                    fileobjpath = self.objpath(filenode.digest)
 -                    if os.path.exists(fileobjpath):
 -                        # already in local cache
 -                        continue
+-
 -                    with tempfile.NamedTemporaryFile(dir=self.tmpdir) as f:
 -                        self._fetch_blob(remote, filenode.digest, f)
+-
 -                        added_digest = self.add_object(path=f.name)
 -                        assert added_digest.hash == filenode.digest.hash
 +                    self._ensure_blob(remote, filenode.digest)
                  # place directory blob only in final location when we've downloaded
                  # all referenced blobs to avoid dangling references in the repository
@@ -942,12 +1005,12 @@ class CASCache(ArtifactCache):
              finished = False
              remaining = digest.size_bytes
              while not finished:
 -                chunk_size = min(remaining, 64 * 1024)
 +                chunk_size = min(remaining, _MAX_PAYLOAD_BYTES)
                  remaining -= chunk_size
                  request = bytestream_pb2.WriteRequest()
                  request.write_offset = offset
 -                # max. 64 kB chunks
 +                # max. _MAX_PAYLOAD_BYTES chunks
                  request.data = instream.read(chunk_size)
                  request.resource_name = resname
                  request.finish_write = remaining <= 0
@@ -1035,11 +1098,78 @@ class _CASRemote():
              self.bytestream = bytestream_pb2_grpc.ByteStreamStub(self.channel)
              self.cas = remote_execution_pb2_grpc.ContentAddressableStorageStub(self.channel)
 +            self.capabilities = remote_execution_pb2_grpc.CapabilitiesStub(self.channel)
              self.ref_storage = buildstream_pb2_grpc.ReferenceStorageStub(self.channel)
 +            self.max_batch_total_size_bytes = _MAX_PAYLOAD_BYTES
 +            try:
 +                request = remote_execution_pb2.GetCapabilitiesRequest()
 +                response = self.capabilities.GetCapabilities(request)
 +                server_max_batch_total_size_bytes = response.cache_capabilities.max_batch_total_size_bytes
 +                if 0 < server_max_batch_total_size_bytes < self.max_batch_total_size_bytes:
 +                    self.max_batch_total_size_bytes = server_max_batch_total_size_bytes
 +            except grpc.RpcError as e:
 +                # Simply use the defaults for servers that don't implement GetCapabilities()
 +                if e.code() != grpc.StatusCode.UNIMPLEMENTED:
 +                    raise
++
 +            # Check whether the server supports BatchReadBlobs()
 +            self.batch_read_supported = False
 +            try:
 +                request = remote_execution_pb2.BatchReadBlobsRequest()
 +                response = self.cas.BatchReadBlobs(request)
 +                self.batch_read_supported = True
 +            except grpc.RpcError as e:
 +                if e.code() != grpc.StatusCode.UNIMPLEMENTED:
 +                    raise
++
              self._initialized = True
 +# Represents a batch of blobs queued for fetching.
 +#
 +class _CASBatchRead():
 +    def __init__(self, remote):
 +        self._remote = remote
 +        self._max_total_size_bytes = remote.max_batch_total_size_bytes
 +        self._request = remote_execution_pb2.BatchReadBlobsRequest()
 +        self._size = 0
 +        self._sent = False
++
 +    def add(self, digest):
 +        assert not self._sent
++
 +        new_batch_size = self._size + digest.size_bytes
 +        if new_batch_size > self._max_total_size_bytes:
 +            # Not enough space left in current batch
 +            return False
++
 +        request_digest = self._request.digests.add()
 +        request_digest.hash = digest.hash
 +        request_digest.size_bytes = digest.size_bytes
 +        self._size = new_batch_size
 +        return True
++
 +    def send(self):
 +        assert not self._sent
 +        self._sent = True
++
 +        if len(self._request.digests) == 0:
 +            return
++
 +        batch_response = self._remote.cas.BatchReadBlobs(self._request)
++
 +        for response in batch_response.responses:
 +            if response.status.code != grpc.StatusCode.OK.value[0]:
 +                raise ArtifactError("Failed to download blob {}: {}".format(
 +                    response.digest.hash, response.status.code))
 +            if response.digest.size_bytes != len(response.data):
 +                raise ArtifactError("Failed to download blob {}: expected {} bytes, received {} bytes".format(
 +                    response.digest.hash, response.digest.size_bytes, len(response.data)))
++
 +            yield (response.digest, response.data)
++
++
  def _grouper(iterable, n):
      while True:
          try:

buildstream/_artifactcache/casserver.py

@@ -38,8 +38,9 @@ from .._context import Context
  from .cascache import CASCache
 -# The default limit for gRPC messages is 4 MiB
 -_MAX_BATCH_TOTAL_SIZE_BYTES = 4 * 1024 * 1024
 +# The default limit for gRPC messages is 4 MiB.
 +# Limit payload to 1 MiB to leave sufficient headroom for metadata.
 +_MAX_PAYLOAD_BYTES = 1024 * 1024
  # Trying to push an artifact that is too large
@@ -158,7 +159,7 @@ class _ByteStreamServicer(bytestream_pb2_grpc.ByteStreamServicer):
                  remaining = client_digest.size_bytes - request.read_offset
                  while remaining > 0:
 -                    chunk_size = min(remaining, 64 * 1024)
 +                    chunk_size = min(remaining, _MAX_PAYLOAD_BYTES)
                      remaining -= chunk_size
                      response = bytestream_pb2.ReadResponse()
@@ -242,7 +243,7 @@ class _ContentAddressableStorageServicer(remote_execution_pb2_grpc.ContentAddres
          for digest in request.digests:
              batch_size += digest.size_bytes
 -            if batch_size > _MAX_BATCH_TOTAL_SIZE_BYTES:
 +            if batch_size > _MAX_PAYLOAD_BYTES:
                  context.set_code(grpc.StatusCode.INVALID_ARGUMENT)
                  return response
@@ -269,7 +270,7 @@ class _CapabilitiesServicer(remote_execution_pb2_grpc.CapabilitiesServicer):
          cache_capabilities = response.cache_capabilities
          cache_capabilities.digest_function.append(remote_execution_pb2.SHA256)
          cache_capabilities.action_cache_update_capabilities.update_enabled = False
 -        cache_capabilities.max_batch_total_size_bytes = _MAX_BATCH_TOTAL_SIZE_BYTES
 +        cache_capabilities.max_batch_total_size_bytes = _MAX_PAYLOAD_BYTES
          cache_capabilities.symlink_absolute_path_strategy = remote_execution_pb2.CacheCapabilities.ALLOWED
          response.deprecated_api_version.major = 2

buildstream/_frontend/cli.py

@@ -630,7 +630,7 @@ def shell(app, element, sysroot, mount, isolate, build_, command):
  @click.option('--force', '-f', default=False, is_flag=True,
                help="Allow files to be overwritten")
  @click.option('--deps', '-d', default='run',
 -              type=click.Choice(['run', 'none']),
 +              type=click.Choice(['run', 'build', 'none']),
                help='The dependencies to checkout (default: run)')
  @click.option('--integrate/--no-integrate', default=True, is_flag=True,
                help="Whether to run integration commands")

buildstream/element.py

@@ -423,7 +423,7 @@ class Element(Plugin):
                                                  visited=visited, recursed=True)
          # Yeild self only at the end, after anything needed has been traversed
 -        if should_yield and (recurse or recursed) and (scope == Scope.ALL or scope == Scope.RUN):
 +        if should_yield and (recurse or recursed) and scope != Scope.BUILD:
              yield self
      def search(self, scope, name):
@@ -671,37 +671,37 @@ class Element(Plugin):
          if self.__can_build_incrementally() and workspace.last_successful:
              old_dep_keys = self.__get_artifact_metadata_dependencies(workspace.last_successful)
 -        for dep in self.dependencies(scope):
 +        for element in self.dependencies(scope):
              # If we are workspaced, and we therefore perform an
              # incremental build, we must ensure that we update the mtimes
              # of any files created by our dependencies since the last
              # successful build.
              to_update = None
              if workspace and old_dep_keys:
 -                dep.__assert_cached()
 +                element.__assert_cached()
 -                if dep.name in old_dep_keys:
 -                    key_new = dep._get_cache_key()
 -                    key_old = old_dep_keys[dep.name]
 +                if element.name in old_dep_keys:
 +                    key_new = element._get_cache_key()
 +                    key_old = old_dep_keys[element.name]
                      # We only need to worry about modified and added
                      # files, since removed files will be picked up by
                      # build systems anyway.
 -                    to_update, _, added = self.__artifacts.diff(dep, key_old, key_new, subdir='files')
 -                    workspace.add_running_files(dep.name, to_update + added)
 -                    to_update.extend(workspace.running_files[dep.name])
 +                    to_update, _, added = self.__artifacts.diff(element, key_old, key_new, subdir='files')
 +                    workspace.add_running_files(element.name, to_update + added)
 +                    to_update.extend(workspace.running_files[element.name])
                      # In case we are running `bst shell`, this happens in the
                      # main process and we need to update the workspace config
                      if utils._is_main_process():
                          self._get_context().get_workspaces().save_config()
 -            result = dep.stage_artifact(sandbox,
 -                                        path=path,
 -                                        include=include,
 -                                        exclude=exclude,
 -                                        orphans=orphans,
 -                                        update_mtimes=to_update)
 +            result = element.stage_artifact(sandbox,
 +                                            path=path,
 +                                            include=include,
 +                                            exclude=exclude,
 +                                            orphans=orphans,
 +                                            update_mtimes=to_update)
              if result.overwritten:
                  for overwrite in result.overwritten:
                      # Completely new overwrite
@@ -710,13 +710,13 @@ class Element(Plugin):
                          # written the element before
                          for elm, contents in files_written.items():
                              if overwrite in contents:
 -                                overlaps[overwrite] = [elm, dep.name]
 +                                overlaps[overwrite] = [elm, element.name]
                      else:
 -                        overlaps[overwrite].append(dep.name)
 -            files_written[dep.name] = result.files_written
 +                        overlaps[overwrite].append(element.name)
 +            files_written[element.name] = result.files_written
              if result.ignored:
 -                ignored[dep.name] = result.ignored
 +                ignored[element.name] = result.ignored
          if overlaps:
              overlap_warning = False
@@ -1318,17 +1318,24 @@ class Element(Plugin):
                  if scope == Scope.BUILD:
                      self.stage(sandbox)
                  elif scope == Scope.RUN:
++
 +                    if deps == 'build':
 +                        dependency_scope = Scope.BUILD
 +                    elif deps == 'run':
 +                        dependency_scope = Scope.RUN
 +                    else:
 +                        dependency_scope = Scope.NONE
++
                      # Stage deps in the sandbox root
 -                    if deps == 'run':
 -                        with self.timed_activity("Staging dependencies", silent_nested=True):
 -                            self.stage_dependency_artifacts(sandbox, scope)
 +                    with self.timed_activity("Staging dependencies", silent_nested=True):
 +                        self.stage_dependency_artifacts(sandbox, dependency_scope)
 -                        # Run any integration commands provided by the dependencies
 -                        # once they are all staged and ready
 -                        if integrate:
 -                            with self.timed_activity("Integrating sandbox"):
 -                                for dep in self.dependencies(scope):
 -                                    dep.integrate(sandbox)
 +                    # Run any integration commands provided by the dependencies
 +                    # once they are all staged and ready
 +                    if integrate:
 +                        with self.timed_activity("Integrating sandbox"):
 +                            for dep in self.dependencies(dependency_scope):
 +                                dep.integrate(sandbox)
              yield sandbox
@@ -1532,8 +1539,6 @@ class Element(Plugin):
              with _signals.terminator(cleanup_rootdir), \
                  self.__sandbox(rootdir, output_file, output_file, self.__sandbox_config) as sandbox:  # nopep8
 -                sandbox_vroot = sandbox.get_virtual_directory()
+-
                  # By default, the dynamic public data is the same as the static public data.
                  # The plugin's assemble() method may modify this, though.
                  self.__dynamic_public = _yaml.node_copy(self.__public)
@@ -1581,7 +1586,6 @@ class Element(Plugin):
                  finally:
                      if collect is not None:
                          try:
 -                            # Sandbox will probably have replaced its virtual directory, so get it again
                              sandbox_vroot = sandbox.get_virtual_directory()
                              collectvdir = sandbox_vroot.descend(collect.lstrip(os.sep).split(os.sep))
                          except VirtualDirectoryError:
@@ -1606,6 +1610,7 @@ class Element(Plugin):
                          collectvdir.export_files(filesdir, can_link=True)
                      try:
 +                        sandbox_vroot = sandbox.get_virtual_directory()
                          sandbox_build_dir = sandbox_vroot.descend(
                              self.get_variable('build-root').lstrip(os.sep).split(os.sep))
                          # Hard link files from build-root dir to buildtreedir directory

buildstream/element_enums.py

@@ -59,3 +59,7 @@ class Scope(Enum):
      """All elements required for running the element. Including the element
      itself.
      """
++
 +    NONE = 4
 +    """Just the element itself, no dependencies.
 +    """

buildstream/sandbox/sandbox.py

@@ -110,6 +110,10 @@ class Sandbox():
              os.makedirs(directory_, exist_ok=True)
          self._vdir = None
 +        # This is set if anyone requests access to the underlying
 +        # directory via get_directory.
 +        self._never_cache_vdirs = False
++
      def get_directory(self):
          """Fetches the sandbox root directory
@@ -122,24 +126,28 @@ class Sandbox():
          """
          if self.__allow_real_directory:
 +            self._never_cache_vdirs = True
              return self._root
          else:
              raise BstError("You can't use get_directory")
      def get_virtual_directory(self):
 -        """Fetches the sandbox root directory
 +        """Fetches the sandbox root directory as a virtual Directory.
          The root directory is where artifacts for the base
 -        runtime environment should be staged. Only works if
 -        BST_VIRTUAL_DIRECTORY is not set.
 +        runtime environment should be staged.
++
 +        Use caution if you use get_directory and
 +        get_virtual_directory.  If you alter the contents of the
 +        directory returned by get_directory, all objects returned by
 +        get_virtual_directory or derived from them are invalid and you
 +        must call get_virtual_directory again to get a new copy.
          Returns:
 -           (str): The sandbox root directory
 +           (Directory): The sandbox root directory
          """
 -        if not self._vdir:
 -            # BST_CAS_DIRECTORIES is a deliberately hidden environment variable which
 -            # can be used to switch on CAS-based directories for testing.
 +        if self._vdir is None or self._never_cache_vdirs:
              if 'BST_CAS_DIRECTORIES' in os.environ:
                  self._vdir = CasBasedDirectory(self.__context, ref=None)
              else:

tests/frontend/buildcheckout.py

@@ -61,13 +61,14 @@ def test_build_checkout(datafiles, cli, strict, hardlinks):
  @pytest.mark.datafiles(DATA_DIR)
 -@pytest.mark.parametrize("deps", [("run"), ("none")])
 +@pytest.mark.parametrize("deps", [("run"), ("none"), ("build")])
  def test_build_checkout_deps(datafiles, cli, deps):
      project = os.path.join(datafiles.dirname, datafiles.basename)
      checkout = os.path.join(cli.directory, 'checkout')
 +    element_name = "checkout-deps.bst"
      # First build it
 -    result = cli.run(project=project, args=['build', 'target.bst'])
 +    result = cli.run(project=project, args=['build', element_name])
      result.assert_success()
      # Assert that after a successful build, the builddir is empty
@@ -76,20 +77,25 @@ def test_build_checkout_deps(datafiles, cli, deps):
      assert not os.listdir(builddir)
      # Now check it out
 -    result = cli.run(project=project, args=['checkout', 'target.bst', '--deps', deps, checkout])
 +    result = cli.run(project=project, args=['checkout', element_name, '--deps', deps, checkout])
      result.assert_success()
 -    # Check that the executable hello file is found in the checkout
 -    filename = os.path.join(checkout, 'usr', 'bin', 'hello')
 +    # Verify output of this element
 +    filename = os.path.join(checkout, 'etc', 'buildstream', 'config')
 +    if deps == "build":
 +        assert not os.path.exists(filename)
 +    else:
 +        assert os.path.exists(filename)
 -    if deps == "run":
 +    # Verify output of this element's build dependencies
 +    filename = os.path.join(checkout, 'usr', 'include', 'pony.h')
 +    if deps == "build":
          assert os.path.exists(filename)
      else:
          assert not os.path.exists(filename)
 -    # Check that the executable hello file is found in the checkout
 -    filename = os.path.join(checkout, 'usr', 'include', 'pony.h')
+-
 +    # Verify output of this element's runtime dependencies
 +    filename = os.path.join(checkout, 'usr', 'bin', 'hello')
      if deps == "run":
          assert os.path.exists(filename)
      else:

tests/frontend/project/elements/checkout-deps.bst

 +kind: import
 +description: It is important for this element to have both build and runtime dependencies
 +sources:
 +- kind: local
 +  path: files/etc-files
 +depends:
 +- filename: import-dev.bst
 +  type: build
 +- filename: import-bin.bst
 +  type: runtime

tests/frontend/project/files/etc-files/etc/buildstream/config

+config

[Notes] [Git][BuildStream/buildstream][chandan/bst-checkout-build] 12 commits: _artifactcache/cascache.py: Add _ensure_blob helper

Chandan Singh pushed to branch chandan/bst-checkout-build at BuildStream / buildstream

Commits:

10 changed files:

Changes: