[Notes] [Git][BuildStream/buildstream][jennis/quota_declaration

James Ennis pushed to branch jennis/quota_declaration_fix at BuildStream / buildstream

Commits:

2838d080

by Jim MacArthur at 2018-11-08T11:33:35Z

_sandboxremote.py: Use the standard SandboxError.

Replaces the custom one which was erroneously added during development.
Fixes #746.

59b37f70

by James Ennis at 2018-11-08T11:33:35Z

using_config.rst: Update our instructions for how we declare caches in user config

dcb97637

by James Ennis at 2018-11-08T11:33:35Z

format_project.rst: Update instructions regarding how we declare remote caches in project.conf

3847889a

by James Ennis at 2018-11-08T11:33:35Z

using_configuring_artifact_server.rst: Remove examples and redirect to user config/project config

c55de7b3

by Jim MacArthur at 2018-11-08T11:33:35Z

element.py: Use _force_rmtree instead of custom code.

There are arguments for and against using onerror - see issue #735
for details.

e2821417

by James Ennis at 2018-11-08T11:33:35Z

using_config.rst: Add documentation to showing how to impose quotas on the local cache

This patch partially resolves #700

d91c755b

by James Ennis at 2018-11-08T11:44:01Z

artifactcache.py: Fix misleading error message when using % cache quota

6 changed files:

buildstream/_artifactcache/artifactcache.py
buildstream/element.py
buildstream/sandbox/_sandboxremote.py
doc/source/format_project.rst
doc/source/using_config.rst
doc/source/using_configuring_artifact_server.rst

Changes:

buildstream/_artifactcache/artifactcache.py

@@ -937,15 +937,22 @@ class ArtifactCache():
                              "Invalid cache quota ({}): ".format(utils._pretty_size(cache_quota)) +
                              "BuildStream requires a minimum cache quota of 2G.")
          elif cache_quota > cache_size + available_space:  # Check maximum
 +            if '%' in self.context.config_cache_quota:
 +                available = (available_space / (stat.f_blocks * stat.f_bsize)) * 100
 +                available = '{}% of total disk space'.format(round(available,1))
 +            else:
 +                available = utils._pretty_size(available_space)
++
              raise LoadError(LoadErrorReason.INVALID_DATA,
                              ("Your system does not have enough available " +
                               "space to support the cache quota specified.\n" +
 -                             "You currently have:\n" +
 -                             "- {used} of cache in use at {local_cache_path}\n" +
 -                             "- {available} of available system storage").format(
 -                                 used=utils._pretty_size(cache_size),
 -                                 local_cache_path=self.context.artifactdir,
 -                                 available=utils._pretty_size(available_space)))
 +                             "\nYou have specified a quota of {quota} of total disk space.\n" +
 +                             "- The filesystem containing {local_cache_path} only " +
 +                             "has: {available_size} available.")
 +                            .format(
 +                                quota=self.context.config_cache_quota,
 +                                local_cache_path=self.context.artifactdir,
 +                                available_size=available))
          # Place a slight headroom (2e9 (2GB) on the cache_quota) into
          # cache_quota to try and avoid exceptions.

buildstream/element.py

@@ -1410,16 +1410,9 @@ class Element(Plugin):
              finally:
                  # Staging may produce directories with less than 'rwx' permissions
 -                # for the owner, which will break tempfile, so we need to use chmod
 -                # occasionally.
 -                def make_dir_writable(fn, path, excinfo):
 -                    os.chmod(os.path.dirname(path), 0o777)
 -                    if os.path.isdir(path):
 -                        os.rmdir(path)
 -                    else:
 -                        os.remove(path)
 -                shutil.rmtree(temp_staging_directory, onerror=make_dir_writable)
+-
 +                # for the owner, which breaks tempfile. _force_rmtree will deal
 +                # with these.
 +                utils._force_rmtree(temp_staging_directory)
          # Ensure deterministic mtime of sources at build time
          vdirectory.set_deterministic_mtime()
          # Ensure deterministic owners of sources at build time

buildstream/sandbox/_sandboxremote.py

@@ -28,10 +28,7 @@ from ..storage._filebaseddirectory import FileBasedDirectory
  from ..storage._casbaseddirectory import CasBasedDirectory
  from .._protos.build.bazel.remote.execution.v2 import remote_execution_pb2, remote_execution_pb2_grpc
  from .._protos.google.rpc import code_pb2
+-
+-
 -class SandboxError(Exception):
 -    pass
 +from .._exceptions import SandboxError
  # SandboxRemote()

doc/source/format_project.rst

@@ -190,19 +190,34 @@ for more detail.
  Artifact server
  ~~~~~~~~~~~~~~~
  If you have setup an :ref:`artifact server <artifacts>` for your
 -project then it is convenient to configure this in your ``project.conf``
 +project then it is convenient to configure the following in your ``project.conf``
  so that users need not have any additional configuration to communicate
  with an artifact share.
  .. code:: yaml
 +  #
 +  # Artifacts
 +  #
    artifacts:
 +    # A remote cache from which to download prebuilt artifacts
 +    - url: https://foo.com/artifacts:11001
 +      server.cert: server.crt
 +    # A remote cache from which to upload/download built/prebuilt artifacts
 +    - url: https://foo.com/artifacts:11002
 +      server-cert: server.crt
 +      client-cert: client.crt
 +      client-key: client.key
 -    # A url from which to download prebuilt artifacts
 -    url: https://foo.com/artifacts
 +.. note::
++
 +    You can also specify a list of different caches here; earlier entries in the
 +    list will have higher priority than later ones.
++
 +The use of ports are required to distinguish between pull only access and
 +push/pull access. For information regarding the server/client certificates
 +and keys, please see: :ref:`Key pair for the server <server_authentication>`.
 -You can also specify a list of caches here; earlier entries in the list
 -will have higher priority than later ones.
  Remote execution
  ~~~~~~~~~~~~~~~~

doc/source/using_config.rst

@@ -32,38 +32,75 @@ the supported configurations on a project wide basis are listed here.
  Artifact server
  ~~~~~~~~~~~~~~~
 -The project you build will often specify a :ref:`remote artifact cache
 -<artifacts>` already, but you may want to specify extra caches. There are two
 -ways to do this.  You can add one or more global caches:
 +Although project's often specify a :ref:`remote artifact cache <artifacts>` in
 +their ``project.conf``, you may also want to specify extra caches.
 -**Example**
 +Assuming that your host/server is reachable on the internet as ``artifacts.com``
 +(for example), there are two ways to declare remote caches in your user
 +configuration:
++
 +1. Adding global caches:
  .. code:: yaml
 +   #
 +   # Artifacts
 +   #
     artifacts:
 -     url: https://artifacts.com/artifacts
 +     # Add a cache to pull from
 +     - url: https://artifacts.com/artifacts:11001
 +       server-cert: server.crt
 +     # Add a cache to push/pull to/from
 +     - url: https://artifacts.com/artifacts:11002
 +       server-cert: server.crt
 +       client-cert: client.crt
 +       client-key: client.key
 +       push: true
 +     # Add another cache to pull from
 +     - url: https://anothercache.com/artifacts:8080
 +       server-cert: another_server.crt
 -Caches listed there will be considered lower priority than those specified
 -by the project configuration.
 +.. note::
 -You can also add project-specific caches:
 +    Caches declared here will be used by **all** BuildStream project's on the user's
 +    machine and are considered a lower priority than those specified in the project
 +    configuration.
 -**Example**
++
 +2. Specifying caches for a specific project within the user configuration:
  .. code:: yaml
     projects:
       project-name:
         artifacts:
 -         - url: https://artifacts.com/artifacts1
 -         - url: ssh://user artifacts com/artifacts2
 +         # Add a cache to pull from
 +         - url: https://artifacts.com/artifacts:11001
 +           server-cert: server.crt
 +         # Add a cache to push/pull to/from
 +         - url: https://artifacts.com/artifacts:11002
 +           server-cert: server.crt
 +           client-cert: client.crt
 +           client-key: client.key
             push: true
 +         # Add another cache to pull from
 +         - url: https://ourprojectcache.com/artifacts:8080
 +           server-cert: project_server.crt
++
++
 +.. note::
++
 +    Caches listed here will be considered a higher priority than those specified
 +    by the project. Furthermore, for a given list of URLs, earlier entries will
 +    have higher priority.
++
++
 +Notice that the use of different ports for the same server distinguishes between
 +pull only access and push/pull access. For information regarding this and the
 +server/client certificates and keys, please see:
 +:ref:`Key pair for the server <server_authentication>`.
 -Caches listed here will be considered higher priority than those specified
 -by the project.
 -If you give a list of URLs, earlier entries in the list will have higher
 -priority than later ones.
  Strict build plan
  ~~~~~~~~~~~~~~~~~
@@ -110,6 +147,43 @@ The default mirror is defined by its name, e.g.
     ``--default-mirror`` command-line option.
 +Local cache expiry
 +~~~~~~~~~~~~~~~~~~
 +BuildStream locally caches artifacts, build trees, log files and sources within a
 +cache located at ``~/.cache/buildstream`` (unless a $XDG_CACHE_HOME environment
 +variable exists). When building large projects, this cache can get very large,
 +thus BuildStream will attempt to clean up the cache automatically by expiring the least
 +recently *used* artifacts.
++
 +By default, cache expiry will begin once the file system which contains the cache
 +approaches maximum usage. However, it is also possible to impose a quota on the local
 +cache in the user configuration. This can be done in two ways:
++
 +1. By restricting the maximum size of the cache directory itself.
++
 +For example, to ensure that BuildStream's cache does not grow beyong 100 GB,
 +simply declare the following in your user configuration (``~/.config/buildstream.conf``):
++
 +.. code:: yaml
++
 +  cache:
 +    quota: 100G
++
++
 +This quota defines the maximum size of the artifact cache in bytes.
 +Other accepted values are: K, M, G or T (or you can simply declare the value in bytes, without the suffix)
++
 +2. By expiring artifacts once the file system which contains the cache exceeds a specified usage.
++
 +To ensure that we start cleaning the cache once we've used 80% of local disk space (on the file system
 +which mounts the cache):
++
 +.. code:: yaml
++
 +  cache:
 +    quota: 80%
++
++
  Default configuration
  ---------------------
  The default BuildStream configuration is specified here for reference:

doc/source/using_configuring_artifact_server.rst

@@ -98,6 +98,8 @@ Command reference
     :prog: bst-artifact-server
 +.. _server_authentication:
++
  Key pair for the server
  ~~~~~~~~~~~~~~~~~~~~~~~
@@ -237,52 +239,12 @@ We can then check if the services are successfully running with:
  For more information on systemd services see:
  `Creating Systemd Service Files <https://www.devdungeon.com/content/creating-systemd-service-files>`_.
 -User configuration
 -~~~~~~~~~~~~~~~~~~
 -The user configuration for artifacts is documented with the rest
 -of the :ref:`user configuration documentation <user_config>`.
+-
 -Note that for self-signed certificates, the public key fields are mandatory.
+-
 -Assuming you have the same setup used in this document, and that your
 -host is reachable on the internet as ``artifacts.com`` (for example),
 -then a user can use the following user configuration:
+-
 -Pull-only:
+-
 -.. code:: yaml
+-
 -   #
 -   #    Artifacts
 -   #
 -   artifacts:
+-
 -     url: https://artifacts.com:11001
+-
 -     # Optional server certificate if not trusted by system root certificates
 -     server-cert: server.crt
 +Declaring remote artifact caches
 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 +Remote artifact caches can be declared within either:
 -Pull and push:
+-
 -.. code:: yaml
+-
 -   #
 -   #    Artifacts
 -   #
 -   artifacts:
+-
 -     url: https://artifacts.com:11002
+-
 -     # Optional server certificate if not trusted by system root certificates
 -     server-cert: server.crt
+-
 -     # Optional client key pair for authentication
 -     client-key: client.key
 -     client-cert: client.crt
+-
 -     push: true
+-
 -.. note::
 +1. The :ref:`project configuration <project_essentials_artifacts>`, or
 +2. The :ref:`user configuration <config_artifacts>`.
 -    Equivalent statements can be delcared in a project's configuration file
 -    (the ``project.conf``).
 +Please follow the above links to see examples showing how we declare remote
 +caches in both the project configuration and the user configuration, respectively.

[Notes] [Git][BuildStream/buildstream][jennis/quota_declaration_fix] 7 commits: _sandboxremote.py: Use the standard SandboxError.

James Ennis pushed to branch jennis/quota_declaration_fix at BuildStream / buildstream

Commits:

6 changed files:

Changes: