[Notes] [Git][BuildStream/buildstream][valentindavid/remote_execution

Valentin David pushed to branch valentindavid/remote_execution_configuration at BuildStream / buildstream

Commits:

bb9df2a1

by Chandan Singh at 2019-01-02T11:59:35Z

doc/Makefile: Update comment about sphinx entrypoint

https://github.com/sphinx-doc/sphinx/issues/4375 has been closed as
`wontfix`. Update the comment in doc/Makefile that mentions it, based on
the discussion on the issue.

43eaf6ee

by Chandan Singh at 2019-01-02T13:42:10Z

Merge branch 'chandan/update-doc-makefile-note' into 'master'

doc/Makefile: Update comment about sphinx entrypoint

See merge request BuildStream/buildstream!1026

3b5c8a28

by Tristan Maat at 2019-01-02T15:57:13Z

.gitlab-ci.yml: Use new testsuite images with pycodestyle 1.4.0

We need to bump the testsuite image versions to use the new
pycodestyle update that allows excluding file paths, so that we can
exclude generated python code.

bd60e8a9

by Javier Jardón at 2019-01-02T16:59:41Z

Use pycodestyle instead of pep8

pep8 package was renamed to pycodestyle to reduce confusion.

Disabled warnings:
W504 - Line break after binary operator (seems people like this)
W605 - Invalid escape sequence (some of our regexes use these)

32c47d1c

by Tristan Maat at 2019-01-02T17:32:09Z

Merge branch 'jjardon/pycodestyle' into 'master'

Use pycodestyle instead pep8 python module

See merge request BuildStream/buildstream!638

ada0c70b

by Valentin David at 2019-01-03T12:32:13Z

Use relative path to project directory for remote execution certificates/keys

035cf501

by Valentin David at 2019-01-03T12:32:13Z

Add support for user remote execution configuration

Fixes #631.

7f0ae3ca

by Valentin David at 2019-01-03T12:32:13Z

Add support for https channel to remote execution and actions servers

Fixes #780.

Changes:

.gitlab-ci.yml

 -image: buildstream/testsuite-debian:9-master-123-7ce6581b
 +image: buildstream/testsuite-debian:9-06bab030-b848172c
  cache:
    key: "$CI_JOB_NAME-"
@@ -91,23 +91,23 @@ source_dist:
      - ${COVERAGE_DIR}
  tests-debian-9:
 -  image: buildstream/testsuite-debian:9-master-123-7ce6581b
 +  image: buildstream/testsuite-debian:9-06bab030-b848172c
    <<: *tests
  tests-fedora-27:
 -  image: buildstream/testsuite-fedora:27-master-123-7ce6581b
 +  image: buildstream/testsuite-fedora:27-06bab030-b848172c
    <<: *tests
  tests-fedora-28:
 -  image: buildstream/testsuite-fedora:28-master-123-7ce6581b
 +  image: buildstream/testsuite-fedora:28-06bab030-b848172c
    <<: *tests
  tests-ubuntu-18.04:
 -  image: buildstream/testsuite-ubuntu:18.04-master-123-7ce6581b
 +  image: buildstream/testsuite-ubuntu:18.04-06bab030-b848172c
    <<: *tests
  overnight-fedora-28-aarch64:
 -  image: buildstream/testsuite-fedora:aarch64-28-master-123-7ce6581b
 +  image: buildstream/testsuite-fedora:aarch64-28-06bab030-32a101f6
    tags:
      - aarch64
    <<: *tests
@@ -120,7 +120,7 @@ overnight-fedora-28-aarch64:
  tests-unix:
    # Use fedora here, to a) run a test on fedora and b) ensure that we
    # can get rid of ostree - this is not possible with debian-8
 -  image: buildstream/testsuite-fedora:27-master-123-7ce6581b
 +  image: buildstream/testsuite-fedora:27-06bab030-b848172c
    <<: *tests
    variables:
      BST_FORCE_BACKEND: "unix"
@@ -140,7 +140,7 @@ tests-unix:
  tests-fedora-missing-deps:
    # Ensure that tests behave nicely while missing bwrap and ostree
 -  image: buildstream/testsuite-fedora:28-master-123-7ce6581b
 +  image: buildstream/testsuite-fedora:28-06bab030-b848172c
    <<: *tests
    script:

buildstream/_context.py

@@ -34,6 +34,7 @@ from ._artifactcache import ArtifactCache
  from ._artifactcache.cascache import CASCache
  from ._workspaces import Workspaces, WorkspaceProjectCache, WORKSPACE_PROJECT_FILE
  from .plugin import _plugin_lookup
 +from .sandbox import SandboxRemote
  # Context()
@@ -72,6 +73,9 @@ class Context():
          # The locations from which to push and pull prebuilt artifacts
          self.artifact_cache_specs = None
 +        # The global remote execution configuration
 +        self.remote_execution_specs = None
++
          # The directory to store build logs
          self.logdir = None
@@ -187,7 +191,7 @@ class Context():
          _yaml.node_validate(defaults, [
              'sourcedir', 'builddir', 'artifactdir', 'logdir',
              'scheduler', 'artifacts', 'logging', 'projects',
 -            'cache', 'prompt', 'workspacedir',
 +            'cache', 'prompt', 'workspacedir', 'remote-execution'
          ])
          for directory in ['sourcedir', 'builddir', 'artifactdir', 'logdir', 'workspacedir']:
@@ -212,6 +216,8 @@ class Context():
          # Load artifact share configuration
          self.artifact_cache_specs = ArtifactCache.specs_from_config_node(defaults)
 +        self.remote_execution_specs = SandboxRemote.specs_from_config_node(defaults)
++
          # Load pull build trees configuration
          self.pull_buildtrees = _yaml.node_get(cache, bool, 'pull-buildtrees')
@@ -271,7 +277,8 @@ class Context():
          # Shallow validation of overrides, parts of buildstream which rely
          # on the overrides are expected to validate elsewhere.
          for _, overrides in _yaml.node_items(self._project_overrides):
 -            _yaml.node_validate(overrides, ['artifacts', 'options', 'strict', 'default-mirror'])
 +            _yaml.node_validate(overrides, ['artifacts', 'options', 'strict', 'default-mirror',
 +                                            'remote-execution'])
          profile_end(Topics.LOAD_CONTEXT, 'load')

buildstream/_ostree.py

@@ -34,7 +34,7 @@ from ._exceptions import BstError, ErrorDomain
  # pylint: disable=wrong-import-position,wrong-import-order
  gi.require_version('OSTree', '1.0')
 -from gi.repository import GLib, Gio, OSTree  # nopep8
 +from gi.repository import GLib, Gio, OSTree  # noqa
  # For users of this file, they must expect (except) it.

buildstream/_project.py

@@ -507,7 +507,16 @@ class Project():
          self.artifact_cache_specs = ArtifactCache.specs_from_config_node(config, self.directory)
          # Load remote-execution configuration for this project
 -        self.remote_execution_specs = SandboxRemote.specs_from_config_node(config, self.directory)
 +        project_specs = SandboxRemote.specs_from_config_node(config, self.directory)
 +        override_specs = SandboxRemote.specs_from_config_node(
 +            self._context.get_overrides(self.name), self.directory)
++
 +        if override_specs is not None:
 +            self.remote_execution_specs = override_specs
 +        elif project_specs is not None:
 +            self.remote_execution_specs = project_specs
 +        else:
 +            self.remote_execution_specs = self._context.remote_execution_specs
          # Load sandbox environment variables
          self.base_environment = _yaml.node_get(config, Mapping, 'environment')

buildstream/_signals.py

@@ -38,7 +38,7 @@ def terminator_handler(signal_, frame):
          terminator_ = terminator_stack.pop()
          try:
              terminator_()
 -        except:                                                 # pylint: disable=bare-except
 +        except:                               # noqa pylint: disable=bare-except
              # Ensure we print something if there's an exception raised when
              # processing the handlers. Note that the default exception
              # handler won't be called because we os._exit next, so we must

buildstream/element.py

@@ -1568,7 +1568,7 @@ class Element(Plugin):
                  utils._force_rmtree(rootdir)
              with _signals.terminator(cleanup_rootdir), \
 -                self.__sandbox(rootdir, output_file, output_file, self.__sandbox_config) as sandbox:  # nopep8
 +                self.__sandbox(rootdir, output_file, output_file, self.__sandbox_config) as sandbox:  # noqa
                  # By default, the dynamic public data is the same as the static public data.
                  # The plugin's assemble() method may modify this, though.

buildstream/sandbox/_sandboxremote.py

@@ -41,7 +41,7 @@ from .._protos.google.longrunning import operations_pb2, operations_pb2_grpc
  from .._artifactcache.cascache import CASRemote, CASRemoteSpec
 -class RemoteExecutionSpec(namedtuple('RemoteExecutionSpec', 'exec_service storage_service action_service')):
 +class RemoteExecutionSpec(namedtuple('RemoteExecutionSpec', 'exec_service storage_service action_service basedir')):
      pass
@@ -59,29 +59,59 @@ class SandboxRemote(Sandbox):
          if config is None:
              return
 +        def resolve_path(path):
 +            if config.basedir and path:
 +                return os.path.join(config.basedir, path)
 +            else:
 +                return path
++
          self.storage_url = config.storage_service['url']
          self.exec_url = config.exec_service['url']
 +        exec_certs = {}
 +        for key in ['client-cert', 'client-key', 'server-cert']:
 +            if key in config.exec_service:
 +                with open(resolve_path(config.exec_service[key]), 'rb') as f:
 +                    exec_certs[key] = f.read()
++
 +        self.exec_credentials = grpc.ssl_channel_credentials(
 +            root_certificates=exec_certs.get('server-cert'),
 +            private_key=exec_certs.get('client-key'),
 +            certificate_chain=exec_certs.get('client-cert'))
++
 +        action_certs = {}
 +        for key in ['client-cert', 'client-key', 'server-cert']:
 +            if key in config.action_service:
 +                with open(resolve_path(config.exec_service[key]), 'rb') as f:
 +                    action_certs[key] = f.read()
++
          if config.action_service:
              self.action_url = config.action_service['url']
 +            self.action_credentials = grpc.ssl_channel_credentials(
 +                root_certificates=action_certs.get('server-cert'),
 +                private_key=action_certs.get('client-key'),
 +                certificate_chain=action_certs.get('client-cert'))
          else:
              self.action_url = None
 +            self.action_credentials = None
          self.server_instance = config.exec_service.get('instance', None)
          self.storage_instance = config.storage_service.get('instance', None)
 -        self.storage_remote_spec = CASRemoteSpec(self.storage_url, push=True,
 -                                                 server_cert=config.storage_service['server-cert'],
 -                                                 client_key=config.storage_service['client-key'],
 -                                                 client_cert=config.storage_service['client-cert'],
 -                                                 instance_name=self.storage_instance)
 +        self.storage_remote_spec = CASRemoteSpec(
 +            self.storage_url, push=True,
 +            server_cert=resolve_path(config.storage_service['server-cert']),
 +            client_key=resolve_path(config.storage_service['client-key']),
 +            client_cert=resolve_path(config.storage_service['client-cert']),
 +            instance_name=self.storage_instance)
++
          self.operation_name = None
      def info(self, msg):
          self._get_context().message(Message(None, MessageType.INFO, msg))
      @staticmethod
 -    def specs_from_config_node(config_node, basedir):
 +    def specs_from_config_node(config_node, basedir=None):
          def require_node(config, keyname):
              val = config.get(keyname)
@@ -109,7 +139,7 @@ class SandboxRemote(Sandbox):
          remote_exec_storage_config = require_node(remote_config, 'storage-service')
          remote_exec_action_config = remote_config.get('action-cache-service', {})
 -        _yaml.node_validate(remote_exec_service_config, ['url', 'instance'])
 +        _yaml.node_validate(remote_exec_service_config, ['url', 'instance'] + tls_keys)
          _yaml.node_validate(remote_exec_storage_config, ['url', 'instance'] + tls_keys)
          if remote_exec_action_config:
              _yaml.node_validate(remote_exec_action_config, ['url'])
@@ -137,7 +167,8 @@ class SandboxRemote(Sandbox):
          spec = RemoteExecutionSpec(remote_config['execution-service'],
                                     remote_config['storage-service'],
 -                                   remote_exec_action_config)
 +                                   remote_exec_action_config,
 +                                   basedir)
          return spec
      def run_remote_command(self, channel, action_digest):
@@ -295,6 +326,8 @@ class SandboxRemote(Sandbox):
                                 "for example: http://buildservice:50051.")
          if url.scheme == 'http':
              channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
 +        elif url.scheme == 'https':
 +            channel = grpc.secure_channel('{}:{}'.format(url.hostname, url.port), self.exec_credentials)
          else:
              raise SandboxError("Remote execution currently only supports the 'http' protocol "
                                 "and '{}' was supplied.".format(url.scheme))
@@ -352,11 +385,11 @@ class SandboxRemote(Sandbox):
          if not url.port:
              raise SandboxError("You must supply a protocol and port number in the action-cache-service url, "
                                 "for example: http://buildservice:50051.")
 -        if not url.scheme == "http":
 -            raise SandboxError("Currently only support http for the action cache"
 -                               "and {} was supplied".format(url.scheme))
 +        if url.scheme == 'http':
 +            channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
 +        elif url.scheme == 'https':
 +            channel = grpc.secure_channel('{}:{}'.format(url.hostname, url.port), self.action_credentials)
 -        channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
          request = remote_execution_pb2.GetActionResultRequest(action_digest=action_digest)
          stub = remote_execution_pb2_grpc.ActionCacheStub(channel)
          try:

dev-requirements.txt

  coverage == 4.4.0
 -pep8
  pylint
  pytest >= 3.9
 +pytest-codestyle >= 1.4.0
  pytest-cov >= 2.5.0
  pytest-datafiles >= 2.0
  pytest-env
 -pytest-pep8
  pytest-pylint
  pytest-xdist
  pytest-timeout

doc/Makefile

@@ -6,10 +6,10 @@
  # we dont use the standard `sphinx-build` and `sphinx-apidoc` entry points.
+ #
  # The following technique works as long as sphinx is installed for python3,
 -# regardless of the entry point which might be in /usr/bin or PATH, but
 -# will stop working in sphinx >= 2.0. Hopefully by then, the mentioned
 -# bug will be fixed and we can use a standard python3 specific script to
 -# invoke sphnix.
 +# regardless of the entry point which might be in /usr/bin or PATH.
 +#
 +# Since Sphinx 2.0 is planned to be Python 3-only, this workaround should not
 +# be needed once Spinx 2.0 is released, and we upgrade to it
+ #
  SPHINXOPTS    =
  SPHINXBUILD   = python3 -m sphinx

doc/source/format_project.rst

@@ -218,6 +218,7 @@ The use of ports are required to distinguish between pull only access and
  push/pull access. For information regarding the server/client certificates
  and keys, please see: :ref:`Key pair for the server <server_authentication>`.
 +.. _project_remote_execution:
  Remote execution
  ~~~~~~~~~~~~~~~~
@@ -243,9 +244,6 @@ using the `remote-execution` option:
      action-cache-service:
        url: http://bar.action.com:50052
 -The execution-service part of remote execution does not support encrypted
 -connections yet, so the protocol must always be http.
+-
  storage-service specifies a remote CAS store and the parameters are the
  same as those used to specify an :ref:`artifact server <artifacts>`.
@@ -268,6 +266,9 @@ instance names.
  The Remote Execution API can be found via https://github.com/bazelbuild/remote-apis.
 +Remote execution configuration can be also provided in the `user
 +configuration <user_config_remote_execution>`.
++
  .. _project_essentials_mirrors:
  Mirrors

doc/source/using_config.rst

@@ -100,6 +100,54 @@ pull only access and push/pull access. For information regarding this and the
  server/client certificates and keys, please see:
  :ref:`Key pair for the server <server_authentication>`.
 +.. _user_config_remote_execution:
++
 +Remote execution
 +~~~~~~~~~~~~~~~~
++
 +The same configuration for :ref:`remote execution <project_remote_execution>`
 +in ``project.conf`` can be provided in the user configuation.
++
 +There is only one remote execution configuration used per project.
++
 +The project overrides will be taken in priority. The global
 +configuration will be used as fallback.
++
 +1. Global remote execution fallback:
++
 +.. code:: yaml
++
 +  remote-execution:
 +    execution-service:
 +      url: http://execution.fallback.example.com:50051
 +      instance-name: main
 +    storage-service:
 +      url: https://storage.fallback.example.com:11002/
 +      server-cert: /keys/server.crt
 +      client-cert: /keys/client.crt
 +      client-key: /keys/client.key
 +      instance-name: main
 +    action-cache-service:
 +      url: http://action.flalback.example.com:50052
++
 +2. Project override:
++
 +.. code:: yaml
++
 +  projects:
 +    some_project:
 +      remote-execution:
 +        execution-service:
 +          url: http://execution.some_project.example.com:50051
 +          instance-name: main
 +        storage-service:
 +          url: https://storage.some_project.example.com:11002/
 +          server-cert: /some_project_keys/server.crt
 +          client-cert: /some_project_keys/client.crt
 +          client-key: /some_project_keys/client.key
 +          instance-name: main
 +        action-cache-service:
 +          url: http://action.some_project.example.com:50052
  Strict build plan

setup.cfg

@@ -11,20 +11,11 @@ parentdir_prefix = BuildStream-
  test=pytest
  [tool:pytest]
 -addopts = --verbose --basetemp ./tmp --pep8 --pylint --pylint-rcfile=.pylintrc --cov=buildstream --cov-config .coveragerc --durations=20
 +addopts = --verbose --basetemp ./tmp --codestyle --pylint --pylint-rcfile=.pylintrc --cov=buildstream --cov-config .coveragerc --durations=20
  norecursedirs = tests/integration/project integration-cache tmp __pycache__ .eggs
  python_files = tests/*/*.py
 -pep8maxlinelength = 119
 -pep8ignore =
 -    * E129
 -    * E125
 -    doc/source/conf.py ALL
 -    tmp/* ALL
 -    */lib/python3* ALL
 -    */bin/* ALL
 -    buildstream/_fuse/fuse.py ALL
 -    .eggs/* ALL
 -    *_pb2.py ALL
 -    *_pb2_grpc.py ALL
 +codestyle_max_line_length = 119
 +codestyle_ignore = E129 E125 W504 W605
 +codestyle_exclude = doc/source/conf.py buildstream/_fuse/fuse.py buildstream/_protos/**/*py
  env =
      D:BST_TEST_SUITE=True

tests/testutils/http_server.py

@@ -29,7 +29,7 @@ class RequestHandler(SimpleHTTPRequestHandler):
                  expected_password, directory = self.server.users[user]
                  if password == expected_password:
                      return directory
 -            except:
 +            except:                           # noqa
                  raise Unauthorized('unauthorized')
              return None

[Notes] [Git][BuildStream/buildstream][valentindavid/remote_execution_configuration] 8 commits: doc/Makefile: Update comment about sphinx entrypoint

Valentin David pushed to branch valentindavid/remote_execution_configuration at BuildStream / buildstream

Commits:

13 changed files:

Changes: