[Notes] [Git][BuildStream/buildstream][tpollard/829] 7 commits:

Tom Pollard pushed to branch tpollard/829 at BuildStream / buildstream

Commits:

43797617

by Richard Maw at 2019-01-21T10:41:37Z

_gitsourcebase.py: Fetch with explicit refspecs

Old versions of git lack --force and --tags,
but the same effect can be had by specifying refspecs.

ce8dab0f

by Richard Maw at 2019-01-21T10:41:37Z

tests/frontend/workspace.py: Skip test_open_multi_unwritable when root

The test assumes that a directory with write permission removed isn't writable,
this isn't the case if the process running the tests has CAP_DAC_OVERRIDE
which is common when running as root.

50165081

by Richard Maw at 2019-01-21T10:41:37Z

tests/sources/git.py: Skip tests that assume too new a git

test_track_invalid_submodule depends on being able to remove a submodule
by `git rm $submoduledir`, but old versions of git don't update .gitmodules
so BuildStream still thinks there's a submodule present.

For expediency the test is skipped rather than changed to manually remove
the entry from .gitmodules if git hasn't done it,
since in the common case git is new enough to do that itself.

test_git_describe expects --first-parent to find another tag,
but `bst track` will gracefully degrade if the option doesn't work
so a different history will be retained with old versions of git.

It's of marginal benefit to add additional cruft
to test for different output on old versions of git that won't persist forever.

8677a256

by Benjamin Schubert at 2019-01-21T10:41:37Z

Add a Centos runner

Centos is apparently different enough from fedora when running tests.

33782865

by Valentin David at 2019-01-21T11:47:18Z

Merge branch 'richardmaw/centos-oldgit-test-fixes' into 'master'

Fix CentOS

Closes #833

See merge request BuildStream/buildstream!1085

098b777d

by Tom Pollard at 2019-01-21T12:54:27Z

_stream.py: Add use_artifact_config opt arg for load_selection()

use_artifact_config added as an optional default arg, allowing
for loading of given elements artifact remote config.

d0c98ed5

by Tom Pollard at 2019-01-21T12:54:27Z

Download buildtrees on demand for bst shell --use-buildtree

Provide bst shell --use-buildtree the ability to attempt to
acquire missing buildtrees, given respective option, user
pull-buildtree context and remote availability.

_frontend/cli.py: Refactor logic for determining --use-buildtree
option with given opportunity to attempt pulling a non-local
buildtree. Element loaded with artifact_config to allow remote
querying.

_stream.py: With given user option and element state, construct
PullQueue to fetch remote buildtree. Continue or Error without
buildtree if cannot be attained.

tests/integration/build-tree.py: Update to support new usecases

Changes:

.gitlab-ci.yml

@@ -31,6 +31,7 @@ variables:
    - df -h
    script:
 +  - mkdir -p "${INTEGRATION_CACHE}"
    - useradd -Um buildstream
    - chown -R buildstream:buildstream .
@@ -70,6 +71,10 @@ tests-python-3.7-stretch:
      # some of our base dependencies declare it as their runtime dependency.
      TOXENV: py37
 +tests-centos-7.6:
 +  <<: *tests
 +  image: buildstream/testsuite-centos:7.6-5da27168-32c47d1c
++
  overnight-fedora-28-aarch64:
    image: buildstream/testsuite-fedora:aarch64-28-5da27168-32c47d1c
    tags:

buildstream/_frontend/cli.py

@@ -579,7 +579,7 @@ def shell(app, element, sysroot, mount, isolate, build_, cli_buildtree, command)
      else:
          scope = Scope.RUN
 -    use_buildtree = False
 +    use_buildtree = None
      with app.initialized():
          if not element:
@@ -587,7 +587,8 @@ def shell(app, element, sysroot, mount, isolate, build_, cli_buildtree, command)
              if not element:
                  raise AppError('Missing argument "ELEMENT".')
 -        dependencies = app.stream.load_selection((element,), selection=PipelineSelection.NONE)
 +        dependencies = app.stream.load_selection((element,), selection=PipelineSelection.NONE,
 +                                                 use_artifact_config=True)
          element = dependencies[0]
          prompt = app.shell_prompt(element)
          mounts = [
@@ -596,18 +597,19 @@ def shell(app, element, sysroot, mount, isolate, build_, cli_buildtree, command)
+         ]
          cached = element._cached_buildtree()
 -        if cli_buildtree == "always":
 -            if cached:
 -                use_buildtree = True
 -            else:
 -                raise AppError("No buildtree is cached but the use buildtree option was specified")
 -        elif cli_buildtree == "never":
 -            pass
 -        elif cli_buildtree == "try":
 -            use_buildtree = cached
 +        if cli_buildtree in ("always", "try"):
 +            use_buildtree = cli_buildtree
 +            if not cached and use_buildtree == "always":
 +                click.echo("Warning: buildtree is not cached locally, will attempt to pull from available remotes")
          else:
 -            if app.interactive and cached:
 -                use_buildtree = bool(click.confirm('Do you want to use the cached buildtree?'))
 +            # If the value has defaulted to ask and in non interactive mode, don't consider the buildtree, this
 +            # being the default behaviour of the command
 +            if app.interactive and cli_buildtree == "ask":
 +                if cached and bool(click.confirm('Do you want to use the cached buildtree?')):
 +                    use_buildtree = "always"
 +                elif not cached and bool(click.confirm('Do you want to attempt to pull the cached buildtree?')):
 +                    use_buildtree = "try"
++
          if use_buildtree and not element._cached_success():
              click.echo("Warning: using a buildtree from a failed build.")

buildstream/_gitsourcebase.py

@@ -112,7 +112,8 @@ class GitMirror(SourceFetcher):
          else:
              remote_name = "origin"
 -        self.source.call([self.source.host_git, 'fetch', remote_name, '--prune', '--force', '--tags'],
 +        self.source.call([self.source.host_git, 'fetch', remote_name, '--prune',
 +                          '+refs/heads/*:refs/heads/*', '+refs/tags/*:refs/tags/*'],
                           fail="Failed to fetch from remote git repository: {}".format(url),
                           fail_temporarily=True,
                           cwd=self.mirror)

buildstream/_stream.py

@@ -100,16 +100,19 @@ class Stream():
      #    targets (list of str): Targets to pull
      #    selection (PipelineSelection): The selection mode for the specified targets
      #    except_targets (list of str): Specified targets to except from fetching
 +    #    use_artifact_config (bool): If artifact remote config should be loaded
+     #
      # Returns:
      #    (list of Element): The selected elements
      def load_selection(self, targets, *,
                         selection=PipelineSelection.NONE,
 -                       except_targets=()):
 +                       except_targets=(),
 +                       use_artifact_config=False):
          elements, _ = self._load(targets, (),
                                   selection=selection,
                                   except_targets=except_targets,
 -                                 fetch_subprojects=False)
 +                                 fetch_subprojects=False,
 +                                 use_artifact_config=use_artifact_config)
          return elements
      # shell()
@@ -124,7 +127,7 @@ class Stream():
      #    mounts (list of HostMount): Additional directories to mount into the sandbox
      #    isolate (bool): Whether to isolate the environment like we do in builds
      #    command (list): An argv to launch in the sandbox, or None
 -    #    usebuildtree (bool): Wheather to use a buildtree as the source.
 +    #    usebuildtree (str): Whether to use a buildtree as the source, given cli option
+     #
      # Returns:
      #    (int): The exit code of the launched shell
@@ -134,7 +137,7 @@ class Stream():
                mounts=None,
                isolate=False,
                command=None,
 -              usebuildtree=False):
 +              usebuildtree=None):
          # Assert we have everything we need built, unless the directory is specified
          # in which case we just blindly trust the directory, using the element
@@ -149,8 +152,30 @@ class Stream():
                  raise StreamError("Elements need to be built or downloaded before staging a shell environment",
                                    detail="\n".join(missing_deps))
 +        # Check if we require a pull queue attempt, with given artifact state and context
 +        if usebuildtree:
 +            if not element._cached_buildtree():
 +                buildtree = False
 +                require_buildtree = self._buildtree_pull_required([element])
 +                # Attempt a pull queue for the given element if remote and context allow it
 +                if require_buildtree:
 +                    self._message(MessageType.INFO, "Attempting to fetch missing artifact buildtree")
 +                    self._add_queue(PullQueue(self._scheduler))
 +                    self._enqueue_plan(require_buildtree)
 +                    self._run()
 +                    # Now check if the buildtree was successfully fetched
 +                    if element._cached_buildtree():
 +                        buildtree = True
 +                if not buildtree:
 +                    if usebuildtree == "always":
 +                        raise StreamError("Buildtree is not cached locally or in available remotes")
 +                    else:
 +                        self._message(MessageType.INFO, """Buildtree is not cached locally or in available remotes,
 +                                                        shell will be loaded without it""")
 +                        usebuildtree = False
++
          return element._shell(scope, directory, mounts=mounts, isolate=isolate, prompt=prompt, command=command,
 -                              usebuildtree=usebuildtree)
 +                              usebuildtree=bool(usebuildtree))
      # build()
+     #

tests/frontend/workspace.py

@@ -204,6 +204,7 @@ def test_open_multi(cli, tmpdir, datafiles):
              assert not ('.bzr' in workspace_lsdir)
 +@pytest.mark.skipif(os.geteuid() == 0, reason="root may have CAP_DAC_OVERRIDE and ignore permissions")
  @pytest.mark.datafiles(DATA_DIR)
  def test_open_multi_unwritable(cli, tmpdir, datafiles):
      workspace_object = WorkspaceCreater(cli, tmpdir, datafiles)

tests/integration/build-tree.py

@@ -140,7 +140,7 @@ def test_buildtree_pulled(cli, tmpdir, datafiles):
          res.assert_success()
 -# This test checks for correct behaviour if a buildtree is not present.
 +# This test checks for correct behaviour if a buildtree is not present in the local cache.
  @pytest.mark.datafiles(DATA_DIR)
  @pytest.mark.skipif(IS_LINUX and not HAVE_BWRAP, reason='Only available with bubblewrap on Linux')
  def test_buildtree_options(cli, tmpdir, datafiles):
@@ -155,6 +155,7 @@ def test_buildtree_options(cli, tmpdir, datafiles):
          result = cli.run(project=project, args=['build', element_name])
          result.assert_success()
          assert cli.get_element_state(project, element_name) == 'cached'
 +        assert share.has_artifact('test', element_name, cli.get_element_key(project, element_name))
          # Discard the cache
          cli.configure({
@@ -167,8 +168,6 @@ def test_buildtree_options(cli, tmpdir, datafiles):
          result = cli.run(project=project, args=['pull', '--deps', 'all', element_name])
          result.assert_success()
 -        # The above is the simplest way I know to create a local cache without any buildtrees.
+-
          # Check it's not using the cached build tree
          res = cli.run(project=project, args=[
              'shell', '--build', element_name, '--use-buildtree', 'never', '--', 'cat', 'test'
@@ -176,24 +175,51 @@ def test_buildtree_options(cli, tmpdir, datafiles):
          res.assert_shell_error()
          assert 'Hi' not in res.output
 -        # Check it's not correctly handling the lack of buildtree
 +        # Check it's not using the cached build tree, default is to ask, and fall back to not
 +        # for non interactive behavior
          res = cli.run(project=project, args=[
 -            'shell', '--build', element_name, '--use-buildtree', 'try', '--', 'cat', 'test'
 +            'shell', '--build', element_name, '--', 'cat', 'test'
          ])
          res.assert_shell_error()
          assert 'Hi' not in res.output
 -        # Check it's not using the cached build tree, default is to ask, and fall back to not
 -        # for non interactive behavior
 +        # Check correctly handling the lack of buildtree, with 'try' not attempting to
 +        # pull the buildtree as the user context is by default set to not pull them
          res = cli.run(project=project, args=[
 -            'shell', '--build', element_name, '--', 'cat', 'test'
 +            'shell', '--build', element_name, '--use-buildtree', 'try', '--', 'cat', 'test'
          ])
 -        res.assert_shell_error()
          assert 'Hi' not in res.output
 +        assert 'Attempting to fetch missing artifact buildtrees' not in res.stderr
 +        assert """Buildtree is not cached locally or in available remotes,
 +                shell will be loaded without it"""
 -        # Check it's using the cached build tree
 +        # Check correctly handling the lack of buildtree, with 'try' attempting and succeeding
 +        # to pull the buildtree as the user context allow the pulling of buildtrees and it is
 +        # available in the remote
 +        res = cli.run(project=project, args=[
 +            '--pull-buildtrees', 'shell', '--build', element_name, '--use-buildtree', 'try', '--', 'cat', 'test'
 +        ])
 +        assert 'Attempting to fetch missing artifact buildtree' in res.stderr
 +        assert 'Hi' in res.output
 +        shutil.rmtree(os.path.join(os.path.join(cli.directory, 'artifacts2')))
 +        assert cli.get_element_state(project, element_name) != 'cached'
++
 +        # Check it's not loading the shell at all with always set for the buildtree, when the
 +        # user context does not allow for buildtree pulling
 +        result = cli.run(project=project, args=['pull', '--deps', 'all', element_name])
 +        result.assert_success()
          res = cli.run(project=project, args=[
              'shell', '--build', element_name, '--use-buildtree', 'always', '--', 'cat', 'test'
          ])
          res.assert_main_error(ErrorDomain.PROG_NOT_FOUND, None)
 +        assert 'Buildtree is not cached locally or in available remotes' in res.stderr
          assert 'Hi' not in res.output
 +        assert 'Attempting to fetch missing artifact buildtree' not in res.stderr
++
 +        # Check that when user context is set to pull buildtrees and a remote has the buildtree,
 +        # 'always' will attempt and succeed at pulling the missing buildtree.
 +        res = cli.run(project=project, args=[
 +            '--pull-buildtrees', 'shell', '--build', element_name, '--use-buildtree', 'always', '--', 'cat', 'test'
 +        ])
 +        assert 'Hi' in res.output
 +        assert 'Attempting to fetch missing artifact buildtree' in res.stderr

tests/sources/git.py

@@ -30,7 +30,7 @@ from buildstream import _yaml
  from buildstream.plugin import CoreWarnings
  from tests.testutils import cli, create_repo
 -from tests.testutils.site import HAVE_GIT
 +from tests.testutils.site import HAVE_GIT, HAVE_OLD_GIT
  DATA_DIR = os.path.join(
      os.path.dirname(os.path.realpath(__file__)),
@@ -664,6 +664,7 @@ def test_invalid_submodule(cli, tmpdir, datafiles, fail):
  @pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
 +@pytest.mark.skipif(HAVE_OLD_GIT, reason="old git rm does not update .gitmodules")
  @pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
  @pytest.mark.parametrize("fail", ['warn', 'error'])
  def test_track_invalid_submodule(cli, tmpdir, datafiles, fail):
@@ -772,6 +773,7 @@ def test_track_fetch(cli, tmpdir, datafiles, ref_format, tag, extra_commit):
  @pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
 +@pytest.mark.skipif(HAVE_OLD_GIT, reason="old git describe lacks --first-parent")
  @pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
  @pytest.mark.parametrize("ref_storage", [('inline'), ('project.refs')])
  @pytest.mark.parametrize("tag_type", [('annotated'), ('lightweight')])

tests/testutils/site.py

@@ -2,6 +2,7 @@
  # so we dont have to repeat this everywhere
+ #
  import os
 +import subprocess
  import sys
  from buildstream import _site, utils, ProgramNotFoundError
@@ -16,8 +17,12 @@ except ProgramNotFoundError:
  try:
      utils.get_host_tool('git')
      HAVE_GIT = True
 +    out = str(subprocess.check_output(['git', '--version']), "utf-8")
 +    version = tuple(int(x) for x in out.split(' ', 2)[2].split('.'))
 +    HAVE_OLD_GIT = version < (1, 8, 5)
  except ProgramNotFoundError:
      HAVE_GIT = False
 +    HAVE_OLD_GIT = False
  try:
      utils.get_host_tool('ostree')

[Notes] [Git][BuildStream/buildstream][tpollard/829] 7 commits: _gitsourcebase.py: Fetch with explicit refspecs

Tom Pollard pushed to branch tpollard/829 at BuildStream / buildstream

Commits:

8 changed files:

Changes: