[Notes] [Git][BuildStream/buildstream][bschubert/pipeline] 9 commits:

Benjamin Schubert pushed to branch bschubert/pipeline at BuildStream / buildstream

Commits:

542cdaf0

by Jürg Billeter at 2019-01-14T18:46:57Z

_frontend/cli.py: Also check original args for --config in bash completion

The path of the config file generated by testutils for completion tests
is passed as regular argument, not via COMP_WORDS. Use that config file
in complete_artifact() to ensure the test uses the right artifact
directory.

ff666e76

by James Ennis at 2019-01-14T18:46:57Z

completions.py: Add a test for our artifact ref autocompletions

Since the artifact subcommand group has been created, we have been able
to tab complete both element names and artifact refs as arguments to
the artifact subcommands (e.g. bst artifact log), this commit adds a
test for this.

273b0f55

by Tristan Van Berkom at 2019-01-14T20:24:57Z

Merge branch 'jennis/add_artifacts_completion' into 'master'

completions.py: Add a test for our artifact ref autocompletions

See merge request BuildStream/buildstream!1054

2e3c2ea2

by Javier Jardón at 2019-01-14T22:26:04Z

.gitlab-ci.yml: Use latest bst-external

which include flatpak_repo plugin needed to build fdsdk

d60d2e31

by Javier Jardón at 2019-01-15T00:39:22Z

Merge branch 'jjardon/bst_external_0_9' into 'master'

.gitlab-ci.yml: Use latest bst-external

Closes #861

See merge request BuildStream/buildstream!1068

605836c1

by Chandan Singh at 2019-01-15T00:57:05Z

.gitlab-ci.yml: Add job to attempt to update dependencies

This will help us check if BuildStream is working with the latest
version of dependencies, as per our constraints. This job is allowed to
fail but its failure should signal that we need to add stricter
constraints in some of our `.in` requirements files.

006370af

by Chandan Singh at 2019-01-15T01:21:33Z

Merge branch 'chandan/reqs-update-test' into 'master'

.gitlab-ci.yml: Add job to attempt to update dependencies

See merge request BuildStream/buildstream!1038

2028d982
by Benjamin Schubert at 2019-01-15T10:35:06Z
```
Blah
```
7ed23936
by Benjamin Schubert at 2019-01-15T10:35:45Z
```
fixup! Blah
```

7 changed files:

.gitlab-ci.yml
buildstream/_frontend/cli.py
buildstream/_scheduler/queues/buildqueue.py
buildstream/_scheduler/queues/fetchqueue.py
buildstream/_scheduler/queues/pullqueue.py
buildstream/element.py
tests/completions/completions.py

Changes:

.gitlab-ci.yml

@@ -96,7 +96,6 @@ tests-unix:
      # Since the unix platform is required to run as root, no user change required
      - ${TEST_COMMAND}
+-
  tests-fedora-missing-deps:
    # Ensure that tests behave nicely while missing bwrap and ostree
    image: buildstream/testsuite-fedora:28-5da27168-32c47d1c
@@ -114,6 +113,22 @@ tests-fedora-missing-deps:
      - ${TEST_COMMAND}
 +tests-fedora-update-deps:
 +  # Check if the tests pass after updating requirements to their latest
 +  # allowed version.
 +  allow_failure: true
 +  image: buildstream/testsuite-fedora:28-5da27168-32c47d1c
 +  <<: *tests
++
 +  script:
 +    - useradd -Um buildstream
 +    - chown -R buildstream:buildstream .
++
 +    - make --always-make --directory requirements
 +    - cat requirements/*.txt
++
 +    - su buildstream -c "${TEST_COMMAND}"
++
  # Lint separately from testing
  lint:
    stage: test
@@ -146,7 +161,7 @@ docs:
    stage: test
    variables:
      BST_EXT_URL: git+https://gitlab.com/BuildStream/bst-external.git
 -    BST_EXT_REF: 573843768f4d297f85dc3067465b3c7519a8dcc3 # 0.7.0
 +    BST_EXT_REF: 0.9.0-0-g63a19e8068bd777bd9cd59b1a9442f9749ea5a85
      FD_SDK_REF: freedesktop-sdk-18.08.25-0-g250939d465d6dd7768a215f1fa59c4a3412fc337
    before_script:
    - |

buildstream/_frontend/cli.py

@@ -2,6 +2,7 @@ import os
  import sys
  from contextlib import ExitStack
  from fnmatch import fnmatch
 +from functools import partial
  from tempfile import TemporaryDirectory
  import click
@@ -111,14 +112,25 @@ def complete_target(args, incomplete):
      return complete_list
 -def complete_artifact(args, incomplete):
 +def complete_artifact(orig_args, args, incomplete):
      from .._context import Context
      ctx = Context()
      config = None
 -    for i, arg in enumerate(args):
 -        if arg in ('-c', '--config'):
 -            config = args[i + 1]
 +    if orig_args:
 +        for i, arg in enumerate(orig_args):
 +            if arg in ('-c', '--config'):
 +                try:
 +                    config = orig_args[i + 1]
 +                except IndexError:
 +                    pass
 +    if args:
 +        for i, arg in enumerate(args):
 +            if arg in ('-c', '--config'):
 +                try:
 +                    config = args[i + 1]
 +                except IndexError:
 +                    pass
      ctx.load(config)
      # element targets are valid artifact names
@@ -128,8 +140,9 @@ def complete_artifact(args, incomplete):
      return complete_list
 -def override_completions(cmd, cmd_param, args, incomplete):
 +def override_completions(orig_args, cmd, cmd_param, args, incomplete):
      """
 +    :param orig_args: original, non-completion args
      :param cmd_param: command definition
      :param args: full list of args typed before the incomplete arg
      :param incomplete: the incomplete text to autocomplete
@@ -150,7 +163,7 @@ def override_completions(cmd, cmd_param, args, incomplete):
                  cmd_param.opts == ['--track-except']):
              return complete_target(args, incomplete)
          if cmd_param.name == 'artifacts':
 -            return complete_artifact(args, incomplete)
 +            return complete_artifact(orig_args, args, incomplete)
      raise CompleteUnhandled()
@@ -161,7 +174,7 @@ def override_main(self, args=None, prog_name=None, complete_var=None,
      # Hook for the Bash completion.  This only activates if the Bash
      # completion is actually enabled, otherwise this is quite a fast
      # noop.
 -    if main_bashcomplete(self, prog_name, override_completions):
 +    if main_bashcomplete(self, prog_name, partial(override_completions, args)):
          # If we're running tests we cant just go calling exit()
          # from the main process.

buildstream/_scheduler/queues/buildqueue.py

@@ -71,9 +71,6 @@ class BuildQueue(Queue):
          return element._assemble()
      def status(self, element):
 -        # state of dependencies may have changed, recalculate element state
 -        element._update_state()
+-
          if not element._is_required():
              # Artifact is not currently required but it may be requested later.
              # Keep it in the queue.

buildstream/_scheduler/queues/fetchqueue.py

@@ -44,9 +44,6 @@ class FetchQueue(Queue):
          element._fetch()
      def status(self, element):
 -        # state of dependencies may have changed, recalculate element state
 -        element._update_state()
+-
          if not element._is_required():
              # Artifact is not currently required but it may be requested later.
              # Keep it in the queue.

buildstream/_scheduler/queues/pullqueue.py

@@ -39,9 +39,6 @@ class PullQueue(Queue):
              raise SkipJob(self.action_name)
      def status(self, element):
 -        # state of dependencies may have changed, recalculate element state
 -        element._update_state()
+-
          if not element._is_required():
              # Artifact is not currently required but it may be requested later.
              # Keep it in the queue.

buildstream/element.py

@@ -197,6 +197,7 @@ class Element(Plugin):
          self.__runtime_dependencies = []        # Direct runtime dependency Elements
          self.__build_dependencies = []          # Direct build dependency Elements
 +        self.__reverse_build_dependencies = []  # Direct reverse dependency Elements
          self.__sources = []                     # List of Sources
          self.__weak_cache_key = None            # Our cached weak cache key
          self.__strict_cache_key = None          # Our cached cache key for strict builds
@@ -439,6 +440,16 @@ class Element(Plugin):
          if should_yield and (recurse or recursed) and scope != Scope.BUILD:
              yield self
 +    def reverse_build_dependencies(self):
 +        def recurse_rdeps(element):
 +            yield element
++
 +            for rdep in element.__reverse_build_dependencies:
 +                yield from recurse_rdeps(rdep)
++
 +        for rdep in self.__reverse_build_dependencies:
 +            yield from recurse_rdeps(rdep)
++
      def search(self, scope, name):
          """Search for a dependency by name
@@ -930,6 +941,7 @@ class Element(Plugin):
          for meta_dep in meta.build_dependencies:
              dependency = Element._new_from_meta(meta_dep)
              element.__build_dependencies.append(dependency)
 +            dependency.__reverse_build_dependencies.append(element)
          return element
@@ -1306,6 +1318,9 @@ class Element(Plugin):
          self._update_state()
 +        for reverse_dep in self.reverse_build_dependencies():
 +            reverse_dep._update_state()
++
      # _track():
+     #
      # Calls track() on the Element sources
@@ -1503,6 +1518,9 @@ class Element(Plugin):
          self._update_state()
          if self._get_workspace() and self._cached_success():
 +            for rdep in self.reverse_build_dependencies():
 +                rdep._update_state()
++
              assert utils._is_main_process(), \
                  "Attempted to save workspace configuration from child process"
+             #

tests/completions/completions.py

@@ -281,3 +281,44 @@ def test_argument_element_invalid(datafiles, cli, project, cmd, word_idx, expect
  ])
  def test_help_commands(cli, cmd, word_idx, expected):
      assert_completion(cli, cmd, word_idx, expected)
++
++
 +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'project'))
 +def test_argument_artifact(cli, tmpdir, datafiles):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
++
 +    # Build an import element with no dependencies (as there will only be ONE cache key)
 +    result = cli.run(project=project, args=['build', 'import-bin.bst'])  # Has no dependencies
 +    result.assert_success()
++
 +    # Get the key and the artifact ref ($project/$element_name/$key)
 +    key = cli.get_element_key(project, 'import-bin.bst')
 +    artifact = os.path.join('test', 'import-bin', key)
++
 +    # Test autocompletion of the artifact
 +    cmds = [
 +        'bst artifact log ',
 +        'bst artifact log t',
 +        'bst artifact log test/'
 +    ]
++
 +    for i, cmd in enumerate(cmds):
 +        word_idx = 3
 +        result = cli.run(project=project, cwd=project, env={
 +            '_BST_COMPLETION': 'complete',
 +            'COMP_WORDS': cmd,
 +            'COMP_CWORD': str(word_idx)
 +        })
 +        words = []
 +        if result.output:
 +            words = result.output.splitlines()  # This leaves an extra space on each e.g. ['foo.bst ']
 +            words = [word.strip() for word in words]
++
 +            if i == 0:
 +                expected = PROJECT_ELEMENTS + [artifact]  # We should now be able to see the artifact
 +            elif i == 1:
 +                expected = ['target.bst', artifact]
 +            elif i == 2:
 +                expected = [artifact]
++
 +            assert expected == words

[Notes] [Git][BuildStream/buildstream][bschubert/pipeline] 9 commits: _frontend/cli.py: Also check original args for --config in bash completion

Benjamin Schubert pushed to branch bschubert/pipeline at BuildStream / buildstream

Commits:

7 changed files:

Changes: