[Notes] [Git][BuildStream/buildstream][bschubert/pipeline-bak] 18 commit

Benjamin Schubert pushed to branch bschubert/pipeline-bak at BuildStream / buildstream

Commits:

fc3dcec8

by Angelos Evripiotis at 2019-01-11T16:45:00Z

dev-requirements: pytest-cov==2.6.1 for '--no-cov'

Bump the version of pytest-cov, so that we pick up the fix for
the '--no-cov' option breaking in pytest 4.0:

    https://github.com/pytest-dev/pytest-cov/pull/230

This was the error you would get prior to this version:

    _pytest.warning_types.RemovedInPytest4Warning: config.warn has been
    deprecated, use warnings.warn instead

Although it says 'deprecated', it did result in
termination with stack trace.

f86b7ff3

by Angelos Evripiotis at 2019-01-11T16:45:00Z

contributing: '--last-failed' and '--no-cov' tips

To help new folks get to grips with pytest, add some tips on other
frequently used features.

d983f231

by Angelos Evripiotis at 2019-01-11T17:19:06Z

Merge branch 'aevri/tox_no_cov' into 'master'

pytest-cov==2.6.1, contributing: '--last-failed' and '--no-cov' tips

See merge request BuildStream/buildstream!1059

829a2b93

by Tristan Van Berkom at 2019-01-12T21:32:13Z

_scheduler/queues/queue.py: Put elements in the skip list, not jobs

Minor correction, looks like we're not observing this queue
otherwise we'd be seeing crashes.

993e30ae

by Tristan Van Berkom at 2019-01-12T22:02:55Z

Merge branch 'tristan/fix-skipped-elements' into 'master'

_scheduler/queues/queue.py: Put elements in the skip list, not jobs

See merge request BuildStream/buildstream!1063

32732e01

by Javier Jardón at 2019-01-14T09:04:01Z

.gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25

4f5f1184

by Valentin David at 2019-01-14T09:40:56Z

Merge branch 'jjardon/freedesktop-sdk_latest' into 'master'

.gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25

Closes #858

See merge request BuildStream/buildstream!1064

bb80a2b8

by Chandan Singh at 2019-01-14T14:30:15Z

.gitlab-ci.yml: Use newer image for overning aarch tests

During recent reorganizing of `.gitlab-ci.yml` to work with `tox`, seems
like we missed to update the image used by the overnight aarch tests,
meaning that they currently fail due to `tox` being missing from them.

While these tests will be skipped on MRs usually, here is an example of
what the tests look like if they are actually run with this change:
https://gitlab.com/BuildStream/buildstream/-/jobs/145449561.

Fixes #859.

10b3ee62

by Chandan Singh at 2019-01-14T14:57:08Z

Merge branch 'chandan/fix-overnight-aarch' into 'master'

Chandan/fix overnight aarch

Closes #859

See merge request BuildStream/buildstream!1065

542cdaf0

by Jürg Billeter at 2019-01-14T18:46:57Z

_frontend/cli.py: Also check original args for --config in bash completion

The path of the config file generated by testutils for completion tests
is passed as regular argument, not via COMP_WORDS. Use that config file
in complete_artifact() to ensure the test uses the right artifact
directory.

ff666e76

by James Ennis at 2019-01-14T18:46:57Z

completions.py: Add a test for our artifact ref autocompletions

Since the artifact subcommand group has been created, we have been able
to tab complete both element names and artifact refs as arguments to
the artifact subcommands (e.g. bst artifact log), this commit adds a
test for this.

273b0f55

by Tristan Van Berkom at 2019-01-14T20:24:57Z

Merge branch 'jennis/add_artifacts_completion' into 'master'

completions.py: Add a test for our artifact ref autocompletions

See merge request BuildStream/buildstream!1054

2e3c2ea2

by Javier Jardón at 2019-01-14T22:26:04Z

.gitlab-ci.yml: Use latest bst-external

which include flatpak_repo plugin needed to build fdsdk

d60d2e31

by Javier Jardón at 2019-01-15T00:39:22Z

Merge branch 'jjardon/bst_external_0_9' into 'master'

.gitlab-ci.yml: Use latest bst-external

Closes #861

See merge request BuildStream/buildstream!1068

605836c1

by Chandan Singh at 2019-01-15T00:57:05Z

.gitlab-ci.yml: Add job to attempt to update dependencies

This will help us check if BuildStream is working with the latest
version of dependencies, as per our constraints. This job is allowed to
fail but its failure should signal that we need to add stricter
constraints in some of our `.in` requirements files.

006370af

by Chandan Singh at 2019-01-15T01:21:33Z

Merge branch 'chandan/reqs-update-test' into 'master'

.gitlab-ci.yml: Add job to attempt to update dependencies

See merge request BuildStream/buildstream!1038

2d82468a

by Benjamin Schubert at 2019-01-15T11:37:14Z

buildqueue: only update state of direct reverse dependencies

Previously, we would update the state of all elements in the pipeline,
everytime an element in the queue would change state.

This is inefficient. We know that the only impacted elements will
be the direct build reverse dependencies. We can therefore, by recording
the direct build reverse dependencies, only update those and reduce
the time we spend updating states.

df8c4324
by Benjamin Schubert at 2019-01-15T11:37:15Z
```
WIP: notes
```

12 changed files:

.gitlab-ci.yml
CONTRIBUTING.rst
buildstream/_frontend/cli.py
buildstream/_pipeline.py
buildstream/_scheduler/queues/buildqueue.py
buildstream/_scheduler/queues/fetchqueue.py
buildstream/_scheduler/queues/pullqueue.py
buildstream/_scheduler/queues/queue.py
buildstream/_stream.py
buildstream/element.py
requirements/dev-requirements.txt
tests/completions/completions.py

Changes:

.gitlab-ci.yml

@@ -61,7 +61,7 @@ tests-ubuntu-18.04:
    <<: *tests
  overnight-fedora-28-aarch64:
 -  image: buildstream/testsuite-fedora:aarch64-28-06bab030-32a101f6
 +  image: buildstream/testsuite-fedora:aarch64-28-5da27168-32c47d1c
    tags:
      - aarch64
    <<: *tests
@@ -70,6 +70,12 @@ overnight-fedora-28-aarch64:
    except: []
    only:
    - schedules
 +  before_script:
 +    # grpcio needs to be compiled from source on aarch64 so we additionally
 +    # need a C++ compiler here.
 +    # FIXME: Ideally this would be provided by the base image. This will be
 +    # unblocked by https://gitlab.com/BuildStream/buildstream-docker-images/issues/34
 +    - dnf install -y gcc-c++
  tests-unix:
    # Use fedora here, to a) run a test on fedora and b) ensure that we
@@ -90,7 +96,6 @@ tests-unix:
      # Since the unix platform is required to run as root, no user change required
      - ${TEST_COMMAND}
+-
  tests-fedora-missing-deps:
    # Ensure that tests behave nicely while missing bwrap and ostree
    image: buildstream/testsuite-fedora:28-5da27168-32c47d1c
@@ -108,6 +113,22 @@ tests-fedora-missing-deps:
      - ${TEST_COMMAND}
 +tests-fedora-update-deps:
 +  # Check if the tests pass after updating requirements to their latest
 +  # allowed version.
 +  allow_failure: true
 +  image: buildstream/testsuite-fedora:28-5da27168-32c47d1c
 +  <<: *tests
++
 +  script:
 +    - useradd -Um buildstream
 +    - chown -R buildstream:buildstream .
++
 +    - make --always-make --directory requirements
 +    - cat requirements/*.txt
++
 +    - su buildstream -c "${TEST_COMMAND}"
++
  # Lint separately from testing
  lint:
    stage: test
@@ -140,8 +161,8 @@ docs:
    stage: test
    variables:
      BST_EXT_URL: git+https://gitlab.com/BuildStream/bst-external.git
 -    BST_EXT_REF: 573843768f4d297f85dc3067465b3c7519a8dcc3 # 0.7.0
 -    FD_SDK_REF: 612f66e218445eee2b1a9d7dd27c9caba571612e # freedesktop-sdk-18.08.19-54-g612f66e2
 +    BST_EXT_REF: 0.9.0-0-g63a19e8068bd777bd9cd59b1a9442f9749ea5a85
 +    FD_SDK_REF: freedesktop-sdk-18.08.25-0-g250939d465d6dd7768a215f1fa59c4a3412fc337
    before_script:
    - |
      mkdir -p "${HOME}/.config"

CONTRIBUTING.rst

@@ -1534,6 +1534,10 @@ You can always abort on the first failure by running::
    tox -- -x
 +Similarly, you may also be interested in the ``--last-failed`` and
 +``--failed-first`` options as per the
 +`pytest cache <https://docs.pytest.org/en/latest/cache.html>`_ documentation.
++
  If you want to run a specific test or a group of tests, you
  can specify a prefix to match. E.g. if you want to run all of
  the frontend tests you can do::
@@ -1545,6 +1549,12 @@ If you wanted to run the test_build_track test within frontend/buildtrack.py you
    tox -- tests/frontend/buildtrack.py::test_build_track
 +When running only a few tests, you may find the coverage and timing output
 +excessive, there are options to trim them. Note that coverage step will fail.
 +Here is an example::
++
 +  tox -- --no-cov --durations=1 tests/frontend/buildtrack.py::test_build_track
++
  We also have a set of slow integration tests that are disabled by
  default - you will notice most of them marked with SKIP in the pytest
  output. To run them, you can use::

buildstream/_frontend/cli.py

@@ -2,6 +2,7 @@ import os
  import sys
  from contextlib import ExitStack
  from fnmatch import fnmatch
 +from functools import partial
  from tempfile import TemporaryDirectory
  import click
@@ -111,14 +112,25 @@ def complete_target(args, incomplete):
      return complete_list
 -def complete_artifact(args, incomplete):
 +def complete_artifact(orig_args, args, incomplete):
      from .._context import Context
      ctx = Context()
      config = None
 -    for i, arg in enumerate(args):
 -        if arg in ('-c', '--config'):
 -            config = args[i + 1]
 +    if orig_args:
 +        for i, arg in enumerate(orig_args):
 +            if arg in ('-c', '--config'):
 +                try:
 +                    config = orig_args[i + 1]
 +                except IndexError:
 +                    pass
 +    if args:
 +        for i, arg in enumerate(args):
 +            if arg in ('-c', '--config'):
 +                try:
 +                    config = args[i + 1]
 +                except IndexError:
 +                    pass
      ctx.load(config)
      # element targets are valid artifact names
@@ -128,8 +140,9 @@ def complete_artifact(args, incomplete):
      return complete_list
 -def override_completions(cmd, cmd_param, args, incomplete):
 +def override_completions(orig_args, cmd, cmd_param, args, incomplete):
      """
 +    :param orig_args: original, non-completion args
      :param cmd_param: command definition
      :param args: full list of args typed before the incomplete arg
      :param incomplete: the incomplete text to autocomplete
@@ -150,7 +163,7 @@ def override_completions(cmd, cmd_param, args, incomplete):
                  cmd_param.opts == ['--track-except']):
              return complete_target(args, incomplete)
          if cmd_param.name == 'artifacts':
 -            return complete_artifact(args, incomplete)
 +            return complete_artifact(orig_args, args, incomplete)
      raise CompleteUnhandled()
@@ -161,7 +174,7 @@ def override_main(self, args=None, prog_name=None, complete_var=None,
      # Hook for the Bash completion.  This only activates if the Bash
      # completion is actually enabled, otherwise this is quite a fast
      # noop.
 -    if main_bashcomplete(self, prog_name, override_completions):
 +    if main_bashcomplete(self, prog_name, partial(override_completions, args)):
          # If we're running tests we cant just go calling exit()
          # from the main process.

buildstream/_pipeline.py

@@ -136,9 +136,6 @@ class Pipeline():
                  # Preflight
                  element._preflight()
 -                # Determine initial element state.
 -                element._update_state()
+-
      # dependencies()
+     #
      # Generator function to iterate over elements and optionally

buildstream/_scheduler/queues/buildqueue.py

@@ -71,9 +71,6 @@ class BuildQueue(Queue):
          return element._assemble()
      def status(self, element):
 -        # state of dependencies may have changed, recalculate element state
 -        element._update_state()
+-
          if not element._is_required():
              # Artifact is not currently required but it may be requested later.
              # Keep it in the queue.

buildstream/_scheduler/queues/fetchqueue.py

@@ -44,9 +44,6 @@ class FetchQueue(Queue):
          element._fetch()
      def status(self, element):
 -        # state of dependencies may have changed, recalculate element state
 -        element._update_state()
+-
          if not element._is_required():
              # Artifact is not currently required but it may be requested later.
              # Keep it in the queue.
@@ -72,7 +69,7 @@ class FetchQueue(Queue):
          if status == JobStatus.FAIL:
              return
 -        element._update_state()
+-
 +        element._fetch_done()
++
          # Successful fetch, we must be CACHED now
          assert element._get_consistency() == Consistency.CACHED

buildstream/_scheduler/queues/pullqueue.py

@@ -39,9 +39,6 @@ class PullQueue(Queue):
              raise SkipJob(self.action_name)
      def status(self, element):
 -        # state of dependencies may have changed, recalculate element state
 -        element._update_state()
+-
          if not element._is_required():
              # Artifact is not currently required but it may be requested later.
              # Keep it in the queue.

buildstream/_scheduler/queues/queue.py

@@ -170,9 +170,9 @@ class Queue():
          skip = [job for job in jobs if self.status(job.element) == QueueStatus.SKIP]
          wait = [job for job in jobs if job not in skip]
 +        self.skipped_elements.extend([job.element for job in skip])
          self._wait_queue.extend(wait)
          self._done_queue.extend(skip)
 -        self.skipped_elements.extend(skip)
      # dequeue()
+     #

buildstream/_stream.py

@@ -1018,17 +1018,6 @@ class Stream():
          _, status = self._scheduler.run(self.queues)
 -        # Force update element states after a run, such that the summary
 -        # is more coherent
 -        try:
 -            for element in self.total_elements:
 -                element._update_state()
 -        except BstError as e:
 -            self._message(MessageType.ERROR, "Error resolving final state", detail=str(e))
 -            set_last_task_error(e.domain, e.reason)
 -        except Exception as e:   # pylint: disable=broad-except
 -            self._message(MessageType.BUG, "Unhandled exception while resolving final state", detail=str(e))
+-
          if status == SchedStatus.ERROR:
              raise StreamError()
          elif status == SchedStatus.TERMINATED:

buildstream/element.py

@@ -197,6 +197,7 @@ class Element(Plugin):
          self.__runtime_dependencies = []        # Direct runtime dependency Elements
          self.__build_dependencies = []          # Direct build dependency Elements
 +        self.__reverse_build_dependencies = []    # Direct reverse build dependency Elements
          self.__sources = []                     # List of Sources
          self.__weak_cache_key = None            # Our cached weak cache key
          self.__strict_cache_key = None          # Our cached cache key for strict builds
@@ -227,6 +228,8 @@ class Element(Plugin):
          self.__metadata_workspaced = {}               # Boolean of whether it's workspaced
          self.__metadata_workspaced_dependencies = {}  # List of which dependencies are workspaced
 +        self.__is_workspaced = None
++
          # Ensure we have loaded this class's defaults
          self.__init_defaults(plugin_conf)
@@ -370,6 +373,13 @@ class Element(Plugin):
      #############################################################
      #                       Public Methods                      #
      #############################################################
 +    @property
 +    def is_workspaced(self):
 +        if self.__is_workspaced is None:
 +            self.__is_workspaced = self._get_workspace() is not None
++
 +        return self.__is_workspaced
++
      def sources(self):
          """A generator function to enumerate the element sources
@@ -439,6 +449,27 @@ class Element(Plugin):
          if should_yield and (recurse or recursed) and scope != Scope.BUILD:
              yield self
 +    def reverse_build_dependencies(self, recurse=False):
 +        if not recurse:
 +            yield from self.__reverse_build_dependencies
 +            return
++
 +        # visited = set()
++
 +        def recurse_rdeps(element):
 +            # if element in visited:
 +                # return
++
 +            # visited.add(element)
++
 +            yield element
++
 +            for rdep in element.__reverse_build_dependencies:
 +                yield from recurse_rdeps(rdep)
++
 +        for rdep in self.__reverse_build_dependencies:
 +            yield from recurse_rdeps(rdep)
++
      def search(self, scope, name):
          """Search for a dependency by name
@@ -930,6 +961,7 @@ class Element(Plugin):
          for meta_dep in meta.build_dependencies:
              dependency = Element._new_from_meta(meta_dep)
              element.__build_dependencies.append(dependency)
 +            dependency.__reverse_build_dependencies.append(element)
          return element
@@ -1279,6 +1311,9 @@ class Element(Plugin):
          for source in self.sources():
              source._preflight()
 +        # Determine initial element state.
 +        self._update_state()
++
      # _schedule_tracking():
+     #
      # Force an element state to be inconsistent. Any sources appear to be
@@ -1306,6 +1341,9 @@ class Element(Plugin):
          self._update_state()
 +        for rdep in self.reverse_build_dependencies(recurse=True):
 +            rdep._update_state()
++
      # _track():
+     #
      # Calls track() on the Element sources
@@ -1446,6 +1484,7 @@ class Element(Plugin):
      # This unblocks pull/fetch/build.
+     #
      def _set_required(self):
 +        # FIXME: this should enqueue stuff in the queue, it should not be here by default
          if self.__required:
              # Already done
              return
@@ -1456,6 +1495,7 @@ class Element(Plugin):
          for dep in self.dependencies(Scope.RUN, recurse=False):
              dep._set_required()
 +        # FIXME: this should not be done at all here
          self._update_state()
      # _is_required():
@@ -1499,10 +1539,16 @@ class Element(Plugin):
          self.__assemble_scheduled = False
          self.__assemble_done = True
+-
 +        # FIXME: only if workspaced
          self._update_state()
 -        if self._get_workspace() and self._cached_success():
 +        if self.is_workspaced:
++
 +            # Update the state of all reverse dependencies
 +            for reverse_dependency in self.reverse_build_dependencies(recurse=True):
 +                reverse_dependency._update_state()
++
 +        if self.is_workspaced and self._cached_success():
              assert utils._is_main_process(), \
                  "Attempted to save workspace configuration from child process"
+             #
@@ -2035,6 +2081,9 @@ class Element(Plugin):
                  source._fetch(previous_sources)
              previous_sources.append(source)
 +    def _fetch_done(self):
 +        self._update_state()
++
      #############################################################
      #                   Private Local Methods                   #
      #############################################################

requirements/dev-requirements.txt

@@ -2,7 +2,7 @@ coverage==4.4
  pylint==2.2.2
  pycodestyle==2.4.0
  pytest==4.0.2
 -pytest-cov==2.6.0
 +pytest-cov==2.6.1
  pytest-datafiles==2.0
  pytest-env==0.6.2
  pytest-xdist==1.25.0

tests/completions/completions.py

@@ -281,3 +281,44 @@ def test_argument_element_invalid(datafiles, cli, project, cmd, word_idx, expect
  ])
  def test_help_commands(cli, cmd, word_idx, expected):
      assert_completion(cli, cmd, word_idx, expected)
++
++
 +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'project'))
 +def test_argument_artifact(cli, tmpdir, datafiles):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
++
 +    # Build an import element with no dependencies (as there will only be ONE cache key)
 +    result = cli.run(project=project, args=['build', 'import-bin.bst'])  # Has no dependencies
 +    result.assert_success()
++
 +    # Get the key and the artifact ref ($project/$element_name/$key)
 +    key = cli.get_element_key(project, 'import-bin.bst')
 +    artifact = os.path.join('test', 'import-bin', key)
++
 +    # Test autocompletion of the artifact
 +    cmds = [
 +        'bst artifact log ',
 +        'bst artifact log t',
 +        'bst artifact log test/'
 +    ]
++
 +    for i, cmd in enumerate(cmds):
 +        word_idx = 3
 +        result = cli.run(project=project, cwd=project, env={
 +            '_BST_COMPLETION': 'complete',
 +            'COMP_WORDS': cmd,
 +            'COMP_CWORD': str(word_idx)
 +        })
 +        words = []
 +        if result.output:
 +            words = result.output.splitlines()  # This leaves an extra space on each e.g. ['foo.bst ']
 +            words = [word.strip() for word in words]
++
 +            if i == 0:
 +                expected = PROJECT_ELEMENTS + [artifact]  # We should now be able to see the artifact
 +            elif i == 1:
 +                expected = ['target.bst', artifact]
 +            elif i == 2:
 +                expected = [artifact]
++
 +            assert expected == words

[Notes] [Git][BuildStream/buildstream][bschubert/pipeline-bak] 18 commits: dev-requirements: pytest-cov==2.6.1 for '--no-cov'

Benjamin Schubert pushed to branch bschubert/pipeline-bak at BuildStream / buildstream

Commits:

12 changed files:

Changes: