[Notes] [Git][BuildStream/buildstream][bschubert/add-fedora-nobwrap-tests] 42 commits: _project.py: Validate nodes early in Project._load

[Benjamin Schubert <gitlab mg gitlab com>]

Title: GitLab

Benjamin Schubert pushed to branch bschubert/add-fedora-nobwrap-tests at BuildStream / buildstream

Commits:

• ae5ccd76
  by Phillip Smyth at 2018-11-12T14:57:51Z

  _project.py: Validate nodes early in Project._load
  
  _project.py: Added validate_nodes() helper function to prevent duplicate lists
  
  element-path was being used before node validation resulting in uncaught errors
  

• 56a54161
  by Phillip Smyth at 2018-11-12T14:57:51Z

  tests/frontend: Add test for invalid element-path
  

• e55a9703
  by Jürg Billeter at 2018-11-12T15:44:45Z

  Merge branch 'element-path_not_validated' into 'master'
  
  Element path not validated before use
  
  See merge request BuildStream/buildstream!937

• c87d6e45
  by Valentin David at 2018-11-14T11:43:33Z

  Run tests on aarch64
  
  Fixes #755
  

• 16b01489
  by Valentin David at 2018-11-14T11:43:33Z

  Disable cachekey tests on other architectures than x86_64
  

• 85046b29
  by Valentin David at 2018-11-14T11:43:33Z

  Disable tests on example on other architectures than x86_64
  

• dd5a073b
  by Valentin David at 2018-11-14T11:43:33Z

  Add support for aarch64 in a test
  

• c5d72ae7
  by Valentin David at 2018-11-14T11:52:53Z

  Test aarch64 only overnight
  

• 264a57f6
  by Javier Jardón at 2018-11-14T12:22:03Z

  Merge branch 'valentindavid/integration-tests-aarch64' into 'master'
  
  Run tests on aarch64
  
  Closes #369 and #755
  
  See merge request BuildStream/buildstream!948

• da735e56
  by Richard Maw at 2018-11-14T13:30:34Z

  _platform/linux.py: Move get_bwrap_version into _site.py
  
  It's inconvenient to have to create a Linux platform to parse the bwrap version
  and we want to get the version in a consistent manner.
  

• e7633500
  by Richard Maw at 2018-11-14T13:30:34Z

  buildstream/sandbox/_sandboxbwrap.py: Distinguish sandbox failure from command failure
  
  If `bwrap` fails to set up the sandbox and start the payload command
  it won't write an exit-code in --json-status-fd,
  so we can report if it was a sandboxing failure if we don't get exit-code status
  and a payload command failure if we do and it's non-zero.
  
  Closes https://gitlab.com/BuildStream/buildstream/issues/286
  

• e9e08823
  by Richard Maw at 2018-11-14T13:30:34Z

  tests/testutils/site.py: Check for bwrap supporting --json-status-fd
  

• 90ca007e
  by Richard Maw at 2018-11-14T13:30:34Z

  tests/integration/sandbox-bwrap.py: Test distinguishing sandbox exit code from command
  

• 327b19dd
  by richardmaw-codethink at 2018-11-14T13:59:16Z

  Merge branch 'richardmaw/distinguish-sandboxing-build-fail' into 'master'
  
  Distinguish between bubblewrap sandboxing failure and command failure
  
  Closes #286
  
  See merge request BuildStream/buildstream!868

• 082b3811
  by Angelos Evripiotis at 2018-11-15T19:48:12Z

  contributing: add guidance on unit tests
  
  Decrease uncertainty around whether unit tests are welcome in the
  project or not.
  

• b5b79056
  by Jürg Billeter at 2018-11-15T20:24:58Z

  Merge branch 'aevri/unit_tests' into 'master'
  
  contributing: add guidance on unit tests
  
  See merge request BuildStream/buildstream!943

• 199bfff1
  by Tom Pollard at 2018-11-17T13:07:59Z

  Add cli main and user config option for 'pull-buildtrees' context.
  
  _context.py: Add pull_buildtrees global user context, the default
  of which is set to False via the addition of pull-buildtrees to
  userconfig.yaml cache group.
  
  _frontend/app.py & cli.py: Add --pull-buildtrees as a bst main
  option, which when passed will override the default or user defined
  context for pull_buildtrees.
  
  tests/completions/completions.py: Update for the added flag.
  

• 454dcaf1
  by Tom Pollard at 2018-11-17T13:07:59Z

  Don't pull artifact build trees by default.
  
  The addition of cached build trees being included in element
  artifacts has led to mostly redundant download overheads when
  pulling from a remote artifact server. As such the default behaviour
  of pull shouldn't fetch the build tree object if available.
  
  element.py: extend relevant pull logic for specified subdir
  consideration and ensure push logic does not lead to partial
  artifact pushes. Change assumption that the buildtree can be
  extracted if artifact is cached. __cached_buildtree() and
  __pull_directories() helpers added.
  
  _artifactcache/: artifactcache.py & cascache.py inclusion of
  helper functions for subdir artifact checking & fetching, fetch
  logic extended to only pull required artifact directories.
  extract & checkout updated to handle full/partial operation.
  
  tests/: addition of integration test pullbuildtrees.py,
  buildtrees.py adapted cli options, testutils/artifactshare.py
  has_artifact changed to return artifact digest on true condition.
  

• 0d309478
  by Jürg Billeter at 2018-11-17T13:49:58Z

  Merge branch 'tpollard/494' into 'master'
  
  Don't pull artifact buildtrees by default
  
  See merge request BuildStream/buildstream!786

• 27932739
  by Benjamin Schubert at 2018-11-19T09:46:39Z

  plugins/sources/pip.py: also look for python version named "python"
  
  In some virtual environments, we might not have the binary with the
  exact python version, which would lead the current pip plugin to
  fail to find a working pip version.
  
  This adds "python" at the start of the list of valid python versions
  

• ea2de561
  by Benjamin Schubert at 2018-11-19T10:19:24Z

  Merge branch 'bschubert/fix-pip-python' into 'master'
  
  plugins/sources/pip.py: also look for python version named "python"
  
  Closes #758
  
  See merge request BuildStream/buildstream!942

• f23b6031
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  tests/frontend/buildtrack.py: check for success after invocation
  
  Previously the result of the first call in test_build_track was not
  checked to be successful, which would lead to potential erros later on
  that were hard to debug.
  
  Let's check it everytime
  

• a6defc0b
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  Fix os.rename in git source element to correctly handle error codes
  
  According to the documentation
  (https://www.unix.com/man-page/POSIX/3posix/rename/), when the directory
  already is there, either EEXIST or ENOTEMPTY could be thrown.
  
  Previously only ENOTEMPTY was checked.
  
  Done:
    - Separated the move into its own function
    - Check for both errors
    - Create unit tests for it, covering most test cases
  

• 88089d2d
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  Extract atomic move function to utils.py
  
  Moving atomically a file/directory can be tricky since different
  errors might be raised for the same underlying problem.
  
  Having a utility function to reduce this discrepancies will help
  in ensuring we have correct behavior
  

• fd9e46be
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  cascache.py: use move_atomic instead of manual error checking
  

• d32e0b83
  by Benjamin Schubert at 2018-11-19T11:39:51Z

  sources/pip.py: Use move_atomic instead of manual os.rename
  
  This uses move_atomic insteand of the manual os.rename and manual error
  checking and throws a SourceError for consistency with other modules.
  

• 6f837118
  by Benjamin Schubert at 2018-11-19T12:22:40Z

  Merge branch 'bschubert/fix-atomic-move-git-repo' into 'master'
  
  Fix os.rename in git source element to correctly handle error codes
  
  See merge request BuildStream/buildstream!938

• 30b72244
  by Chandan Singh at 2018-11-19T12:35:01Z

  buildstream/_pipeline.py: Simplify error message for assert_consistent()
  
  Currently, `Pipeline.assert_consistent()` prints an error message like
  the one below when certain elements have inconsistent sources:
  
  ```
  ...
        Element: hello.bst is inconsistent
          Source tar source at hello.bst [line 16 column 2] is missing ref
  ...
  ```
  
  Drop the word "Source" from the beginning of the message as
  "Source <source-kind> source ..." is kind of awkward to read.
  

• 76c5d2f8
  by Chandan Singh at 2018-11-19T12:35:01Z

  Add `bst source-checkout` command
  
  As discussed in
  https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
  add `bst source-checkout` command. This will allow users to checkout
  sources for a given target.
  
  * _frontend/cli.py: Add source-checkout command
  * _pipeline.py: Add assert_sources_cached() method
  * _stream.py: Add source_checkout method, abstract out
    __check_location_writable() method that used to part of checkout()
  

• 9f629638
  by Chandan Singh at 2018-11-19T13:03:36Z

  Merge branch 'chandan/source-checkout' into 'master'
  
  Add `bst source-checkout` command
  
  See merge request BuildStream/buildstream!820

• bc827cc8
  by Abderrahim Kitouni at 2018-11-19T14:55:15Z

  plugins/elements/cmake.yaml: always specify variable types
  
  cmake sometimes misinterprets relative paths as relative to the current directory
  if this is not specified. See freedesktop-sdk/freedesktop-sdk#431
  
  adjust tests/format/variables.py accordingly.
  

• 5fbc5f41
  by Valentin David at 2018-11-19T15:31:05Z

  Merge branch 'abderrahim/cmake-variable-types' into 'master'
  
  plugins/elements/cmake.yaml: always specify variable types
  
  Closes #761
  
  See merge request BuildStream/buildstream!947

• 06e28860
  by Benjamin Schubert at 2018-11-19T15:52:24Z

  Don't cache sandbox errors
  
  Sandbox errors (like missing host tools) are dependent on the host
  system and rarely on what is actually done.
  
  It is therefore better to not cache them as they are subject to
  change between two runs.
  
  Also add test to ensure sandbox failure are not cached
  

• dd36cfbc
  by Benjamin Schubert at 2018-11-19T15:52:24Z

  element.py: Only cache on element error or success
  
  Reverse the way we were handling caching by only caching
  when we know it's a problem with the build or the build is a
  success in order not to cache transient errors.
  

• eac7274d
  by Benjamin Schubert at 2018-11-19T16:29:07Z

  Merge branch 'bschubert/dont-cache-errors-from-host-tools' into 'master'
  
  Don't cache sandbox failures
  
  Closes #727
  
  See merge request BuildStream/buildstream!895

• 795ae0fd
  by Benjamin Schubert at 2018-11-19T16:30:34Z

  gitlab-ci.yml: Use CI_PROJECT_DIR instead of $(pwd) for cache
  

• 62e69402
  by Benjamin Schubert at 2018-11-19T16:30:34Z

  Export variables globally for less repetition.
  
  This doesn't change the test behavior and enable for better
  composability of tests
  

• 0f0ce630
  by Benjamin Schubert at 2018-11-19T16:39:53Z

  gitlab-cy.yml: make linux-tests-template more composable.
  
  This will allow for further de-duplication of work.
  

• f01564e7
  by Benjamin Schubert at 2018-11-19T16:39:53Z

  gitlab-ci.yml: remove unused artifacts for unix tests
  
  The unix-logs is always empty, there is therefore no reasons of keeping
  it there
  

• 11a5dcd5
  by Benjamin Schubert at 2018-11-19T16:42:03Z

  gitlab-ci.yml: use template for unix tests
  
  This makes all tests use the same template, which makes reasoning
  about them simpler
  

• 9abbcc28
  by Benjamin Schubert at 2018-11-19T17:46:53Z

  gitlab-ci.yml: Add linux without brwap/ostree to ensure tests behave
  
  This allows us to make sure that tests behave nicely while missing brwap
  and ostree.
  

• 07bc2b82
  by Benjamin Schubert at 2018-11-19T17:46:54Z

  gitlab-ci.yml: Extract test command as a variable
  
  This removes the need of having it synchronized in multiple places
  

30 changed files:

• .gitlab-ci.yml
• CONTRIBUTING.rst
• NEWS
• buildstream/_artifactcache/artifactcache.py
• buildstream/_artifactcache/cascache.py
• buildstream/_context.py
• buildstream/_frontend/app.py
• buildstream/_frontend/cli.py
• buildstream/_pipeline.py
• buildstream/_platform/linux.py
• buildstream/_project.py
• buildstream/_site.py
• buildstream/_stream.py
• buildstream/data/userconfig.yaml
• buildstream/element.py
• buildstream/plugins/elements/cmake.yaml
• buildstream/plugins/sources/git.py
• buildstream/plugins/sources/pip.py
• buildstream/sandbox/_sandboxbwrap.py
• buildstream/utils.py
• conftest.py
• tests/cachekey/cachekey.py
• tests/completions/completions.py
• tests/examples/autotools.py
• tests/examples/developing.py
• tests/examples/flatpak-autotools.py
• tests/examples/integration-commands.py
• tests/examples/junctions.py
• tests/examples/running-commands.py
• tests/format/list-directive-type-error/project.conf

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.