[Notes] [Git][BuildStream/buildstream][tpollard/483] 59 commits: cascache: move tmp directory one level up

[Tom Pollard <gitlab mg gitlab com>]

Title: GitLab

Tom Pollard pushed to branch tpollard/483 at BuildStream / buildstream

Commits:

• c68dcab8
  by Tiago Gomes at 2018-08-09T09:18:43Z

  cascache: move tmp directory one level up
  
  The CAS uses a temp directory while manipulating the cache, temporary
  files can be added and removed while adding artifacts to the cache here.
  
  Since calculation of the cache size happens in parallel to artifact
  cache additions, this causes race conditions in the size calculation
  job, as we end up calling `stat` on temporary files which are being
  removed in parallel.
  
  Handle this by moving the temporary directory out of the way, and avoid
  considering the tmp directory when calculating the cache size
  

• 35ab0335
  by Tiago Gomes at 2018-08-09T10:16:24Z

  Merge branch 'tiagogomes/issue-520' into 'master'
  
  Fix race condition when calculating disk usage
  
  See merge request BuildStream/buildstream!600

• 8aa33e23
  by Valentin David at 2018-08-09T12:49:17Z

  Keep original flags for create in SafeHardlinks.
  
  When open(2) is used with flags O_CREAT|O_RDWR, the file descriptor
  must be readable. Unfortunately O_RDWR was not passed which made
  read fail with EBADF and mmap to signal SIGBUS.
  
  This issue happened with man-db for example.
  
  Fixes #143.
  

• ef7810f3
  by Valentin David at 2018-08-09T14:06:50Z

  Merge branch 'valentindavid/fuse-create-flags' into 'master'
  
  Keep original flags for create in SafeHardlinks.
  
  Closes #143
  
  See merge request BuildStream/buildstream!624

• 2d061173
  by Javier Jardón at 2018-08-09T16:31:15Z

  buildstream/_project.py:
  
  Multiprocessing with n_jobs given by multiprocessing.cpu_count() is not optimal for
  systems where not all CPU can be used (in particular, Docker, CI etc).
  
  As suggested in the multiprocessing docs
  (http://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count)
  using len(os.sched_getaffinity(0)) might be better.
  

• 4e1488ee
  by Javier Jardón at 2018-08-09T16:31:15Z

  buildstream/_project.py: Restrict max-jobs
  
  even if the machine have mores cores available
  
  Patch taken from YBD: https://gitlab.com/baserock/ybd/blob/master/ybd/app.py#L227
  

• de955834
  by Javier Jardón at 2018-08-09T17:22:24Z

  Merge branch 'jjardon/max-jobs' into 'master'
  
  buildstream/_project.py: Restrict max-jobs
  
  See merge request BuildStream/buildstream!620

• 97595585
  by Jim MacArthur at 2018-08-09T17:35:19Z

  setup.py: Pin ruamel.yaml version to <= 0.15
  

• a602365c
  by Tristan Van Berkom at 2018-08-09T20:03:52Z

  Merge branch 'jmac/pin-ruamel-version' into 'master'
  
  Pin ruamel.yaml version to <= 0.15
  
  See merge request BuildStream/buildstream!630

• db0478ab
  by Phillip Smyth at 2018-08-10T10:17:47Z

  Implementing relative workspaces
  
  This fixes #191
  
  A note has been added to NEWS explaining backwards
  compatibility issues
  

• 04cee9a9
  by Phillip Smyth at 2018-08-10T11:12:31Z

  Merge branch 'relative_workspaces' into 'master'
  
  Patch for issue #191 support relative workspaces
  
  Closes #191
  
  See merge request BuildStream/buildstream!504

• 5dcecbad
  by Valentin David at 2018-08-10T12:18:55Z

  Set environment in bwrap command line instead of its environment
  
  Fixes #498
  

• 2e8db54e
  by Valentin David at 2018-08-10T12:53:19Z

  Merge branch 'valentindavid/498_bwrap_environment' into 'master'
  
  Set environment in bwrap command line instead of its environment
  
  Closes #498
  
  See merge request BuildStream/buildstream!565

• 95920f48
  by Chandan Singh at 2018-08-11T15:46:05Z

  Move development reqirements to dev-requirements.txt
  
  In some cases, such as when working inside a virtual environment, it can
  be desirable to install all dependencies for running tests using `pip`.
  This is currently not possible since setuptools does not support
  installing these dependencies in a virtual environment (by design).
  (See https://stackoverflow.com/a/21003259.)
  
  To circumvent this issue, move such requirements to
  `dev-requirements.txt` file that can be used easily with
  `pip install -r`. This also enables tests to be run directly using
  `pytest`, which can be more convenient than `-addopts` approach when one
  needs to add multiple options.
  
  This will also be useful in creating better testuite images, and fix
  some of the issues noticed in
  https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/56.
  

• a9f63c5e
  by Javier Jardón at 2018-08-11T16:53:39Z

  Merge branch 'chandan/dev-requirements' into 'master'
  
  Move development reqirements to dev-requirements.txt
  
  See merge request BuildStream/buildstream!637

• 48c7c0a5
  by Chandan Singh at 2018-08-11T23:41:05Z

  Fix typo in dev-requirements.txt
  

• 88115648
  by Chandan Singh at 2018-08-12T00:38:36Z

  Merge branch 'chandan/fix-dev-reqs-typo' into 'master'
  
  Fix typo in dev-requirements.txt
  
  See merge request BuildStream/buildstream!641

• 0b8beb42
  by Phil Dawson at 2018-08-12T00:40:57Z

  .gitlab-ci-yml: Add ubuntu 18.04 test
  

• 499df6a5
  by Javier Jardón at 2018-08-12T01:37:12Z

  Merge branch 'phil/add-ubuntu-ci-job' into 'master'
  
  .gitlab-ci-yml: Add ubuntu 18.04 test
  
  See merge request BuildStream/buildstream!523

• 60c2873b
  by Javier Jardón at 2018-08-12T01:38:29Z

  .gitlab-ci.yml: Run tests in fedora28 image
  

• 2f27d428
  by Javier Jardón at 2018-08-12T01:38:29Z

  .gitlab-ci.yml: Use the same version of the docker image for all jobs
  

• 1819a288
  by Javier Jardón at 2018-08-12T03:37:04Z

  dev-requirements.txt: Remove obsolete comments
  
  This also fixes an error with pylint in setup.py:
  setup.py:226:19: R1718: Consider using a set comprehension (consider-using-set-comprehension)
  

• 48c715e3
  by Javier Jardón at 2018-08-12T04:42:45Z

  Merge branch 'jjardon/ci_fedora28' into 'master'
  
  Add job to run tests in fedora 28
  
  See merge request BuildStream/buildstream!643

• ccf6e479
  by Valentin David at 2018-08-12T05:54:36Z

  buildstream/plugins/sources/local.py: Make staging deterministic.
  
  Instead of copying metadata on files staged by local, we manually set
  mode to 0755 or 0644 depending on whether user execution was enabled
  on source file.
  
  This makes file modes deterministic independently on the way source
  was distributed.
  
  Non-deterministic mode copying all metadata can still be enabled by
  disable 'deterministic' Boolean configuration on the plugin.
  
  Fixes #527.
  

• 322cab58
  by Valentin David at 2018-08-12T05:54:36Z

  Use deterministic umask when staging sources.
  
  This fix is applied to plugins bzr, git, patch.
  
  Fixes #543 #544 #555.
  

• d08699bd
  by Valentin David at 2018-08-12T05:54:36Z

  buildstream/plugins/sources/zip.py: Fix non-determism in staging.
  
  Staging could end-up with file with different rights depending on the
  umask. The extracted files need to get their access rights fixed.
  

• 70fb7f17
  by Valentin David at 2018-08-12T05:54:36Z

  Add some integration tests for source plugin determinism.
  

• 67df3904
  by Valentin David at 2018-08-12T05:54:36Z

  Bump BST_CORE_ARTIFACT_VERSION for deterministic source plugins
  

• c1fdebb3
  by Tristan Van Berkom at 2018-08-12T06:59:46Z

  Merge branch 'valentindavid/deterministic-source' into 'master'
  
  Deterministic staging
  
  Closes #543, #544, #555, and #527
  
  See merge request BuildStream/buildstream!616

• 3e5aa8ff
  by Chandan Singh at 2018-08-12T10:57:15Z

  doc: Fix install instructions for Fedora
  
  `python3-arpy` package is not available on Fedora so it has to be
  installed using `pip`. Update instructions accordingly.
  
  For reference, here is a pipeline that failed when trying to install
  `arpy` using current instructions:
  https://gitlab.com/BuildStream/buildstream-docker-images/-/jobs/88553245.
  

• 7a13c834
  by Javier Jardón at 2018-08-12T20:11:12Z

  Merge branch 'chandan/update-fedora-install-docs' into 'master'
  
  doc: Fix install instructions for Fedora
  
  See merge request BuildStream/buildstream!640

• c2cc62e7
  by Chandan Singh at 2018-08-13T00:17:21Z

  remote source: Add cachekey test
  
  Add cachekey tests for the recently added `remote` source plugin to
  ensure that future changes do not break API compatibility.
  

• a791e09c
  by Tristan Van Berkom at 2018-08-13T07:43:52Z

  Merge branch 'chandan/remote-source-cachekey-test' into 'master'
  
  remote source: Add cachekey test
  
  See merge request BuildStream/buildstream!636

• 353a6cc2
  by Valentin David at 2018-08-13T09:29:40Z

  Fix tracking of junctions used in project.conf.
  
  Stream._load() now returns early without resolving build pipeline when
  only tracking. Resolving track pipelines does not require to fully
  load project configurations when when elements to track are only
  junctions.
  
  However build pipelines require to fully load project
  configurations. This might not be possible in the case a project
  configuration includes a file from a junction that yet needs to be
  tracked.
  
  Fixes #565.
  

• 5a667107
  by Tristan Van Berkom at 2018-08-13T10:58:12Z

  Merge branch 'valentindavid/fix_included_junction_track' into 'master'
  
  Fix tracking of junctions used in project.conf
  
  Closes #565
  
  See merge request BuildStream/buildstream!619

• 6e759d8a
  by Valentin David at 2018-08-13T12:39:23Z

  Set version of ruaml.yaml to strictly less than 0.15.52.
  
  This allows to use version 0.15.51 rather than 0.15.0 which is
  required for Python 3.
  
  Fixes #571.
  

• 56f2b2e4
  by Valentin David at 2018-08-13T13:36:45Z

  Merge branch 'valentindavid/ruamel-version' into 'master'
  
  Set version of ruaml.yaml to at least 0.15.41 but strictly less than 0.15.52.
  
  Closes #571
  
  See merge request BuildStream/buildstream!646

• 4f6881d6
  by Jürg Billeter at 2018-08-14T08:14:32Z

  NEWS: Move recent entries to a new 1.3.1 section
  
  Caching failed builds and relative workspaces were not part of the 1.1.5
  developer snapshot.
  

• 0c8d0b0b
  by Jonathan Maw at 2018-08-14T08:54:31Z

  loader: Add 'build-depends' and 'runtime-depends' fields to elements
  

• 95798475
  by Jonathan Maw at 2018-08-14T08:54:31Z

  tests: Add tests for loading builddeps and runtime deps
  

• 476fcfa6
  by Jonathan Maw at 2018-08-14T08:54:31Z

  versions: Bump format version to 14
  
  Format version raised because of a change to the core element format.
  

• c51bbff3
  by Jonathan Maw at 2018-08-14T08:54:31Z

  docs: Add documentation of build-depends and runtime-depends fields
  

• 45117239
  by Jonathan Maw at 2018-08-14T08:58:03Z

  Add NEWS
  

• 9d074d7f
  by Jonathan Maw at 2018-08-14T11:52:59Z

  Merge branch '463-make-dependency-type-default-to-build-2' into 'master'
  
  Resolve "Make dependency type default to build"
  
  Closes #463
  
  See merge request BuildStream/buildstream!633

• 69c5ccde
  by Valentin David at 2018-08-14T13:31:03Z

  buildstream/plugins/sources/git.py: Fix clone side effect in getting fetchers
  
  We now delay refreshing submodules until we have a clone, that is after
  we have fetched the main fetcher.
  
  Fixes #537
  

• 820af29d
  by Valentin David at 2018-08-14T13:31:03Z

  tests/frontend/mirror.py: Re-enable a test for git
  

• ca349fb4
  by Jonathan Maw at 2018-08-14T13:31:03Z

  tests: Test that fetching passes when upstream is absent
  

• cb468469
  by Jonathan Maw at 2018-08-14T13:31:03Z

  tests: Add a test of the git source with submodules
  

• dab5bb14
  by Valentin David at 2018-08-14T13:31:03Z

  Test git mirroring fallback on submodules when main repo is not mirrored.
  

• 1a3c4814
  by Valentin David at 2018-08-14T13:31:03Z

  Test we can discovered submodules on fallback mirrored git repositories
  

• c52f6f9a
  by Tristan Van Berkom at 2018-08-15T09:28:35Z

  Merge branch 'valentindavid/fallback_mirror_git' into 'master'
  
  Delay refreshing git submodule until we have a clone
  
  Closes #537
  
  See merge request BuildStream/buildstream!656

• e6457725
  by Josh Smith at 2018-08-15T10:17:29Z

  examples: Switch ftp.gnu.org to mirror
  
  This should avoid current issues accessing the necessary sources which
  are blocking CI currently.
  

• 9f675c64
  by Qinusty at 2018-08-15T11:15:08Z

  Merge branch 'Qinusty/gnu-mirror' into 'master'
  
  Fix CI - ftp.gnu.org unreachable
  
  See merge request BuildStream/buildstream!665

• f3d58233
  by Josh Smith at 2018-08-15T11:15:58Z

  _project.py: Add fatal-warnings configuration item
  
  This allows for users to configure fatal-warnings to be either a list of
  warnings.
  
  This commit deprecates the use of fail-on-overlap within project.conf,
  this will now use the fatal-warnings configuration item.
  
  element.py: Cache key calculation now takes into account all of the
  fatal-warnings
  
  tests: This modifys the tests/frontend/overlaps.py tests to support the
  new fatal-warnings configuration. Backwards compatibility is also
  tested for `fail-on-overlap`
  
  _versions.py: BST_FORMAT_VERSION bumped to 15 for fatal-warnings
  	      BST_CORE_ARTIFACT_VERSION bumpted to 5 for fatal-warnings
  
  Fixes: #526
  

• 4669309e
  by Josh Smith at 2018-08-15T11:15:58Z

  git.py: Add configurable warning for inconsistent-submodule
  
  This follows the implementation of configurable warnings.
  

• c9dd51f2
  by Josh Smith at 2018-08-15T11:15:58Z

  docs: Add documentation for Configurable Warnings
  
  This includes detailing the use of `fatal-warnings` within project.conf
  

• 03527d5a
  by Josh Smith at 2018-08-15T11:15:58Z

  tests: Add tests for configurable warnings
  
  This adds multiple tests for custom plugin warnings and core warnings,
  providing checks for both cases which should cause warnings and errors
  when configured as fatal.
  
  Also adds tests for cache key calculations.
  

• 76f34a63
  by Tristan Van Berkom at 2018-08-15T12:16:04Z

  Merge branch 'Qinusty/526-fail-on-warnings' into 'master'
  
  Configurable Warnings
  
  Closes #526
  
  See merge request BuildStream/buildstream!627

• 99b58900
  by Tom Pollard at 2018-08-15T14:58:26Z

  plugins/git.py: Warn if ref is not in given track
  
  Add a helper function assert_ref_in_track to git.py GitMirror()
  which is used when staging & initing the source workspace. It
  checks the element's ref exists in the track (branch/tag) if it
  has been specified, raising a warning if necessary. The warning makes
  use of the warning token 'REF_NOT_IN_TRACK' from the configurable
  CoreWarnings. If the element has been tracked with bst, it is assumed
  that the value of ref exists in the track as it was generated from it
  & as such is not asserted.
  

30 changed files:

• .gitlab-ci.yml
• MANIFEST.in
• NEWS
• buildstream/_artifactcache/artifactcache.py
• buildstream/_artifactcache/cascache.py
• buildstream/_artifactcache/casserver.py
• buildstream/_frontend/widget.py
• buildstream/_fuse/fuse.py
• buildstream/_fuse/hardlinks.py
• buildstream/_loader/loadelement.py
• buildstream/_loader/types.py
• buildstream/_project.py
• buildstream/_stream.py
• buildstream/_versions.py
• buildstream/_workspaces.py
• buildstream/data/projectconfig.yaml
• buildstream/element.py
• buildstream/plugin.py
• buildstream/plugins/sources/git.py
• buildstream/plugins/sources/local.py
• buildstream/plugins/sources/zip.py
• buildstream/sandbox/_sandboxbwrap.py
• buildstream/utils.py
• + dev-requirements.txt
• doc/examples/autotools/project.conf
• doc/examples/junctions/autotools/project.conf
• doc/source/format_declaring.rst
• doc/source/format_project.rst
• doc/source/install_linux_distro.rst
• setup.py

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.