[Notes] [Git][BuildStream/buildstream][jonathan/workspace-fragment-create] 50 commits: Check if command is a str and replace by list before checking existence

[Jonathan Maw <gitlab mg gitlab com>]

Title: GitLab

Jonathan Maw pushed to branch jonathan/workspace-fragment-create at BuildStream / buildstream

Commits:

• 80762ecb
  by Benjamin Schubert at 2018-10-26T09:45:39Z

  Check if command is a str and replace by list before checking existence
  
  Currently we would make sure the sandbox had a command before
  converting it to a list if it was given as a string. That meant that a string
  command would never exist and the check be invalid.
  
  This also adds the same logic in the dummy sandbox for consistency.
  

• f131c407
  by Benjamin Schubert at 2018-10-26T10:09:51Z

  Merge branch 'bschubert/fix-command-sandbox' into 'master'
  
  Check is command is a str and replace by list before checking existence
  
  Closes #728
  
  See merge request BuildStream/buildstream!898

• 54d89aa1
  by Daniel Silverstone at 2018-10-26T10:20:36Z

  _loader/loadelement.py: Use enumerate() in dependency extraction
  
  In cases where elements end up with large numbers of dependencies,
  the n-squared performance in `_extract_depends_from_node` can produce
  significant delays during project load.  This patch corrects that.
  In addition we were double-retrieving provenance which was unnecessary.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• e4c18ba4
  by Phil Dawson at 2018-10-26T10:43:55Z

  Merge branch 'danielsilverstone-ct/use-enumerate' into 'master'
  
  _loader/loadelement.py: Use enumerate() in dependency extraction
  
  See merge request BuildStream/buildstream!901

• 20e5c58a
  by James Ennis at 2018-10-26T11:14:58Z

  using_configuring_artifact_server.rst: Fix lines that are too long
  

• 320dae02
  by James Ennis at 2018-10-26T11:37:42Z

  docs: Add more information to "managing the cache with systemmd" section
  

• 1eb993bf
  by James Ennis at 2018-10-26T12:02:49Z

  Merge branch 'jennis/another_docs_patch' into 'master'
  
  More information regarding the use of systemd to manage the cache
  
  See merge request BuildStream/buildstream!904

• f069d82f
  by Daniel Silverstone at 2018-10-26T12:19:58Z

  various: Move _sentinel from utils.py to _yaml.py
  
  The `_sentinel` in `utils.py` was used only for yaml node tracking.
  As such, simplify matters by removing it from `utils.py` and move it
  to `_yaml.py` which means that we no longer need to try and avoid a
  circular import issue by means of runtime importing.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 283887a5
  by James Ennis at 2018-10-26T13:12:12Z

  Merge branch 'danielsilverstone-ct/yaml-sentinel-rework' into 'master'
  
  Move _sentinel from utils.py to _yaml.py
  
  See merge request BuildStream/buildstream!903

• 2c6cb230
  by Valentin David at 2018-10-26T14:21:18Z

  Fix some ruamel warnings during tests
  

• a8250ca4
  by Valentin David at 2018-10-26T15:04:25Z

  Merge branch 'valentindavid/ruamel_warnings' into 'master'
  
  Fix some ruamel warnings during tests
  
  See merge request BuildStream/buildstream!889

• 1b308fe3
  by Daniel Silverstone at 2018-10-26T15:50:42Z

  _stream.py: Preserve stdout FD across checkout-to-stdout
  
  In order to support things which might need to use the stdout FD
  after checkout has completed writing a tarball to it, preserve it
  by means of `os.dup()` and `os.dup2()`
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 65d90c31
  by Daniel Silverstone at 2018-10-26T15:50:54Z

  tests/testutils/runcli.py: Support binary-mode capture of stdout
  
  In order to test things which write tarballs to stdout correctly,
  we need to capture the binary output cleanly.  This ensures we're not
  potentially mismatching encodings in and out.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• ed733f3e
  by Daniel Silverstone at 2018-10-26T15:51:40Z

  tests/frontend/buildcheckout.py: Enable test_build_checkout_tarball_stdout
  
  With the new support for saving the stdout FDs cleanly, and binary mode
  capture in runcli, enable the tarball-to-stdout test.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• e97d2a75
  by Benjamin Schubert at 2018-10-29T10:17:07Z

  Merge branch 'bschubert/fix-binary-output-capture' into 'master'
  
  Don't close stdout when running 'checkout' to stdout
  
  Closes #486
  
  See merge request BuildStream/buildstream!905

• 1625de0e
  by Tristan Van Berkom at 2018-10-29T11:49:33Z

  doc/source/conf.py: Added `images` directory for statically included files
  

• 38507b4e
  by Tristan Van Berkom at 2018-10-29T11:49:33Z

  doc/source/format_intro.rst: Adding link anchor for include directives
  

• 26164bcd
  by Tristan Van Berkom at 2018-10-29T13:57:51Z

  MANIFEST.in: Include SVG and ODG files in source distributions
  

• 3a23c8f6
  by Tristan Van Berkom at 2018-10-29T13:57:51Z

  doc: Adding new architecture document with initial "Overview of modules" section
  

• 41d370f6
  by Tristan Van Berkom at 2018-10-29T13:57:51Z

  doc: Adding new architecture document describing the data model
  

• bed7f7ac
  by Tristan Van Berkom at 2018-10-29T13:57:51Z

  doc: Adding new architecture document describing the dependency model
  

• 9bdf7f40
  by Tristan Van Berkom at 2018-10-29T13:57:51Z

  doc: Adding new architecture document about highlevel program flow
  

• 6b8bdf35
  by Tristan Van Berkom at 2018-10-29T13:57:51Z

  doc: Adding new architecture document about how the scheduler works.
  

• b1fadaa5
  by Tristan Van Berkom at 2018-10-29T13:57:51Z

  doc: Moved cachekey and sandboxing writings into the architecture section
  

• baf6b578
  by Tristan Van Berkom at 2018-10-29T14:28:41Z

  Merge branch 'tristan/architecture-docs' into 'master'
  
  Tristan/architecture docs
  
  See merge request BuildStream/buildstream!907

• 09ef8b25
  by Jonathan Maw at 2018-10-30T15:42:05Z

  element.py: Always clean up the rootdir
  
  We shouldn't need it to persist now that we cache failed build dirs.
  
  This change breaks the test
  `tests/integration/shell.py::test_sysroot_workspace_visible`.
  I can no longer see a use-case for this test.
  AIUI, it tested that the failed build sysroot stored in the builddir has
  the workspace's files in, despite the workspace being unmounted.
  I believe this behaviour is made redundant by cached buildtrees.
  
  This fixes part of #539
  

• 37afde0a
  by Jonathan Maw at 2018-10-30T15:42:05Z

  Element: Use cached buildtree in build shells and failure shells
  
  This includes changes in app.py:
  * Interactive failure shell no longer uses the failed build sysroot,
    defaulting to the cached build tree.
  
  Changes in element.py are:
  * Errors caused by building don't store the failed build sysroot,
    instead storing that a sandbox can be created to debug the error.
  * When staging sources, will stage the element's cached build tree if it
    exists.
  
  Changes in _exceptions.py:
  * BstError.sandbox is now a flag of whether a sandbox can be opened up
    to debug the error.
  
  Changes in widget.py:
  * Don't try to print any information about the sandbox.
  
  Changes in _message.py:
  * Fix documentation so Message.sandbox is not a directory any more.
  
  This is part of #539
  

• 4bc71773
  by Jonathan Maw at 2018-10-30T15:42:05Z

  NEWS: Add item for cached buildtree behaviour
  

• 9b89564f
  by Jonathan Maw at 2018-10-30T15:42:05Z

  tests: Add test that cached build trees are staged in build shells
  
  This is related to #539
  

• ec76cbe1
  by Jonathan Maw at 2018-10-30T15:42:05Z

  sandbox.py: Remove redundant Sandbox.__directory
  

• 825963b5
  by Jonathan Maw at 2018-10-30T15:42:05Z

  element: Make "--sysroot" take a bare directory
  
  i.e. instead of taking a directory that must contain "root" and
  "scratch", and treating "root" as the root, use the directory directly.
  
  In element.py:
  * __sandbox takes the `bare_sandbox` arg, to pass into the sandbox's constructor
  
  In sandbox.py:
  * If bare_sandbox, `_root` is the passed-in directory, and `__scratch`
    is None.
  * Trying to use `__scratch` when bare_sandbox is True is a bug.
  
  In _mount.py:
  * Don't get the value of `__scratch` if it's not needed.
  
  This is part of #539
  

• cf721838
  by Jonathan Maw at 2018-10-30T15:42:05Z

  NEWS: Add item for bst shell --sysroot changes
  

• 79d1bb7f
  by Jonathan Maw at 2018-10-30T15:42:05Z

  tests: Add tests for 'bst shell --sysroot'
  
  This is related to #539
  

• c7ac7e7d
  by Jonathan Maw at 2018-10-30T16:19:17Z

  Merge branch 'jonathan/debug-remote-failed-builds' into 'master'
  
  Jonathan/debug remote failed builds
  
  See merge request BuildStream/buildstream!869

• 516e990e
  by ctolentino8 at 2018-10-31T11:36:46Z

  _frontend/app.py: Set correct element-path in interactive bst-init
  

• b8a37a63
  by Tristan Van Berkom at 2018-11-01T10:16:25Z

  Merge branch 'chiaratolentino/bst-init-interactive-elementpath' into 'master'
  
  _frontend/app.py: Set correct element-path in interactive bst-init
  
  See merge request BuildStream/buildstream!912

• b27b592a
  by Benjamin Schubert at 2018-11-01T10:49:57Z

  Remove dependency on pytest-runner
  
  This includes a new command mimicking pytest-runner so that we
  can drop this dependency
  
  This was the only setup_requires dependency that we had and
  will make like easier for people behind proxies
  

• 89ace5d7
  by Benjamin Schubert at 2018-11-01T11:16:36Z

  Merge branch 'bschubert/remove-pytest-runner' into 'master'
  
  Remove dependency on pytest-runner
  
  See merge request BuildStream/buildstream!886

• 4cfabce8
  by Angelos Evripiotis at 2018-11-01T11:35:02Z

  BREAK: manual.yaml: don't set any default env vars
  
  Remove the 'MAKEFLAGS' and 'V' defaults.
  
  Now that there is a specialised 'make' element, these make-specific
  defaults don't make sense here. This element is meant to be a blank
  slate for folks to build on.
  
  Note that this is a breaking change, that will require some users to
  make changes to their .bst files if they are expecting these environment
  variable to be set.
  
  _versions.py: BST_FORMAT_VERSION bumped to 18.
  
  This fixes issue #718
  

• 48860aac
  by Tristan Van Berkom at 2018-11-01T12:01:04Z

  Merge branch 'tristan/empty-manual-element' into 'master'
  
  BREAK: manual.yaml: don't set any default env vars
  
  Closes #718
  
  See merge request BuildStream/buildstream!916

• d868b409
  by Daniel Silverstone at 2018-11-01T13:40:24Z

  _yaml.py: Implement `get()` for `ChainMap`
  
  Since the core Python `ChainMap.get()` implements with:
  
      self[key] if key in self else default
  
  The double-chain-lookup is expensive.  This simple change solves that
  for our ChainMap derived structure.  As such it improves matters
  for #466 somewhat.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 7f79b9ce
  by Tristan Van Berkom at 2018-11-01T14:25:57Z

  Merge branch 'danielsilverstone-ct/improve-chainmap-get' into 'master'
  
  _yaml.py: Implement `get()` for `ChainMap`
  
  See merge request BuildStream/buildstream!919

• 002acc16
  by Jonathan Maw at 2018-11-01T16:49:00Z

  _workspaces.py: Add a WorkspaceLocal object
  
  WorkspaceLocal contains all the information needed to generate data
  that's stored within the workspace. Currently this is just a list of
  project directories.
  

• dce148d1
  by Jonathan Maw at 2018-11-01T16:50:52Z

  _stream.py: Opened workspaces store the project that opened them
  

• ea54d840
  by Jonathan Maw at 2018-11-01T16:50:53Z

  _stream.py: workspace_close deletes the stored project reference
  

• 8338e5b5
  by Jonathan Maw at 2018-11-01T16:50:53Z

  tests: Workspace tests check existence of .bstproject.yaml
  

• 52e95a1c
  by Jonathan Maw at 2018-11-01T16:50:53Z

  _project.py: Rename _ensure_project_dir to _find_project_dir
  

• 0469a37a
  by Jonathan Maw at 2018-11-01T16:50:53Z

  utils.py: Add a helper for searching upwards for files
  
  i.e. with a given directory and filename, check parent directories until
  either a directory with the filename is found, or you reach the root of
  the filesystem.
  

• 93912082
  by Jonathan Maw at 2018-11-01T16:50:53Z

  cli.py: Use utils' search upwards helper when searching for project.conf
  

• 627e0894
  by Jonathan Maw at 2018-11-01T16:50:53Z

  _project.py: Use utils' search upwards helper when searching for project.conf
  

30 changed files:

• MANIFEST.in
• NEWS
• buildstream/_exceptions.py
• buildstream/_frontend/app.py
• buildstream/_frontend/cli.py
• buildstream/_frontend/widget.py
• buildstream/_loader/loadelement.py
• buildstream/_message.py
• buildstream/_project.py
• buildstream/_stream.py
• buildstream/_versions.py
• buildstream/_workspaces.py
• buildstream/_yaml.py
• buildstream/element.py
• buildstream/plugin.py
• buildstream/plugins/elements/manual.yaml
• buildstream/sandbox/_mount.py
• buildstream/sandbox/_sandboxbwrap.py
• buildstream/sandbox/_sandboxchroot.py
• buildstream/sandbox/_sandboxdummy.py
• buildstream/sandbox/sandbox.py
• buildstream/utils.py
• doc/source/additional_cachekeys.rst → doc/source/arch_cachekeys.rst
• + doc/source/arch_data_model.rst
• + doc/source/arch_dependency_model.rst
• + doc/source/arch_overview.rst
• + doc/source/arch_program_flow.rst
• doc/source/additional_sandboxing.rst → doc/source/arch_sandboxing.rst
• + doc/source/arch_scheduler.rst
• doc/source/conf.py

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.