[Notes] [Git][BuildStream/buildstream][richardmaw/shell-multi-stage] 106

richardmaw-codethink pushed to branch richardmaw/shell-multi-stage at BuildStream / buildstream

Commits:

e28f668a

by Valentin David at 2018-11-08T09:34:11Z

Fix infinite recursion in default strip debug command

Fixes #645.

83d15350

by Valentin David at 2018-11-08T10:01:39Z

Merge branch 'valentindavid/fix-recursive-strip-debug' into 'master'

Fix infinite recursion in default strip debug command

Closes #645

See merge request BuildStream/buildstream!918

9f0e12f1

by Benjamin Schubert at 2018-11-08T10:21:12Z

Move bwrap checks in platform/linux.py

Remove the bwraps checks from _site.py and put them in platform.linux
which is the only place where they are run.

This allows the removal of a double level of caching, making
reasoning about tests easier

cf2e0059

by Benjamin Schubert at 2018-11-08T10:21:12Z

conftest.py: Ensure platform is not maintained between tests

This removes the `_instance` on the platform object that we use
for caching and not recreating the object everytime at the start
of every test.

This is to ensure our tests share the least amount of state.

The performance penalty is from 5 to 10% accross the whole test suite.
The readings were done 5 times for each before and after the change
and on the same computer.

c51ba01b

by Benjamin Schubert at 2018-11-08T10:21:12Z

Test that helpful messages are raised when missing dependencies

This adds a `reason` to the SandboxEror thrown in sandboxdummy
to be able to understand where the error comes from

f116b9b7

by Benjamin Schubert at 2018-11-08T11:07:12Z

Merge branch 'bschubert/tests-no-chroot-linux' into 'master'

test behavior when there is no sandbox on Linux

Closes #696 and #736

See merge request BuildStream/buildstream!921

cc2e6ae5

by Valentin David at 2018-11-08T12:41:36Z

Fix bug with root mounted as non-artifact in script plugin.

The issue was introduced by 6ccfab0b1b25990e406446d5cbe5aee83a5e158a.

e578a89f

by Valentin David at 2018-11-08T13:11:10Z

Merge branch 'valentindavid/fix-script-install-root' into 'master'

Fix bug with root mounted as non-artifact in script plugin.

Closes #756 and #749

See merge request BuildStream/buildstream!941

c0a8bb66

by Angelos Evripiotis at 2018-11-08T15:49:16Z

contributing: WIP again to be kind to reviewers

As someone coming from GitHub to GitLab, I was pleasantly surprised by
the 'filter by WIP status' option.

To make sure we get the most out of it, add a guideline to keep the
filter clean, to reduce reviewer burden.

f7231e90

by Angelos Evripiotis at 2018-11-08T15:49:16Z

contributing: non-WIP should always be landable

I was recently surprised that we don't prefer addressing review comments
in 'fixup!' commits.

Coming from GitHub, I've found that fixup commits make it easier for
reviewers to see what has changed since their last review. The idea is
to use '--autosquash' before landing to clean up the history again.

It's a pleasant surprise that it's easy to keep track of what changed in
merge-requests between pushes, so we can always keep the history clean.

Document this, so folks like me can see the light sooner.

f7643440

by Angelos Evripiotis at 2018-11-08T15:49:16Z

contributing: emphasise the 'why' in commits

Explain why it's useful to mention the decisions made for a change and
provide links for background, e.g. issue numbers.

My general experience of git histories is that folks don't do enough
explaining, so it's great we have a contributing section for it.

I've found it's easier for folks to stick to rules when they know the
practical reasons for them, so provide some.

dd5e7b04

by Angelos Evripiotis at 2018-11-08T16:17:04Z

Merge branch 'aevri/contributing_gitlab' into 'master'

Add more to GitLab-relevant parts of contributing

See merge request BuildStream/buildstream!935

fe33e328

by James Ennis at 2018-11-08T17:54:18Z

using_config.rst: Add documentation to showing how to impose quotas on the local cache

This patch partially resolves #700

09faf002

by James Ennis at 2018-11-08T17:54:18Z

artifactcache.py: Fix misleading error message when using % cache quota

Due to the changed Exception message, this patch also changes
the test_parse_size_over_1024T test in misc.py

d153453c

by Javier Jardón at 2018-11-08T18:22:54Z

Merge branch 'jennis/quota_declaration_fix' into 'master'

Add local cache expiry documentation and fix misleading error message when specifying a percentage cache quota

Closes #700

See merge request BuildStream/buildstream!939

7738f6df

by Daniel Silverstone at 2018-11-09T11:25:44Z

plugin.py: Redirect DEBUG from `__del__` to `sys.stderr`

Previously, debug messages from `Plugin.__del__()` went to stdout
which was a problem for the test suite in extremely rare and hard to
predict circumstances.  This corrects that by sending the message to
stderr where all the rest of the messages tend to go.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

62f59eaa

by Valentin David at 2018-11-09T13:30:06Z

Merge branch 'danielsilverstone-ct/plugin-destroy-debug-to-stderr' into 'master'

plugin.py: Redirect DEBUG from `__del__` to `sys.stderr`

See merge request BuildStream/buildstream!944

35ec6b6a

by Jim MacArthur at 2018-11-09T14:58:22Z

_casbaseddirectory.py: Enable direct CAS-to-CAS import.

Fixes issue #574.

b93100ec

by Jim MacArthur at 2018-11-09T14:58:22Z

_casbaseddirectory: Fix order of list_modified_paths

This will now return paths in the same order as list_relative_paths.

78691fa0
by Jim MacArthur at 2018-11-09T14:58:22Z
```
virtual_directory_import.py: New test
```

7ce6581b

by Jim MacArthur at 2018-11-09T15:25:46Z

Merge branch 'jmac/cas_to_cas_v2' into 'master'

Direct CAS-to-CAS import

Closes #574

See merge request BuildStream/buildstream!911

6f4351ec

by Benjamin Schubert at 2018-11-09T16:53:42Z

source.py: don't let StopIteration propagate to silence() contextmanager

As per PEP 0479 (https://www.python.org/dev/peps/pep-0479/),
StopIteration thrown in context managers are not valid starting from
Python 3.7.

e6e03451

by richardmaw-codethink at 2018-11-12T11:05:15Z

Merge branch 'bschubert/fix-silence-stopiteration-pep-0479' into 'master'

source.py: don't let StopIteration propagate to silence() contextmanager

See merge request BuildStream/buildstream!945

ae5ccd76

by Phillip Smyth at 2018-11-12T14:57:51Z

_project.py: Validate nodes early in Project._load

_project.py: Added validate_nodes() helper function to prevent duplicate lists

element-path was being used before node validation resulting in uncaught errors

56a54161

by Phillip Smyth at 2018-11-12T14:57:51Z

tests/frontend: Add test for invalid element-path

e55a9703

by Jürg Billeter at 2018-11-12T15:44:45Z

Merge branch 'element-path_not_validated' into 'master'

Element path not validated before use

See merge request BuildStream/buildstream!937

c87d6e45
by Valentin David at 2018-11-14T11:43:33Z
```
Run tests on aarch64

Fixes #755
```

16b01489

by Valentin David at 2018-11-14T11:43:33Z

Disable cachekey tests on other architectures than x86_64

85046b29

by Valentin David at 2018-11-14T11:43:33Z

Disable tests on example on other architectures than x86_64

dd5a073b
by Valentin David at 2018-11-14T11:43:33Z
```
Add support for aarch64 in a test
```
c5d72ae7
by Valentin David at 2018-11-14T11:52:53Z
```
Test aarch64 only overnight
```

264a57f6

by Javier Jardón at 2018-11-14T12:22:03Z

Merge branch 'valentindavid/integration-tests-aarch64' into 'master'

Run tests on aarch64

Closes #369 and #755

See merge request BuildStream/buildstream!948

da735e56

by Richard Maw at 2018-11-14T13:30:34Z

_platform/linux.py: Move get_bwrap_version into _site.py

It's inconvenient to have to create a Linux platform to parse the bwrap version
and we want to get the version in a consistent manner.

e7633500

by Richard Maw at 2018-11-14T13:30:34Z

buildstream/sandbox/_sandboxbwrap.py: Distinguish sandbox failure from command failure

If `bwrap` fails to set up the sandbox and start the payload command
it won't write an exit-code in --json-status-fd,
so we can report if it was a sandboxing failure if we don't get exit-code status
and a payload command failure if we do and it's non-zero.

Closes https://gitlab.com/BuildStream/buildstream/issues/286

e9e08823

by Richard Maw at 2018-11-14T13:30:34Z

tests/testutils/site.py: Check for bwrap supporting --json-status-fd

90ca007e

by Richard Maw at 2018-11-14T13:30:34Z

tests/integration/sandbox-bwrap.py: Test distinguishing sandbox exit code from command

327b19dd

by richardmaw-codethink at 2018-11-14T13:59:16Z

Merge branch 'richardmaw/distinguish-sandboxing-build-fail' into 'master'

Distinguish between bubblewrap sandboxing failure and command failure

Closes #286

See merge request BuildStream/buildstream!868

082b3811

by Angelos Evripiotis at 2018-11-15T19:48:12Z

contributing: add guidance on unit tests

Decrease uncertainty around whether unit tests are welcome in the
project or not.

b5b79056

by Jürg Billeter at 2018-11-15T20:24:58Z

Merge branch 'aevri/unit_tests' into 'master'

contributing: add guidance on unit tests

See merge request BuildStream/buildstream!943

199bfff1

by Tom Pollard at 2018-11-17T13:07:59Z

Add cli main and user config option for 'pull-buildtrees' context.

_context.py: Add pull_buildtrees global user context, the default
of which is set to False via the addition of pull-buildtrees to
userconfig.yaml cache group.

_frontend/app.py & cli.py: Add --pull-buildtrees as a bst main
option, which when passed will override the default or user defined
context for pull_buildtrees.

tests/completions/completions.py: Update for the added flag.

454dcaf1

by Tom Pollard at 2018-11-17T13:07:59Z

Don't pull artifact build trees by default.

The addition of cached build trees being included in element
artifacts has led to mostly redundant download overheads when
pulling from a remote artifact server. As such the default behaviour
of pull shouldn't fetch the build tree object if available.

element.py: extend relevant pull logic for specified subdir
consideration and ensure push logic does not lead to partial
artifact pushes. Change assumption that the buildtree can be
extracted if artifact is cached. __cached_buildtree() and
__pull_directories() helpers added.

_artifactcache/: artifactcache.py & cascache.py inclusion of
helper functions for subdir artifact checking & fetching, fetch
logic extended to only pull required artifact directories.
extract & checkout updated to handle full/partial operation.

tests/: addition of integration test pullbuildtrees.py,
buildtrees.py adapted cli options, testutils/artifactshare.py
has_artifact changed to return artifact digest on true condition.

0d309478

by Jürg Billeter at 2018-11-17T13:49:58Z

Merge branch 'tpollard/494' into 'master'

Don't pull artifact buildtrees by default

See merge request BuildStream/buildstream!786

27932739

by Benjamin Schubert at 2018-11-19T09:46:39Z

plugins/sources/pip.py: also look for python version named "python"

In some virtual environments, we might not have the binary with the
exact python version, which would lead the current pip plugin to
fail to find a working pip version.

This adds "python" at the start of the list of valid python versions

ea2de561

by Benjamin Schubert at 2018-11-19T10:19:24Z

Merge branch 'bschubert/fix-pip-python' into 'master'

plugins/sources/pip.py: also look for python version named "python"

Closes #758

See merge request BuildStream/buildstream!942

f23b6031

by Benjamin Schubert at 2018-11-19T11:39:51Z

tests/frontend/buildtrack.py: check for success after invocation

Previously the result of the first call in test_build_track was not
checked to be successful, which would lead to potential erros later on
that were hard to debug.

Let's check it everytime

a6defc0b

by Benjamin Schubert at 2018-11-19T11:39:51Z

Fix os.rename in git source element to correctly handle error codes

According to the documentation
(https://www.unix.com/man-page/POSIX/3posix/rename/), when the directory
already is there, either EEXIST or ENOTEMPTY could be thrown.

Previously only ENOTEMPTY was checked.

Done:
  - Separated the move into its own function
  - Check for both errors
  - Create unit tests for it, covering most test cases

88089d2d

by Benjamin Schubert at 2018-11-19T11:39:51Z

Extract atomic move function to utils.py

Moving atomically a file/directory can be tricky since different
errors might be raised for the same underlying problem.

Having a utility function to reduce this discrepancies will help
in ensuring we have correct behavior

fd9e46be

by Benjamin Schubert at 2018-11-19T11:39:51Z

cascache.py: use move_atomic instead of manual error checking

d32e0b83

by Benjamin Schubert at 2018-11-19T11:39:51Z

sources/pip.py: Use move_atomic instead of manual os.rename

This uses move_atomic insteand of the manual os.rename and manual error
checking and throws a SourceError for consistency with other modules.

6f837118

by Benjamin Schubert at 2018-11-19T12:22:40Z

Merge branch 'bschubert/fix-atomic-move-git-repo' into 'master'

Fix os.rename in git source element to correctly handle error codes

See merge request BuildStream/buildstream!938

30b72244

by Chandan Singh at 2018-11-19T12:35:01Z

buildstream/_pipeline.py: Simplify error message for assert_consistent()

Currently, `Pipeline.assert_consistent()` prints an error message like
the one below when certain elements have inconsistent sources:

```
...
      Element: hello.bst is inconsistent
        Source tar source at hello.bst [line 16 column 2] is missing ref
...
```

Drop the word "Source" from the beginning of the message as
"Source <source-kind> source ..." is kind of awkward to read.

76c5d2f8

by Chandan Singh at 2018-11-19T12:35:01Z

Add `bst source-checkout` command

As discussed in
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
add `bst source-checkout` command. This will allow users to checkout
sources for a given target.

* _frontend/cli.py: Add source-checkout command
* _pipeline.py: Add assert_sources_cached() method
* _stream.py: Add source_checkout method, abstract out
  __check_location_writable() method that used to part of checkout()

9f629638

by Chandan Singh at 2018-11-19T13:03:36Z

Merge branch 'chandan/source-checkout' into 'master'

Add `bst source-checkout` command

See merge request BuildStream/buildstream!820

bc827cc8

by Abderrahim Kitouni at 2018-11-19T14:55:15Z

plugins/elements/cmake.yaml: always specify variable types

cmake sometimes misinterprets relative paths as relative to the current directory
if this is not specified. See freedesktop-sdk/freedesktop-sdk#431

adjust tests/format/variables.py accordingly.

5fbc5f41

by Valentin David at 2018-11-19T15:31:05Z

Merge branch 'abderrahim/cmake-variable-types' into 'master'

plugins/elements/cmake.yaml: always specify variable types

Closes #761

See merge request BuildStream/buildstream!947

06e28860

by Benjamin Schubert at 2018-11-19T15:52:24Z

Don't cache sandbox errors

Sandbox errors (like missing host tools) are dependent on the host
system and rarely on what is actually done.

It is therefore better to not cache them as they are subject to
change between two runs.

Also add test to ensure sandbox failure are not cached

dd36cfbc

by Benjamin Schubert at 2018-11-19T15:52:24Z

element.py: Only cache on element error or success

Reverse the way we were handling caching by only caching
when we know it's a problem with the build or the build is a
success in order not to cache transient errors.

eac7274d

by Benjamin Schubert at 2018-11-19T16:29:07Z

Merge branch 'bschubert/dont-cache-errors-from-host-tools' into 'master'

Don't cache sandbox failures

Closes #727

See merge request BuildStream/buildstream!895

625dfe1f

by Chandan Singh at 2018-11-19T16:41:17Z

NEWS: Add entry for the new source-checkout command

`bst source-checkout` command was recently added in
https://gitlab.com/BuildStream/buildstream/merge_requests/820. Add a
NEWS entry for the new command.

d4f12184

by Chandan Singh at 2018-11-19T17:15:10Z

Merge branch 'chandan/source-checkout-news' into 'master'

NEWS: Add entry for the new source-checkout command

See merge request BuildStream/buildstream!960

ba6c96e8

by Richard Maw at 2018-11-19T19:54:30Z

CasBasedDirectory: Change constructor to take a CASCache instead of a Context

The Context was only used to obtain a reference to the CASCache
and set the unused cas_directory field.

8a0dc3a3

by Jürg Billeter at 2018-11-19T19:54:30Z

tests/testutils/artifactshare.py: Do not create a fake context

The fake context did not set the cache quota, triggering an error. With
CASCache now separate from ArtifactCache, we can instantiate a CASCache
without context.

8722aced

by Jürg Billeter at 2018-11-19T19:54:30Z

tests/plugins/pipeline.py: Load context default values

fc56ffa4

by Jürg Billeter at 2018-11-19T19:54:30Z

_context.py: Drop duplicated default values for user configuration

The default values are in userconfig.yaml, together with the
documentation. The default values should not be duplicated in
_context.py.

3f663d82

by Jürg Billeter at 2018-11-19T20:36:57Z

Merge branch 'juerg/context-default-values' into 'master'

_context.py: Drop duplicated default values for user configuration

See merge request BuildStream/buildstream!953

b498cce7
by Angelos Evripiotis at 2018-11-20T11:17:38Z
```
_yaml: document node_get()'s default_value arg
```

eb2d376f

by Angelos Evripiotis at 2018-11-20T11:17:38Z

_context: allow 'terminate' for scheduler.on-error

Enable this option of 'terminate', which is mentioned in userconfig.yaml
and handled in _frontend/app.py:_handle_failure(). It appears to have
been left out of the valid_actions as an oversight.

Originally introduced in
https://gitlab.com/BuildStream/buildstream/commit/2622d5da9bd9fefd87436613d6e3e9770fdd0f28

b81c4333

by Angelos Evripiotis at 2018-11-20T11:17:38Z

_context: refactor, extract _node_get_option_str

Use a new helper function to simplify working with nodes that can only
accept certain strings. This will be used when adding the prompt.*
config options.

In later work we can see if this function would be useful elsewhere, and
could be added to '_yaml.py'.

27ca6593

by Angelos Evripiotis at 2018-11-20T11:19:33Z

Add prompt.auto-init buildstream.conf option

Provide an option in buildstream.conf to disable the 'Would you like to
...' prompt when we cannot resolve a project.

Some users prefer not to be interrupted by such prompts, so pave the way
to creating options to disable all those that might get in the way.

Follow the example of the advice.* options 'git-config', and create a
namespace for these UI options grouped by behaviour, rather than an
over-reaching 'ui.*' namespace. In later work perhaps we'll also add
'advice.*' options.

Add a NEWS item for this.

7ae3a3d2

by Angelos Evripiotis at 2018-11-20T11:43:49Z

Add prompt.workspace-... options

Provide options in project.conf to disable the 'Are you sure ...'
prompts when making destructive changes:

- Add prompt.really-workspace-close-remove-dir
- Add prompt.really-workspace-reset-hard

Add a NEWS item for these.

8071c00c

by Angelos Evripiotis at 2018-11-20T12:12:11Z

Merge branch 'aevri/prompt-config' into 'master'

Add 'prompt.*' config options to buildstream.conf

See merge request BuildStream/buildstream!887

fd41b0b5

by Raoul Hidalgo Charman at 2018-11-20T16:32:10Z

_sandboxremote.py: Add sigterm handler that sends CancelOperation

c6306b88

by Jürg Billeter at 2018-11-20T16:59:45Z

Merge branch '725-job-cancellation-on-remote-builds' into 'master'

_sandboxremote.py: Add sigterm handler that sends CancelOperation

Closes #725

See merge request BuildStream/buildstream!900

9fa8a881

by James Ennis at 2018-11-20T17:00:42Z

_yaml.py: Fix incorrect error message

This patch ensures that we receive an appropriate error
message if we specify an absolute path that leads within
the project.

fb8860db

by Jürg Billeter at 2018-11-20T17:32:58Z

Merge branch 'jennis/less_restrictive_pathways' into 'master'

Fix warning when using an absolute path which exists inside the project

See merge request BuildStream/buildstream!956

87751350

by Benjamin Schubert at 2018-11-20T18:10:48Z

gitlab-ci.yml: Use CI_PROJECT_DIR instead of $(pwd) for cache

8668df9f

by Benjamin Schubert at 2018-11-20T18:10:48Z

Export variables globally for less repetition.

This doesn't change the test behavior and enable for better
composability of tests

cc6a7f0d

by Benjamin Schubert at 2018-11-20T18:10:48Z

gitlab-cy.yml: make linux-tests-template more composable.

This will allow for further de-duplication of work.

eb37dd89

by Benjamin Schubert at 2018-11-20T18:10:48Z

gitlab-ci.yml: remove unused artifacts for unix tests

The unix-logs is always empty, there is therefore no reasons of keeping
it there

e030b48e

by Benjamin Schubert at 2018-11-20T18:10:48Z

gitlab-ci.yml: use template for unix tests

This makes all tests use the same template, which makes reasoning
about them simpler

57e6358f

by Benjamin Schubert at 2018-11-20T18:10:48Z

Ensure tests requiring bubblewrap are correctly marked

e70dd1f8

by Benjamin Schubert at 2018-11-20T18:10:48Z

gitlab-ci.yml: Add linux without brwap/ostree to ensure tests behave

This allows us to make sure that tests behave nicely while missing brwap
and ostree.

f7cb4386

by Benjamin Schubert at 2018-11-20T18:10:48Z

gitlab-ci.yml: Extract test command as a variable

This removes the need of having it synchronized in multiple places

7f316a76

by Benjamin Schubert at 2018-11-20T18:41:03Z

Merge branch 'bschubert/add-fedora-nobwrap-tests' into 'master'

Add tests for fedora with no bubblewrap available

See merge request BuildStream/buildstream!949

a50c598e

by Benjamin Schubert at 2018-11-21T09:24:15Z

utils.py: Ensure move_atomic booleans are keyword only arguments

This is to be coherent with the rest of the codebase

b2f604b8
by Benjamin Schubert at 2018-11-21T09:24:15Z
```
utils.py: improve documentation of move_atomic
```

daca0c01

by Benjamin Schubert at 2018-11-21T09:52:42Z

Merge branch 'bschubert/mr938-comments' into 'master'

Followup on MR 938, addressing additional comments

See merge request BuildStream/buildstream!958

050cd1bd

by Tom Pollard at 2018-11-21T11:37:48Z

tests/plugin/pipeline.py: Avoid using host user conf

create_pipeline was passing no config parameter to context.load,
leading to _context.py loading in the host's buildstream user conf
when running tests locally, potentially leading to unwanted errors.

58353aad

by Jürg Billeter at 2018-11-21T14:29:49Z

Merge branch 'tpollard/pipelinehostconfig' into 'master'

tests/plugin/pipeline.py: Avoid using host user conf

See merge request BuildStream/buildstream!961

2022c3e8

by William Salmon at 2018-11-21T15:53:03Z

tests/frontend/workspace.py: Refactor support function

The Refactor now makes it possible to test for the forth coming multi
element workspace open commands and to test for failure.

7bdd44b2

by William Salmon at 2018-11-21T15:53:03Z

Workspace CLI update

This is to update the workspace CLI to as agreed on the mailing list
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00046.html

This patch also introduces the default workspace directory.

6612e708
by William Salmon at 2018-11-21T15:53:03Z
```
Added a news item for workspace open changes
```

4d6e2cd3

by Will Salmon at 2018-11-21T16:20:26Z

Merge branch 'willsalmon/defaultWorkspaces' into 'master'

Updated Workspace CLI

See merge request BuildStream/buildstream!897

9144b6eb