[Notes] [Git][BuildStream/buildstream][gokcennurlu/dont_cache_if_stage

Jürg Billeter pushed to branch gokcennurlu/dont_cache_if_stage_fails at BuildStream / buildstream

Commits:

d2105909

by Raoul Hidalgo Charman at 2018-12-18T11:13:56Z

Add remote execution instance option

This is used when sending execution requests, to specify which instance of the
execution server to use.

Partial fix for #627.

a3bbec23

by Jim MacArthur at 2018-12-18T11:13:56Z

sandboxremote: Add server/storage config with defaults

Adds instance name support for the remote execution storage service.

89219f61

by Jim MacArthur at 2018-12-18T11:13:56Z

_cascache.py: Add instance names to GRPC calls

Adds the 'instance_name' parameter, which may be None, to most GRPC
calls in the CASCache object. ByteStream requests already have
instance_name supplied in the resource name, so do not need the
parameter.

Closes #627.

3dc20963

by Jim MacArthur at 2018-12-18T11:13:56Z

Documentation: Include instance-name in remote execution documentation

644d8b28

by Jim MacArthur at 2018-12-18T11:45:30Z

Merge branch 'raoul/627-RE-instance-config' into 'master'

Remote-execution instance configuration support

Closes #627

See merge request BuildStream/buildstream!952

e29aea36

by William Salmon at 2018-12-19T13:23:19Z

Basic options for shell --build to use buildtrees

Fixes issue #740

898a23a5

by Will Salmon at 2018-12-19T14:35:41Z

Merge branch 'willsalmon/shellBuildTrees' into 'master'

Shell --build has optional buildtrees

Closes #740

See merge request BuildStream/buildstream!986

a2f1d879
by Javier Jardón at 2018-12-19T15:36:11Z
```
README.rst: Add license badge
```

aae5e4b3

by Javier Jardón at 2018-12-19T16:09:48Z

Merge branch 'jjardon/license_badge' into 'master'

README.rst: Add license badge

See merge request BuildStream/buildstream!1014

2b767fe8

by Jürg Billeter at 2018-12-20T10:06:11Z

Move fetch logic from FetchQueue to Element

The queue shouldn't need to know about individual sources. This is in
line with _track() and _get_consistency().

7a102144

by Jürg Billeter at 2018-12-20T10:06:11Z

element.py: Do not call fetch() for cached sources

b325989e

by Jürg Billeter at 2018-12-20T10:07:20Z

tests/sources: Test that fetch() is not called for cached sources

77d8ad45

by Jürg Billeter at 2018-12-20T10:42:39Z

Merge branch 'juerg/fetch' into 'master'

Do not call fetch() for cached sources

See merge request BuildStream/buildstream!992

9835b7f1

by Chandan Singh at 2018-12-20T12:50:02Z

Expose base class for Git source plugins

Add a `_GitSourceBase` class to act as the base class for deriving
source plugins that work with Git. The first user of this base class is
the `git` source plugin that's in core at the moment.

The contents of this base class is almost identical to the existing `GitSource`
class. The two notable differences are:

- the private methods now have leading underscores, as some were missing
  it originally
- the name of the class

Note that we are exposing a private member here as we expect it to move to a
separate package soon. See the following discussion for more details:
https://gitlab.com/BuildStream/buildstream/issues/739#note_124819869

Fixes #739.

14da6955

by Chandan Singh at 2018-12-20T13:34:10Z

Merge branch 'chandan/abstract-git-source' into 'master'

Expose base class for Git source plugins

Closes #739

See merge request BuildStream/buildstream!1019

7368f569

by Angelos Evripiotis at 2018-12-20T13:59:23Z

_project::_find_project_dir: fix error message

Now that we are also looking for WORKSPACE_PROJECT_FILE, we should add
it to the error message. While we're there, also mention the directory
we are looking in - this could help scripters be clear about what went
wrong.

The new error message looks like this:

    Error loading project: None of ['project.conf', '.bstproject.yaml']
    found in '/src/temp/blah' or any of its parent directories

46efc91d

by Angelos Evripiotis at 2018-12-20T13:59:23Z

BREAK: remove auto-init behaviour

In the event that the project could not be found, stop BuildStream from
asking if the user would like to create a new project. Exit with error
instead, and give a hint to the user in case they're new.

As proposed on the mailing list here:
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00082.html

The new interaction looks like this:

    $ bst show nonsuch.bst
    No project found. You can create a new project like so:

        bst init

    Error loading project: None of ['project.conf', '.bstproject.yaml']
    found in '/src/temp/blah' or any of its parent directories

Fixes #826

e0c575c4

by Angelos Evripiotis at 2018-12-20T14:37:38Z

Merge branch 'aevri/rm-autoinit' into 'master'

BREAK: remove auto-init behaviour

Closes #826

See merge request BuildStream/buildstream!1015

c05d8b4f

by Chandan Singh at 2018-12-20T15:53:12Z

tests/frontend/logging.py: Fix regex Deprecation Warning

Use raw strings for regex searches, which is the preferred way to do
regular expressions in Python.

Without this patch, currently we get the following warnings:

```
tests/frontend/logging.py:44
  /builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:44: DeprecationWarning: invalid escape sequence \[
    m = re.search("\[\d\d:\d\d:\d\d\]\[\]\[\] SUCCESS Checking sources", result.stderr)

tests/frontend/logging.py:80
  /builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:80: DeprecationWarning: invalid escape sequence \d
    m = re.search("\d\d:\d\d:\d\d,\d\d:\d\d:\d\d.\d{6},\d\d:\d\d:\d\d,,,SUCCESS,Checking sources", result.stderr)
```

e8055a56

by Chandan Singh at 2018-12-20T15:53:12Z

buildstream/utils.py: Fix regex Deprecation Warning

Specify flags at the start of the _expression_ as per the recommendation
of the standard library.

Without this patch, we currently get the following warning:

```
tests/examples/junctions.py::test_open_cross_junction_workspace
  /builds/BuildStream/buildstream/dist/buildstream/buildstream/utils.py:213: DeprecationWarning: Flags not at the start of the _expression_ '\\/[^/]*\\Z(?ms)'
    regexer = re.compile(_expression_)
```

cd4889af

by Chandan Singh at 2018-12-20T16:34:08Z

Merge branch 'chandan/fix-warning' into 'master'

Fix Deprecation warnings from regex module

See merge request BuildStream/buildstream!1010

ac995236

by Tom Pollard at 2018-12-20T17:18:02Z

_stream.py: fix _buildtree_pull_required logic

c3153dea

by Tom Pollard at 2018-12-21T10:10:58Z

Merge branch 'tpollard/fixbuildtreereq' into 'master'

_stream.py: fix _buildtree_pull_required logic

See merge request BuildStream/buildstream!1021

caf49669

by Tristan Van Berkom at 2018-12-26T19:54:05Z

dev-requirements.txt: Require at least pytest 3.9

Recently we have been adding some tests which use the new tmp_path
fixture from pytest, this is new in 3.9.

425781db

by Tristan Van Berkom at 2018-12-26T19:54:05Z

_signals.py: Fixing new (simplifiable-if-_expression_) linter errors

1f8cc996

by Tristan Van Berkom at 2018-12-26T19:54:05Z

element.py: Fixing new (unnecessary-pass) and (simplifiable-if-_expression_) linter errors

89632150

by Tristan Van Berkom at 2018-12-26T19:54:05Z

source.py: Fixing new (unnecessary-pass) linter error

26d135ba

by Tristan Van Berkom at 2018-12-26T19:54:05Z

_artifactcache/casserver.py: Fixed new (wrong-import-order) linter error

d0988f80

by Tristan Van Berkom at 2018-12-26T19:54:05Z

_frontend/cli.py: Fixing new (unnecessary-pass) linter errors

4d54ab78

by Tristan Van Berkom at 2018-12-26T19:54:05Z

_frontend/complete.py: Fixing new (simplifiable-if-_expression_) linter error

ee0d5bf6

by Tristan Van Berkom at 2018-12-26T19:54:05Z

_frontend/widget.py: Fixing new (wrong-import-order) linter error

36295282

by Tristan Van Berkom at 2018-12-26T19:54:05Z

plugins/elements/import.py: Fixing new (duplicate-string-formatting-argument) linter error

4ffe03c6

by Tristan Van Berkom at 2018-12-26T19:54:05Z

buildstream/storage/_casbaseddirectory.py: Fixing new (unnecessary-pass) linter errors

80c137d5

by Tristan Van Berkom at 2018-12-26T19:54:05Z

dev-requirements.txt: Require new version of pytest-datafiles

Since Thomas Nixon fixed the datafiles plugin to be compatible
with pytest >= 3.8.0, we can now depend on it and avoid the
warnings and errors which the older datafiles plugin causes.

This fixes issue #636

61906a14

by Tristan Van Berkom at 2018-12-26T19:54:05Z

dev-requirements.txt: Remove the restriction on pylint.

After having fixed the more recent linting errors, the
tests should pass with recent versions of pylint. Tested
with pylint 2.2.2.

94b9948c

by Tristan Van Berkom at 2018-12-26T22:35:29Z

Merge branch 'tristan/test-fixes' into 'master'

Misc fixes to the tests and newer linter errors

Closes #636

See merge request BuildStream/buildstream!1011

fe4fc0b5

by Tristan Van Berkom at 2018-12-26T23:19:07Z

_downloadablefilesource.py: Avoid crashes when HOME is unset

The python netrc module will raise OSError in the case
that HOME is not set, this was discovered while running
tests under tox.

c61e1793

by Tristan Van Berkom at 2018-12-26T23:19:07Z

tests/testutils/runcli.py: Fixed broken environment handling

Treat None values in the passed dictionary as keys to be removed
from the environment, this was already happening at restoration time.

50c1bb7f

by Tristan Van Berkom at 2018-12-26T23:19:07Z

tests/sources/tar.py: Test that we don't crash when HOME is unset

This happens when the netrc module is searching for a ~/.netrc
file and it doesnt find any HOME set.

32a101f6

by Tristan Van Berkom at 2018-12-26T23:53:29Z

Merge branch 'tristan/fix-netrc-crasher' into 'master'

Fix netrc crasher

See merge request BuildStream/buildstream!1025

a080b6c7

by Tristan Van Berkom at 2018-12-31T19:03:25Z

tests/loader/junctions.py: Use Result checking APIs instead of manually

Better to use the same API everywhere for thecking CLI results.

d7c8b54d

by Tristan Van Berkom at 2018-12-31T19:14:16Z

tests/loader/junctions.py: Test the error we expect when a cross junction element is not found

560a6342

by Tristan Van Berkom at 2018-12-31T19:49:24Z

Merge branch 'tristan/junction-tests' into 'master'

Junction test addition

See merge request BuildStream/buildstream!1029

8a020a5a

by Chandan Singh at 2018-12-31T19:54:25Z

_loader/loader.py: Refactor warnings about element names in one method

Currently we some duplication in the way we check for invalid filenames.
To make it more robust and allow room for adding more warnings, refactor
it into a separate method `_warn_invalid_elements()` that handles just
this.

87568074

by Chandan Singh at 2018-12-31T19:54:25Z

_loader/loader.py: Add warnings about invalid characters in filename

See https://mail.gnome.org/archives/buildstream-list/2018-December/msg00061.html
for some related discussion.

4b2992c1

by Chandan Singh at 2018-12-31T19:54:25Z

tests/buildcheckout.py: Add tests for invalid chars in filename

ce9e9c2d
by Chandan Singh at 2018-12-31T19:54:25Z
```
doc: Add note about valid element names
```

b848172c

by Tristan Van Berkom at 2018-12-31T20:29:51Z

Merge branch 'chandan/element-filename-requirements' into 'master'

Add warnings about invalid characters in filename

See merge request BuildStream/buildstream!1028

bb9df2a1

by Chandan Singh at 2019-01-02T11:59:35Z

doc/Makefile: Update comment about sphinx entrypoint

https://github.com/sphinx-doc/sphinx/issues/4375 has been closed as
`wontfix`. Update the comment in doc/Makefile that mentions it, based on
the discussion on the issue.

43eaf6ee

by Chandan Singh at 2019-01-02T13:42:10Z

Merge branch 'chandan/update-doc-makefile-note' into 'master'

doc/Makefile: Update comment about sphinx entrypoint

See merge request BuildStream/buildstream!1026

3b5c8a28

by Tristan Maat at 2019-01-02T15:57:13Z

.gitlab-ci.yml: Use new testsuite images with pycodestyle 1.4.0

We need to bump the testsuite image versions to use the new
pycodestyle update that allows excluding file paths, so that we can
exclude generated python code.

bd60e8a9

by Javier Jardón at 2019-01-02T16:59:41Z

Use pycodestyle instead of pep8

pep8 package was renamed to pycodestyle to reduce confusion.

Disabled warnings:
W504 - Line break after binary operator (seems people like this)
W605 - Invalid escape sequence (some of our regexes use these)

32c47d1c

by Tristan Maat at 2019-01-02T17:32:09Z

Merge branch 'jjardon/pycodestyle' into 'master'

Use pycodestyle instead pep8 python module

See merge request BuildStream/buildstream!638

1cbd9a73

by Chandan Singh at 2019-01-02T19:39:58Z

Add requirements files for install, test and plugin dependencies

Add `.in` and `.txt` requirements files for BuildStream's pure python
dependencies. For each pair, the `.in` file is supposed to capture the
loose version requirements, and the corresponding `.txt` file is
supposed to have frozen requirements. We have 3 such sets:

* `requirements`: BuildStream's runtime dependencies
* `dev-requirements`: Dependencies for running tests
* `plugin-requirements`: Dependencies for core plugins

Note that the frozen requirements files will only be used for testing
purposes, and `setup.py` will continue to read loose requirements.

6b0cb5f3

by Chandan Singh at 2019-01-02T19:41:21Z

Add tox.ini to enable running tests using tox

Add `tox.ini` file that will enable us to use
[tox](https://tox.readthedocs.io/) as a frontend for running tests.
Since we share the config via `setup.cfg` and requirements via
requirements files, commands like `python3 setup.py test` will continue
to work.

9d2d1d4f

by Chandan Singh at 2019-01-03T03:31:52Z

.gitlab-ci.yml: Run tests using tox

Instead of invoking tests throung `setup.py`, use `tox` as a frontend in
the CI pipelines.

8ae04283

by Tristan Van Berkom at 2019-01-03T03:31:52Z

CONTRIBUTING.rst: Updated to reflect running tests using tox.

e7e2a5c1

by Chandan Singh at 2019-01-03T03:31:52Z

CONTRIBUTING.rst: Add steps for installing non-python build dependencies

Since we don't allow use of site packages by default in our `tox`
configuration, people will need to install non-python build dependencies
of some of our dependencies that do not provide pre-built wheels. We
have two such packages at the moment:

* psuitl: requires python C headers, compiler
* pygobject: requires python C headers, pkg-config, compiler, cairo
  headers, and gobject libraries

afa0a369

by Chandan Singh at 2019-01-03T03:31:52Z

Move all requirements files into "tools" directory

These new `.in` and `.txt` are making the repository look very
cluttered. Move them to a separate `tools` directory to make it look a
bit cleaner.

3fae3446

by Chandan Singh at 2019-01-03T03:44:17Z

Move sphinx build functionality to tox

Currently the CI and the docs both have to duplicate the same inforation
about how to gather dependencies etc, and have to use hacky ways to run
them.

Add a new `docs` environment to our tox setup so that building docs is
as simple as running `tox -e docs`.

1f7bc655

by Chandan Singh at 2019-01-03T03:44:17Z

CONTRIBUTING.rst: Add instructions to recreate tox environment

When we update our dependencies, developers will eventually need to
recreate their `tox` environments to get correct results. This happens
because `tox` isn't particularly good at recognizing changes in the
requirements files.

669b55b0

by Tristan Van Berkom at 2019-01-03T15:47:27Z

Merge branch 'chandan/toxify' into 'master'

Use tox as a frontend for running tests

See merge request BuildStream/buildstream!1027

ca547f19

by Chandan Singh at 2019-01-03T19:34:49Z

tests/frontend/buildcheckout.py: Fix bad filename issue for Windows

In BuildStream/buildstream!1028, we added a test specifically to test
that BuildStream correctly raises an warning when the name of an element
contains characters that are invalid on Windows. Unfortunately, we
didn't see it coming that it would make it impossible to checkout this
branch on Windows.

Fix it by generating this file, only if we are not running on Windows.

* tests/testutils/site.py: Add `IS_WINDOWS` check
* tests/frontend/buildcheckout.py: Generate file with invalid filename
  on the fly
* Remove tests/frontend/project/elements/invalid-chars.

Fixes #842.

Note that this may still cause issues on WSL when running tests on a
shared filesystem, but that seems to be a generic issue on WSL with
`os.rename`.

01c3761c

by Chandan Singh at 2019-01-03T20:03:32Z

Merge branch 'chandan/fix-bad-filename-mini-disaster' into 'master'

tests/frontend/buildcheckout.py: Fix bad filename issue for Windows

Closes #842

See merge request BuildStream/buildstream!1032

7850e4e2

by Tristan Van Berkom at 2019-01-03T20:55:00Z

setup.cfg: Specify the pycodestyle configurations separately from pytest configuration

3c9d592d

by Tristan Van Berkom at 2019-01-03T20:55:00Z

setup.cfg: Don't lint by default through setup.py test invocations

feba48e7

by Tristan Van Berkom at 2019-01-03T20:55:00Z

tox.ini: Added new 'lint' environment to lint separately

9caa0ea0

by Tristan Van Berkom at 2019-01-03T20:55:00Z

.gitlab-ci.yml: Update the CI to run the linter separately through tox

This should save us some cycles in CI, and also allow the
developer to more conveniently lint separately from testing.

91c8b634

by Tristan Van Berkom at 2019-01-03T21:04:49Z

CONTRIBUTING.rst: Adding instructions to run the linter separately

892bf99e

by Tristan Van Berkom at 2019-01-03T21:31:04Z

tools/requirements.{in,txt}: Fix warning about duplicate jinja

Apparently you need to spell Jinja2 with a capital J, otherwise
we find it listed twice in the resulting requirements.txt files.

7c681c42

by Tristan Van Berkom at 2019-01-03T21:32:40Z

tools/dev-requirements.{in,txt}: Removed requirements on pylint/codestyle pytest plugins

We no longer run the linters through pytest, so we don't need these
plugins anymore.

d6689ed2

by Tristan Van Berkom at 2019-01-03T22:09:02Z

Merge branch 'tristan/toxic-linting' into 'master'

Run the linter separately from the tests

See merge request BuildStream/buildstream!1033

e15278d7

by Thomas Coldrick at 2019-01-03T22:14:01Z

Use collections.abc for Mapping, Iterable

In _yaml.py and _frontend/complete.py we were getting pylint warnings
for using collections.Mapping and collections.Iterable, which are
abstract classes now provided from collections.abc. This patch just
uses the classes from the right place.

a85da591

by Tristan Van Berkom at 2019-01-03T22:36:16Z

Merge branch 'coldtom/collections' into 'master'

Use collections.abc for Mapping, Iterable

Closes #831

See merge request BuildStream/buildstream!1020

cf0e7d26

by Jürg Billeter at 2019-01-04T07:07:59Z

_frontend/cli.py: Reinstate support for guessing targets

Moving fetch and track to the source command group accidentally dropped
the support for guessing targets for these commands when invoked from a
workspace directory. This brings it back.

0751fc8f

by Jürg Billeter at 2019-01-04T11:27:58Z

Merge branch 'juerg/source-guess-target' into 'master'

_frontend/cli.py: Reinstate support for guessing targets

See merge request BuildStream/buildstream!1036

ebd965fe

by Chandan Singh at 2019-01-04T15:21:35Z

Rename "tools" directory to "requirements"

This seems like a better name for the directory, as it more closely
describes the purpose of its contents.

bb712886
by Chandan Singh at 2019-01-04T15:21:35Z
```
Add Makefile to update requirements files
```

b9792837

by Chandan Singh at 2019-01-04T15:21:35Z

CONTRIBUTING.rst: Add instructions to update requirements files

Split the "The MANIFEST.in and setup.py" section in two:
"Managing data files" and "Updating BuildStream's Python dependencies".

Briefly explain the layout of `requirements` directory and add
instructions to use the Makefile added in the last commit.

45692335

by Tristan Van Berkom at 2019-01-04T15:50:14Z

Merge branch 'chandan/update-requirements-one-liner' into 'master'

Add Makefile to update requirements files

See merge request BuildStream/buildstream!1035

f42dcb54

by Chandan Singh at 2019-01-04T17:16:43Z

.gitlab-ci.yml: Remove prepare stage

As we now run tests using `tox`, we don't need to worry about manually
packing and unpacking BuildStream. So, we can remove the preapre stage
entirely.

Update `coverage` and nightly jobs to appropriately cope with this
change. Both these jobs now install all runtime dependencies from
requirements files.

ee29f207

by Chandan Singh at 2019-01-04T17:41:02Z

Merge branch 'chandan/no-sdist' into 'master'

.gitlab-ci.yml: Remove prepare stage

See merge request BuildStream/buildstream!1037

f122cfe8

by Jürg Billeter at 2019-01-06T17:57:33Z

_frontend/app.py: Use buildtree for interactive shell on build failure

Fixes: e29aea36 ("Basic options for shell --build to use buildtrees")

7dfb85b3

by Tristan Van Berkom at 2019-01-06T18:31:25Z

Merge branch 'juerg/shell-buildtree' into 'master'

_frontend/app.py: Use buildtree for interactive shell on build failure

See merge request BuildStream/buildstream!1039

891fcb0e

by Tristan Van Berkom at 2019-01-07T16:47:01Z

Fix stack traces discovered with ^C forceful termination.

  * utils.py:_kill_process_tree(): Ignore NoSuchProcess errors

    These are caused because we issue SIGTERM, and if the process
    has not exited after a timeout, we kill it.

  * _scheduler/jobs/job.py: Stop handling NoSuchProcess errors here
    redundantly, they are already ignored.

It seems that we were ignoring it after sleeping when terminating
tasks from the scheduler... but we were not ignoring it when performing
the same pattern in the `Plugin.call()` -> `utils._call()` path, so
we were still getting these exceptions at termination time from host
tool processes launched by source plugins.

5de42d43

by Tristan Van Berkom at 2019-01-07T18:00:37Z

Merge branch 'tristan/keyboard-interrupt-stack-trace' into 'master'

Fix stack traces discovered with ^C forceful termination.

See merge request BuildStream/buildstream!1043

059035b9

by Tristan Van Berkom at 2019-01-07T18:02:00Z

_scheduler/scheduler.py: Make _schedule_jobs() private

This is not used anywhere outside of the Scheduler, currently
only the Scheduler itself is allowed to queue a job at this level.

If the highlevel business logic for automatic queueing of auxiliary
jobs moves to another location, we can make this public again.

b83d1b1f

by Tristan Van Berkom at 2019-01-07T18:02:00Z

_scheduler/scheduler.py: Only run one cache size job at a time

When queuing the special cache management related cleanup and
cache size jobs, we now treat these jobs as special and do the
following:

  * Avoid queueing a cleanup/cache_size job if one is already queued

    We just drop redundantly queued jobs here.

  * Ensure that jobs of this type only run one at a time

    This could have been done with the Resources mechanics,
    however as these special jobs have the same properties and
    are basically owned by the Scheduler, it seemed more straight
    forward to handle the behaviors of these special jobs together.

This fixes issue #753

16a8816f

by Tristan Van Berkom at 2019-01-07T18:02:00Z

Scheduler: Introduced JobStatus instead of simple success boolean

This changes the deepest callback from when a Job completes to
propagate a JobStatus value instead of a simple boolean, and updates
all of the effected code paths which used to receive a boolean
to now handle the JobStatus values.

This further improves the situation for issue #753, as now we avoid
queueing cache size jobs for pull jobs which are skipped.

c2fc2a5e

by Tristan Van Berkom at 2019-01-07T18:02:00Z

_scheduler/jobs/job.py: Removed 'skipped' property

This is redundant now that we report it through the JobStatus.

3e3984ad

by Tristan Van Berkom at 2019-01-07T18:50:23Z

Merge branch 'tristan/one-cache-size-job' into 'master'

Only queue one cache size job

Closes #753

See merge request BuildStream/buildstream!1040

512c726e

by Tristan Van Berkom at 2019-01-08T03:38:11Z

sandbox/sandbox.py: Fix regression of command logging

Since we added batch commands, the batch commands print the
text of the commands directly in the message text, but this is wrong.

The detail string is the appropriate place for text of unknown lengths
(the user can actually configure how many max lines of commands they
want to see in their log), the message text itself should be controlled
and brief enough to avoid text wrapping.

01171988

by Tristan Van Berkom at 2019-01-08T04:20:14Z

Merge branch 'tristan/fix-command-status-messages' into 'master'

sandbox/sandbox.py: Fix regression of command logging

See merge request BuildStream/buildstream!1044

6c1d06d6

by Phil Dawson at 2019-01-08T10:24:32Z

element.py: remove reference to source bundle command

This command has been replacved by the bst source checkout command

914ecb72

by Jürg Billeter at 2019-01-08T10:54:02Z

Merge branch 'phil/remove-source-bundle-reference' into 'master'

element.py: remove documentation reference to source bundle command

See merge request BuildStream/buildstream!1041

81f7a964

by Tristan Van Berkom at 2019-01-08T16:55:58Z

sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.

When code is faulty, in a plugin or in the core, we should
get a stack trace and a BUG message.

edcc43ed

by Tristan Van Berkom at 2019-01-08T19:19:10Z

Merge branch 'tristan/sandbox-programming-error' into 'master'

sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.

See merge request BuildStream/buildstream!1046

17c5ca2d

by Valentin David at 2019-01-09T10:45:23Z

Use relative path to project directory for remote execution certificates/keys

605f8d11

by Valentin David at 2019-01-09T10:45:23Z

Add support for user remote execution configuration

Fixes #631.

65ea03ab

by Valentin David at 2019-01-09T10:45:23Z

Add support for https channel to remote execution and actions servers

Fixes #780.

ba05c4fa

by Valentin David at 2019-01-09T12:58:59Z

Merge branch 'valentindavid/remote_execution_configuration' into 'master'

Remote execution configuration: HTTPS and user configuration

Closes #780 and #631

See merge request BuildStream/buildstream!1030

eb0dbcfc

by Valentin David at 2019-01-09T13:56:58Z

buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY

ScriptElement does not use Sandbox.get_directory. It works using it
with remote execution.

Fixes #850

f7681925

by Valentin David at 2019-01-09T14:44:22Z

Merge branch 'valentindavid/script_virtual_directory' into 'master'

buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY

Closes #850

See merge request BuildStream/buildstream!1047

f87d1c93

by Jim MacArthur at 2019-01-09T15:01:18Z

sandboxremote.py: Rename 'instance_name' option to 'instance-name'

It was 'instance-name' in the documentation.

ed8bc5bc

by Jim MacArthur at 2019-01-09T15:30:28Z

Merge branch 'jmac/rename_instance_name' into 'master'

Rename 'instance_name' option to 'instance-name'

See merge request BuildStream/buildstream!1048

f29a0995

by Tristan Van Berkom at 2019-01-09T19:51:14Z

Support running test environments in parallel with `detox`

This patch namespaces the test temp directory and the output
coverage report file with the name of the environment under test,
such that separately run tests do not access the same files.

When running tests without tox, directly through setup.py,
then the tmp directory will still be `./tmp`.

  * .gitignore: Added new .coverage-reports/ directory

  * .gitlab-ci.yml: Rely on tox to combine and report coverage, only
    tell tox about the COVERAGE_PREFIX so that results can be namespaced
    by CI job name.

    This change also publishes the sources and final combined `.coverage`
    file in an output gitlab artifact for inspection, and lists some missing
    dependencies to the `coverage` job.

  * tox.ini: Add comments and refactor main [testenv] section so that
    other environments dont inherit too much unrelated cruft.

    Generate the coverate reports in the respective {envtmpdir} so that
    all per-process coverage files are prefixed with a full path, ensuring
    that concurrent runs don't mix reports and addressing concerns
    raised in #844.

    Also implemented new `tox -e coverage` environment to combine
    any found coverage and print a report.

  * .coveragerc: Omit .tox/ directory from coverage stats

Fixes issue #844

d364ad02

by Tristan Van Berkom at 2019-01-09T19:52:22Z

CONTRIBUTING.rst: Updated to mention new `coverage` tox environment.

Also point out that it is possible to run test environments in
parallel using the `detox` tool.

1e352434

by Tristan Van Berkom at 2019-01-09T20:36:06Z

Merge branch 'tristan/detox-tests' into 'master'

Allow using detox, and fix/refactor collection of coverage reports

Closes #844

See merge request BuildStream/buildstream!1051

116c1070

by Tristan Van Berkom at 2019-01-09T20:39:15Z

.coveragerc: Omit some things which are irrelevant to cover.

  * Omit versioneer's _version.py

  * Omit our __main__.py which is used only internally for generating
    documentation

c8f3616d

by Tristan Van Berkom at 2019-01-09T21:12:29Z

Merge branch 'tristan/refine-coverage' into 'master'

.coveragerc: Omit some things which are irrelevant to cover.

See merge request BuildStream/buildstream!1052

c87bb592

by Jürg Billeter at 2019-01-10T09:43:37Z

_platform/platform.py: Add canonicalize_arch() method

26e33346

by Jürg Billeter at 2019-01-10T12:50:15Z

_options/optionarch.py: Accept architecture aliases

Accept common architecture aliases for arch options instead of only
accepting the canonicalized, OS-independent architecture name. This
restores compatibility with existing projects and may simplify option
handling for projects that only target a single OS (and thus, do not
need OS-independent architecture names).

06deb4c4

by Jürg Billeter at 2019-01-10T12:50:15Z

element.py: Accept architecture aliases for sandbox config

Accept common architecture aliases for the sandbox config for
consistency with arch options.

0c2a66b3

by Jürg Billeter at 2019-01-10T13:06:02Z

tests/format/optionarch.py: Add tests for architecture aliases

f86bc760

by Jürg Billeter at 2019-01-10T13:34:56Z

Merge branch 'juerg/arch' into 'master'

Accept architecture aliases

See merge request BuildStream/buildstream!1034

347eb34e

by Angelos Evripiotis at 2019-01-10T14:38:23Z

contributing: fix 'oprtation' and some other typos

6bc27bc1

by Angelos Evripiotis at 2019-01-10T15:07:46Z

Merge branch 'aevri/contrib-typos' into 'master'

contributing: fix 'oprtation' and some other typos

See merge request BuildStream/buildstream!1055

0b83d024

by Tristan Van Berkom at 2019-01-10T20:02:50Z

_pipeline.py: Fix the planner to retain the original order when depth sorting

The algorithm adds elements to a dictionary and then sorts the dictionary
by the discovered depths while recursing - using an OrderedDict is enough
to ensure a stable order.

This fixes `bst show --deps plan ...` reporting different results on
each invocation.

This fixes an aspect of #824

630e26f1

by Tristan Van Berkom at 2019-01-10T20:03:52Z

testutils/runcli.py: Added Result.get_start_order()

Added a function to report the list of elements which appeared in
a given queue in the order of their first appearance.

d1d7de57

by Tristan Van Berkom at 2019-01-11T04:13:16Z

tests/frontend/order.py: Adding tests ensuring stable element processing order

For each sample project and expected result order, this test ensures that:

  * bst show --deps plan: Always shows the expected build order.

  * bst source fetch: Always executes in the expected build order.

  * bst build: Always builds in the expected order (disabled).

The build part of the test is currently disabled as the scheduler
seems to be messing with the order even in a `--builders 1` scenario.

276b8d48

by Tristan Van Berkom at 2019-01-11T04:42:24Z

Merge branch 'tristan/element-processing-order' into 'master'

Make build plan element list stable

See merge request BuildStream/buildstream!1058

fc3dcec8

by Angelos Evripiotis at 2019-01-11T16:45:00Z

dev-requirements: pytest-cov==2.6.1 for '--no-cov'

Bump the version of pytest-cov, so that we pick up the fix for
the '--no-cov' option breaking in pytest 4.0:

    https://github.com/pytest-dev/pytest-cov/pull/230

This was the error you would get prior to this version:

    _pytest.warning_types.RemovedInPytest4Warning: config.warn has been
    deprecated, use warnings.warn instead

Although it says 'deprecated', it did result in
termination with stack trace.

f86b7ff3

by Angelos Evripiotis at 2019-01-11T16:45:00Z

contributing: '--last-failed' and '--no-cov' tips

To help new folks get to grips with pytest, add some tips on other
frequently used features.

d983f231

by Angelos Evripiotis at 2019-01-11T17:19:06Z

Merge branch 'aevri/tox_no_cov' into 'master'

pytest-cov==2.6.1, contributing: '--last-failed' and '--no-cov' tips

See merge request BuildStream/buildstream!1059

829a2b93

by Tristan Van Berkom at 2019-01-12T21:32:13Z

_scheduler/queues/queue.py: Put elements in the skip list, not jobs

Minor correction, looks like we're not observing this queue
otherwise we'd be seeing crashes.

993e30ae

by Tristan Van Berkom at 2019-01-12T22:02:55Z

Merge branch 'tristan/fix-skipped-elements' into 'master'

_scheduler/queues/queue.py: Put elements in the skip list, not jobs

See merge request BuildStream/buildstream!1063

32732e01

by Javier Jardón at 2019-01-14T09:04:01Z

.gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25

4f5f1184

by Valentin David at 2019-01-14T09:40:56Z

Merge branch 'jjardon/freedesktop-sdk_latest' into 'master'

.gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25

Closes #858

See merge request BuildStream/buildstream!1064

bb80a2b8

by Chandan Singh at 2019-01-14T14:30:15Z

.gitlab-ci.yml: Use newer image for overning aarch tests

During recent reorganizing of `.gitlab-ci.yml` to work with `tox`, seems
like we missed to update the image used by the overnight aarch tests,
meaning that they currently fail due to `tox` being missing from them.

While these tests will be skipped on MRs usually, here is an example of
what the tests look like if they are actually run with this change:
https://gitlab.com/BuildStream/buildstream/-/jobs/145449561.

Fixes #859.

10b3ee62

by Chandan Singh at 2019-01-14T14:57:08Z

Merge branch 'chandan/fix-overnight-aarch' into 'master'

Chandan/fix overnight aarch

Closes #859

See merge request BuildStream/buildstream!1065

542cdaf0

by Jürg Billeter at 2019-01-14T18:46:57Z

_frontend/cli.py: Also check original args for --config in bash completion

The path of the config file generated by testutils for completion tests
is passed as regular argument, not via COMP_WORDS. Use that config file
in complete_artifact() to ensure the test uses the right artifact
directory.

ff666e76

by James Ennis at 2019-01-14T18:46:57Z

completions.py: Add a test for our artifact ref autocompletions

Since the artifact subcommand group has been created, we have been able
to tab complete both element names and artifact refs as arguments to
the artifact subcommands (e.g. bst artifact log), this commit adds a
test for this.

273b0f55

by Tristan Van Berkom at 2019-01-14T20:24:57Z

Merge branch 'jennis/add_artifacts_completion' into 'master'

completions.py: Add a test for our artifact ref autocompletions

See merge request BuildStream/buildstream!1054

2e3c2ea2

by Javier Jardón at 2019-01-14T22:26:04Z

.gitlab-ci.yml: Use latest bst-external

which include flatpak_repo plugin needed to build fdsdk

d60d2e31

by Javier Jardón at 2019-01-15T00:39:22Z

Merge branch 'jjardon/bst_external_0_9' into 'master'

.gitlab-ci.yml: Use latest bst-external

Closes #861

See merge request BuildStream/buildstream!1068

605836c1

by Chandan Singh at 2019-01-15T00:57:05Z

.gitlab-ci.yml: Add job to attempt to update dependencies

This will help us check if BuildStream is working with the latest
version of dependencies, as per our constraints. This job is allowed to
fail but its failure should signal that we need to add stricter
constraints in some of our `.in` requirements files.

006370af

by Chandan Singh at 2019-01-15T01:21:33Z

Merge branch 'chandan/reqs-update-test' into 'master'

.gitlab-ci.yml: Add job to attempt to update dependencies

See merge request BuildStream/buildstream!1038

4b544555

by Chandan Singh at 2019-01-15T21:28:40Z

.gitlab-ci.yml: Add tests for python 3.7

We already have tests for python 3.5 and 3.6 but not 3.7.

Fixes https://gitlab.com/BuildStream/buildstream/issues/838.

80fe0d9a

by Javier Jardón at 2019-01-15T22:48:22Z

Merge branch 'chandan/python37-tests' into 'master'

.gitlab-ci.yml: Add tests for python 3.7

Closes #838

See merge request BuildStream/buildstream!1074

c91784ab

by Chandan Singh at 2019-01-15T22:49:01Z

conftest.py: Don't use deprecated get_marker() function

Starting from `pytest` version 4.1.0, `Node.get_marker()` has been
removed, and hence our tests break when running with newer versions of
`pytest`. It was deprecated since a while back but it has recently been
removed completely. Use `get_closest_marker()` as a replacement that is
suggested in the changelog, and seems to work fine for our use case.

See https://github.com/pytest-dev/pytest/pull/4564 for more context on
the upstream issue.

One way of verifying this change is that this should fix the recently
added `tests-fedora-update-deps` job, that was failing before due to
this issue.

ecae4d73

by Javier Jardón at 2019-01-15T23:23:41Z

Merge branch 'chandan/fix-pytest-get-marker' into 'master'

conftest.py: Don't use deprecated get_marker() function

See merge request BuildStream/buildstream!1073

0eac4008

by Valentin David at 2019-01-16T10:04:57Z

buildstream/_gitsourcebase.py: Reduce git history for git describe.

Found during #833.

`git rev-list --boundary tag..HEAD` unfortunately gives boundaries
that are deeper than should when there is a merge commit between `tag`
and `HEAD`. The common ancestory of the two parents of the merge is
a boundary instead of the parent of the branch that is not traversed.

`--ancestry-path` fixes this issue by restricting `git` traversing
those branches.

a405e08f

by Valentin David at 2019-01-16T10:04:57Z

buildstream/_gitsourcebase.py: Fix case where HEAD is tagged

`git rev-list --boundary HEAD..HEAD` does not return any boundary.
So in this case we need to manually tag the HEAD as a boundary.

c0631d48

by Valentin David at 2019-01-16T10:33:52Z

Merge branch 'valentindavid/git-reduced-history' into 'master'

buildstream/_gitsourcebase.py: Reduce git history for git describe.

See merge request BuildStream/buildstream!1069

b608ac86

by James Ennis at 2019-01-16T11:05:44Z

_context.py: Doc fix: get_toplevel_project() returns object not list

97a3beb6

by James Ennis at 2019-01-16T11:05:44Z

_context.py: Add documentation to get_workspaces() command

d6587aa0

by James Ennis at 2019-01-16T11:33:21Z

Merge branch 'jennis/doc_fixes_in_context' into 'master'

Small documentation/comment fixes in context.py

See merge request BuildStream/buildstream!1072

2683f98a

by Raoul Hidalgo Charman at 2019-01-16T11:55:07Z

_cas: Rename artifactcache folder and move that to a root module

Other components will start to reply on cas modules, and not the artifact cache
modules so it should be organized to reflect this.

All relevant imports have been changed.

Part #802

d2cc4798

by Raoul Hidalgo Charman at 2019-01-16T11:55:07Z

casremote.py: Move remote CAS classes into its own file

Part of #802

76f67483

by Raoul Hidalgo Charman at 2019-01-16T11:55:07Z

cas: move remote only functions to CASRemote

List of methods moved
* Initialization check: made it a class method that is run in a subprocess, for
  when checking in the main buildstream process.
* fetch_blobs
* send_blobs
* verify_digest_on_remote
* push_method

Part of #802

6c428bc9

by Jürg Billeter at 2019-01-16T12:56:38Z

Merge branch 'raoul/cas-refactor' into 'master'

Cas refactor

See merge request BuildStream/buildstream!1071

2648b0bd
by Angelos Evripiotis at 2019-01-16T14:44:25Z
```
bst-docker-import: fix Chandan's name
```

780d0f3a

by Angelos Evripiotis at 2019-01-16T15:13:06Z

Merge branch 'chadnan' into 'master'

bst-docker-import: fix Chandan's name

See merge request BuildStream/buildstream!1076

4ca37dcd

by Tristan Van Berkom at 2019-01-16T15:47:21Z

source-determinism.py integration test: Dont use integration_cache fixture

This was just deadcode, it's not really used here.

6286d820

by Tristan Van Berkom at 2019-01-16T15:47:21Z

conftest.py: Use different artifact directory for integration tests

To ensure we can run integration tests in parallel, use a tempdir
for the artifact cache of each separate integration test run.

This makes it possible to run multiple full test runs including
integration tests in parallel under detox.

471af316

by Tristan Van Berkom at 2019-01-16T16:32:52Z

Merge branch 'tristan/detoxing-integration-tests' into 'master'

Make integration tests parallelizable

See merge request BuildStream/buildstream!1077

081e8426

by James Ennis at 2019-01-16T17:58:41Z

setup.py: Import server main from _cas/casserver.py

Since the CAS refactor, we have not been able to execute
bst-artifact-server. This commit ensures that we can.

This fix closes #867

e385660c

by James Ennis at 2019-01-16T18:33:20Z

Merge branch 'jennis/fix_bst_artifact_server' into 'master'

Ensure that we can execute `bst-artifact-server`

Closes #867

See merge request BuildStream/buildstream!1079

6991d6d6

by Tristan Van Berkom at 2019-01-16T19:28:14Z

_scheduler: Refactor of queues and resources.

This branch makes the following changes:

  * jobs/job.py: No longer stores any interested resource list

    Jobs are ephemeral again, they only ever exist while they
    are running.

  * queues/queue.py: Revert to only handling lists of elements

    Elements pass through the queues, Queue.harvest_jobs()
    replaces Queue.pop_ready_jobs() and now the Queue stops
    creating jobs as soon as there are not enough resources
    for the job.

    Also removed unused `prepare()` abstract method.

  * queues/buildqueue.py: Adapt the part where we launch a job

    This part needs to be reworked anyway, just touch it up for
    now so that it doesnt break with the surrounding changes.

  * jobs/{cachesize,cleanup}job.py: Expose uniform complete callback

    Allows the scheduler to manage resource deallocation for these
    two job completions as a custom thing, at the same phase
    that the Queues take care of their own resource deallocation.

  * resources.py: No longer has knowledge of the job

    Since jobs are ephemeral, they are not a suitable place
    to store the resource identifiers, these must be provided
    by the callers wherever needed.

    Now the main Resources object is owned by the Scheduler
    but shared with Queues, each take care of managing the
    resources of the jobs they create through the same
    resource API.

  * scheduler.py: Reverted to only creating jobs on demand

    This changes the flow of the scheduler such that whenever
    jobs complete, the queues are interrogated for as many
    jobs which can run at the moment but not more; and this
    completely removes the waiting list.

    For the internal cache management jobs, we handle this
    with a little state instead of having a waiting list
    and only launch when the resources permit it.

By abolishing the scheduler waiting list and creating jobs
on demand, we fix the order of element processing and consequently
fix issue #712.

04ba59bc

by Tristan Van Berkom at 2019-01-16T19:28:14Z

tests/frontend/order.py: Enable the test for build and fix the fetch tests

With the scheduler changes, fetch jobs get automatically skipped
so the output is changed, using a separate repo for each element
fixes the test such that every fetch job gets a job launched.

32bdded8

by Tristan Van Berkom at 2019-01-16T19:28:14Z

_artifactcache/artifactcache.py: Rephrase failure message

It was saying "There is not enough space to build the given element.",
this makes me think the error is associated to a specific element, but
this does not make sense to show up in a cleanup task.

Instead say "There is not enough space to complete the build.", which
should be more clear that even after cleaning up there is not enough
space.

17f6e5a8

by Tristan Van Berkom at 2019-01-16T20:10:30Z

Merge branch 'tristan/element-processing-order' into 'master'

Scheduler refactor, fix processing order

Closes #712

See merge request BuildStream/buildstream!1067

19adc9e5

by Tristan Van Berkom at 2019-01-16T20:48:10Z

element.py: Cleanup temporary staging directories on termination

Use utils._tempdir() which in turn uses _signals.terminator() for
this purpose.

This also changes utils._tempdir() to use utils._force_rmtree()
so that it is safe to use for cleaning up the staging directory.

This patch fixes orphaned temporary directories being left behind
in the artifact cache directory at forceful termination time.

5e9598c5

by Tristan Van Berkom at 2019-01-16T21:17:26Z

Merge branch 'tristan/cleanup-staging-tempdir' into 'master'

element.py: Cleanup temporary staging directories on termination

See merge request BuildStream/buildstream!1080

5ea2f59d
by Tristan Van Berkom at 2019-01-16T23:35:21Z
```
tests: Migrate completions test to tests/frontend
```

1eafc25c