-
adedd867
by Tristan Maat
at 2018-07-18T05:11:03Z
Bump required python version to 3.5
-
b8543f8a
by Tristan Maat
at 2018-07-18T05:11:03Z
_exceptions.py: Add `detail` to ArtifactErrors
-
fc79ad2c
by Tristan Maat
at 2018-07-18T05:11:03Z
utils.py: Allow `list_relative_paths` to list directories
-
4f9da15d
by Tristan Maat
at 2018-07-18T05:11:03Z
_scheduler/*queue.py: Move queues to a subdirectory
-
24925634
by Tristan Maat
at 2018-07-18T05:45:59Z
Make Jobs abstract and element-independent
-
33984448
by Tristan Maat
at 2018-07-18T05:45:59Z
Add cache_quota to user config
-
1ec5c7b1
by Tristan Maat
at 2018-07-18T05:45:59Z
Make elements keep track of their built artifact size
-
7229d2e5
by Tristan Maat
at 2018-07-18T05:45:59Z
Compute the artifact cache size after each build/pull
-
dc17de38
by Tristan Maat
at 2018-07-18T06:07:24Z
Clean the artifact cache when we hit the cache quota
When the cache quota is hit, we will remove any elements not required
for the current build, until our cache is only filled halfway.
-
6ff12e5e
by Tristan Maat
at 2018-07-18T06:07:24Z
tests/artifactcache/expiry.py: Add expiry tests
-
308d59be
by Tristan Maat
at 2018-07-18T06:07:24Z
NEWS: Add news about cache expiry
-
a950a985
by Tristan Van Berkom
at 2018-07-18T09:59:03Z
_context.py: Adding apis for message recording and task logging.
Added the new Context.recorded_messages() context manager, this
causes messages to be logged in a dedicated log file before being
propagated to the log handler, and also provides an open log handle
for the capturing and logging of stdout/stderr from child processes.
This comes with the additional accessors:
Context.get_log_handle()
Context.get_log_filename()
-
d835c37f
by Tristan Van Berkom
at 2018-07-18T09:59:03Z
Refactor: Use new logging mechanism from Context
o element.py / plugin.py: Removed supporting logging code, and derive
the log handle from Context.
o _scheduler/scheduler.py, _scheduler/queues/queue.py: Adapt to new Job initializer API for the logfile
o _scheduler/jobs/job.py: Run job activities within the new context manager
which turns on logging for a job. Also removed a lot
of custom logging abstract methods which are unneeded.
o _scheduler/jobs: Job implementations need not implement so much custom logging.
-
26b9f6fa
by Tristan Van Berkom
at 2018-07-18T11:52:43Z
_stream.py: Clarifying ambiguous FIXME comment about required artifacts
-
3814905a
by Daniel Playle
at 2018-07-18T12:08:24Z
Reorder app.py imports
When testing locally on some systems, it appears that pylint fails on
the order of imports for `buildstream/_frontend/app.py`. This commit
changes the order of these imports.
This commit addresses issue !453.
-
48f66a3c
by Chandan Singh
at 2018-07-18T18:03:31Z
.gitlab-ci.yml: Use testsuite images for running tests
Instead of re-using the user-facing `buildstream/buildstream-fedora`
image for tests (that comes with BuildStream pre-installed), use
`buildstream/testsuite-fedora` image that is supposed to be used
exclusively for these tests.
Similarly, use the dedicated `buildstream/testsuite-debian` image
instead of `buildstream/buildstream-debian`.
Fixes #458.
For related discussion, see
https://gitlab.com/BuildStream/buildstream-docker-images/issues/8.
-
1016dcb4
by Josh Smith
at 2018-07-19T12:52:21Z
scheduler.py: Correct some anomalies within the docstrings
-
709b9bca
by Francisco Redondo Marchena
at 2018-07-19T14:03:12Z
source-bundle: Enable --except option
Before this option was listed in help but was ignored when
creating the source-bundle.
Issue #468
-
96d07153
by Jordan Petridis
at 2018-07-19T23:54:39Z
Docs: Update the required build packages for fedora based systems.
`ruamel.yaml` seems to require `Python.h` header file to build.
`python3-devel` is what provides it for Fedora.
-
93a77c1d
by Javier Jardón
at 2018-07-19T23:58:28Z
Docs: Update the required build packages for the rest of the systems
Also remove the recomendation to install psutil as we need to build
other python modules anyway (like ruamel)
Completes 96d07153b7817cdaeda57dd163eed52b2b1b31e8
-
57e0d1c5
by William Salmon
at 2018-07-20T00:03:06Z
Added dependency to the Docs
These dependencies are needed to use the plugins that are installed
as part of this guild.
As the dependencies added are not dependencies of the core package
they have been added to a separate section.
The lzip package is needed to build gnome.
https://wiki.gnome.org/Newcomers/BuildSystemComponent
-
1b6688eb
by Javier Jardón
at 2018-07-20T00:07:23Z
install_linux_distro.rst: document plugins installed by default
-
fc63b972
by Javier Jardón
at 2018-07-20T00:12:50Z
install_linux_distro.rst: Document plugins deps for the rest of the systems
-
790fb40b
by Tiago Gomes
at 2018-07-20T09:07:01Z
Update checkout man page for new --deps option
-
9c1f024b
by Tiago Gomes
at 2018-07-20T09:07:01Z
Add support for creating a tarball on bst checkout
One of the tests added is configured to be skipped for now, as dumping
binary data is causing a bad descriptor exception when using the pytest
capture module.
Closes #263.
-
0ddbf0e0
by Tristan Van Berkom
at 2018-07-20T11:52:00Z
setup.py: Specify minimum required version of pytest-cov plugin
This causes the new artifact tests to pass (unless you happened
to already have a recent enough version of pytest-cov, in which case
you didn't notice the breakage).
-
6158ee5a
by Sam Thursfield
at 2018-07-20T13:17:14Z
Fix crash when --debug is passed
I hit the following backtrace running `bst --debug push` and `bst
--debug build`:
pid:16736 id:000[--:--:--][][] START Push
pid:16736 id:000[--:--:--][][] START Loading pipeline
pid:16736 id:000[00:00:00][][] SUCCESS Loading pipeline
pid:16736 id:000[--:--:--][][] START Resolving pipeline
pid:16736 id:000[--:--:--][][] BUG 'MesonElement' object has no attribute '_Element__cache_key'
Traceback (most recent call last):
File "/home/sam/.local/bin/bst", line 8, in <module>
sys.exit(cli())
File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/cli.py", line 162, in override_main
standalone_mode=standalone_mode, **extra)
File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/sam/.local/lib/python3.6/site-packages/click/decorators.py", line 27, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/cli.py", line 471, in push
app.stream.push(elements, selection=deps, remote=remote)
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_stream.py", line 341, in push
fetch_subprojects=True)
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_stream.py", line 814, in _load
fetch_subprojects=fetch_subprojects)
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_pipeline.py", line 119, in load
for meta in meta_elements
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_pipeline.py", line 119, in <listcomp>
for meta in meta_elements
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 894, in _new_from_meta
element = project.create_element(artifacts, meta)
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_project.py", line 184, in create_element
element = self._element_factory.create(self._context, self, artifacts, meta)
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_elementfactory.py", line 57, in create
return element_type(context, project, artifacts, meta, default_config)
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 196, in __init__
super().__init__(meta.name, context, project, meta.provenance, "element")
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 171, in __init__
self.debug("Created: {}".format(self))
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 407, in debug
self.__message(MessageType.DEBUG, brief, detail=detail)
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 652, in __message
self.__context.message(message)
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_context.py", line 338, in message
self._message_handler(message, context=self)
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/app.py", line 665, in _message_handler
text = self.logger.render(message)
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 585, in render
return self._render(message)
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 620, in _render
text += widget.render(message)
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 228, in render
_, key, missing = plugin._get_display_key()
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 1138, in _get_display_key
cache_key = self._get_cache_key()
File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 991, in _get_cache_key
return self.__cache_key
AttributeError: 'MesonElement' object has no attribute '_Element__cache_key'
The issue here is that when `--debug` is enabled, Plugin.__init__()
tries to log a message like this:
pid:16929 id:001[--:--:--][????????][ main:tracker.bst ] DEBUG Created: meson element at tracker.bst [line 1 column 0]
The log formatter is trying to get the cache key of the element in
order to show it in the log, but the attribute hasn't yet been set
because we've not got to the Element() constructor.
-
f484ab4f
by Phillip Smyth
at 2018-07-20T14:29:44Z
Updating .bzr plugin to omit the .bzr dir
Fixes issue #455
-
b662f175
by Tristan Van Berkom
at 2018-07-20T14:29:44Z
git source plugin: Omit the .git directory completely.
This will break builds which use `git describe` to determine
their version number until a more elegant solution is implemented,
and will unblock work on caching of build trees in the artifacts.
Fixes issue #455
-
80effb5c
by Tristan Van Berkom
at 2018-07-22T08:30:40Z
HACKING.rst: No longer requiring issue numbers in commit messages.
Instead, we are enabling the merge commit feature in gitlab and
hopefully reducing some of the friction in review this way.
-
c7124c3d
by Tristan Van Berkom
at 2018-07-22T09:46:05Z
HACKING.rst: Added convention for naming of branches in merge requests.
-
282b1092
by Tristan Van Berkom
at 2018-07-23T08:47:36Z
plugins/sources/git.py: Use --shared instead of --hardlinks
This improves staging performance by avoiding making any
copies of the objects in the cloned repository in the staging
area before removing the .git directory.
Fixes #488
-
fe0df616
by Tristan Van Berkom
at 2018-07-23T09:20:44Z
Merge branch 'tristan/git-stage-with-shared-clone' into 'master'
plugins/sources/git.py: Use --shared instead of --hardlinks
Closes #488
See merge request BuildStream/buildstream!557
-
6559db4c
by Josh Smith
at 2018-07-23T09:22:39Z
element.py: Make Element._get_brief_display_key() available to Internal API
This change will allow for more detailed info messages throughout
the CASCache.
-
6b82e476
by Josh Smith
at 2018-07-23T09:22:39Z
cascache.py: Add push info detailing which remote is being pushed to
Work towards fixing issue #275
-
1d36df25
by Josh Smith
at 2018-07-23T09:22:39Z
cascache.py: Add pull info detailing which remote is being pulled from
Work towards fixing issue #275
-
364706f3
by Tristan Van Berkom
at 2018-07-23T09:57:02Z
Merge branch 'Qinusty/275' into 'master'
Indicate where artifacts are going to and coming from in the log
Closes #275
See merge request BuildStream/buildstream!553
-
e70237a5
by Phillip Smyth
at 2018-07-23T13:26:47Z
Adding caching build trees
buildstream/element.py: Adding build tree to cache
buildstream/_versions.py: Bumping BST_CORE_ARTIFACT_VERSION
-
335afb5b
by Tristan Van Berkom
at 2018-07-23T14:07:07Z
Merge branch 'caching_build_trees_limited' into 'master'
Adding caching build trees (limited)
See merge request BuildStream/buildstream!560
-
60290f31
by Javier Jardón
at 2018-07-23T15:29:37Z
doc/source/install_linux_distro.rst: buildstream doesn't depend on ostree or pygobject anymore
-
de162e43
by Javier Jardón
at 2018-07-24T11:58:16Z
Merge branch 'jjardon/ostree_repo' into 'master'
doc/source/install_linux_distro.rst: buildstream doesn't hard depend on ostree or pygobject anymore
Closes #492
See merge request BuildStream/buildstream!558
-
6106d657
by Josh Smith
at 2018-07-25T10:50:14Z
cascache.py: Remove misleading info message
This addresses issue #501.
-
4a637d63
by Jonathan Maw
at 2018-07-25T12:05:19Z
Merge branch 'Qinusty/501' into 'master'
Remove misleading info message
See merge request BuildStream/buildstream!567
-
a0ad985f
by Phil Dawson
at 2018-07-25T13:03:23Z
tests/frontend/show.py: Add test case for maximum recursion depth being exceeded
Add test to ensure gracefull handling of exception thrown while loading
a pipeline due the python's max recursion depth being exceeded.
This is part of the work for issue #203
-
95b4eae4
by Phil Dawson
at 2018-07-25T13:03:23Z
app.py: Handle exception thrown when recursion limit is exceeded
Because the RecursionError exception was introduced in Python 3.5, until we
drop support for for Python 3.4, we must use RuntimeError.
-
889bf238
by Jim MacArthur
at 2018-07-25T13:43:21Z
Merge branch 'phil/203-BuildStream-crashes-when-dependency-tree-too-deep' into 'master'
Phil/203 BuildStream crashes when dependency tree too deep
See merge request BuildStream/buildstream!512
-
54fee6c8
by Phillip Smyth
at 2018-07-25T14:15:36Z
_stream.py: Added functionality for workspace open -f
tests/frontend/workspace.py: Added tests
-
f62b6cb7
by Jonathan Maw
at 2018-07-25T15:01:33Z
Merge branch 'bst_workspace_open_force_does_nothing' into 'master'
_stream.py: Added functionality for workspace open -f
See merge request BuildStream/buildstream!549
-
c049d021
by Ed Baunton
at 2018-07-25T19:43:51Z
Add remote source plugin
Add a plugin that supports downloading files verbatim from a source with
an optional overridable filename and destination directory. Bumps bst
format version to 10.
Fixes #163
-
bd1196ef
by Ed Baunton
at 2018-07-26T13:22:52Z
Provide explicit reason for SourceError exception on path
When the user provides a path for the filename parameter, provide a reason
-
9a46c16f
by Ed Baunton
at 2018-07-26T14:06:52Z
Merge branch 'edbaunton/remote-source' into 'master'
Add remote source plugin
Closes #163
See merge request BuildStream/buildstream!541
-
7ceacfea
by William Salmon
at 2018-07-26T14:32:49Z
Documentation typos and fixes
Daniel went through the Doc's etc and made a list of suggestions.
These are the result of the most easily patched suggestions and
one or two thing that I noticed as well.
-
8c9fd9e4
by Javier Jardón
at 2018-07-26T15:32:16Z
Merge branch 'willsalmon/documentation_form_notes' into 'master'
Documentation typos and fixes
See merge request BuildStream/buildstream!569
-
9ebd7fe1
by Josh Smith
at 2018-07-27T04:41:26Z
_exceptions.py: Modify BstError API to allow optional retry
job.py: Changes to the logic surrounding retry attempts and child process return codes
element.py, source.py: ElementError and SourceError also implement this change.
These exceptions now have an optional parameter of temporary which defaults to false. This will potentially break
backwards compatibility where exceptions were previously raised and a retry was intended.
To trigger a retry, one must now raise their SourceError or ElementError with temporary=True.
This aims to fix #397.
-
ca224425
by Josh Smith
at 2018-07-27T04:41:26Z
_downloadablefilesource.py: Update network errors to be temporary
This follows the change in 67ecd97a05279a3b7570ad59f05bf0a5973ef04c.
-
0a7cfcdf
by Josh Smith
at 2018-07-27T04:41:26Z
tests/sources: deb, tar and zip all test for retry behaviour
This is following 5d6418a06a3a0262df5f6b9e6e1a36578515fd4a where
DownloadableFileSource was modified to raise 'temporary'
errors when relating to file downloading.
-
9b2bc78f
by Josh Smith
at 2018-07-27T04:41:26Z
cascache.py: Update cache push/pull errors to be temporary
Further work needs to be done for the current grpc exceptions which are reraised.
-
4986527f
by Josh Smith
at 2018-07-27T04:41:26Z
plugin.py: Extend Plugin.call() API
Plugin.call() now takes fail_temporarily as an optional parameter,
when supplied it will cause subsequent failures to trigger temporary
errors as opposed to permanent errors.
This also extends Plugin.check_output() which makes use of Plugin.call()
-
a9a1048e
by Josh Smith
at 2018-07-27T04:41:26Z
git.py: Update git plugin to fail temporarily on clone failures
The git plugin will now make use of the fail_temporarily parameter
to Plugin.call(), allowing failures to trigger a retry.
-
aa071cb6
by Tristan Van Berkom
at 2018-07-27T06:13:53Z
Merge branch 'tristan/temporary-failures' into 'master'
Temorary failures
Closes #397
See merge request BuildStream/buildstream!574
-
4d86e7c5
by Tristan Van Berkom
at 2018-07-27T06:43:48Z
element.py, source.py, plugin.py: Adjusting API version since annotations.
Since we backported the temporary/permanent failures patch for #397
into the `bst-1.2` branch, we need to adjust the since versions in master
down to 1.2.
-
fb222ba1
by Josh Smith
at 2018-07-27T08:50:45Z
cascache.py: Raise ArtifactError on grpc error
This aims to fix issue #502.
-
48916b8a
by Tristan Maat
at 2018-07-27T10:22:41Z
Merge branch 'Qinusty/502' into 'master'
cascache.py: Raise ArtifactError on grpc error
Closes #502
See merge request BuildStream/buildstream!576
-
23e080b9
by James Ennis
at 2018-07-27T11:13:50Z
README.md: Minor corrections
-
196cfffc
by James Ennis
at 2018-07-27T11:13:50Z
README.md: Direct users towards the tutorial before existing projects:
This patch also removes displaying the projects that live within
the buildstream-examples repository, as we wish to deprecate this
repo and migrate the two examples.
- #438 and #439
In addition to this, both examples do not 'build' in their current
state.
-
2b93574e
by James Ennis
at 2018-07-27T11:13:50Z
main_install.rst: Add note to refer Windows and macOS users to docker
-
63e2320e
by Josh Smith
at 2018-07-27T11:16:55Z
.gitignore: Ignore .pytest_cache/
-
6ea97b17
by Javier Jardón
at 2018-07-27T11:54:20Z
Merge branch 'jennis/doc_improvements' into 'master'
Refer readers to our tutorial before referring them to existing bst projects
See merge request BuildStream/buildstream!578
-
8b46e874
by Jonathan Maw
at 2018-07-27T12:24:56Z
project: Parse and store mirrors
-
a2e9c62a
by Jonathan Maw
at 2018-07-27T12:24:56Z
Move _ALIAS_SEPARATOR into utils
The separator is useful in source files other than _project.py
-
7c993ac0
by Jonathan Maw
at 2018-07-27T12:24:56Z
source: Store the url aliases or use an override
This is part of a later plan to implement mirroring without forcing
everyone to update their sources. We use the expected calls to
Source.translate_url() when running Source.configure() to extract the
aliases from the URL. Multiple aliases must be extracted because
sources exist that may fetch from multiple aliases (for example, git
submodules)
Later, we want to substitute another URI where the alias normally reads
from the project - We accomplish this by re-instantiating the Source
with the alias overrides passed as an argument to the constructor.
-
2889003c
by Jonathan Maw
at 2018-07-27T12:24:56Z
source: When fetching, try to fetch from mirrors first
**KLUDGE WARNING**: This involves making the source store its "meta"
object so that it's possible to create a copy of the source inside the
fetch queue, instead of back when the pipeline was being loaded.
This adds the SourceFetcher class, which is intended for sources that
fetch from multiple URLs (e.g. the git source and its submodules)
Fix when fetching
-
f81e8e7b
by Jonathan Maw
at 2018-07-27T12:24:56Z
Set default mirror via command-line or user config
In user config (buildstream.conf), it is set with the "default-mirror"
field.
On the command-line, it is set with "--default-mirror"
-
755ed898
by Jonathan Maw
at 2018-07-27T12:24:56Z
Add mirrored source tracking
-
19c01a56
by Jonathan Maw
at 2018-07-27T12:24:56Z
bzr.py: Improve mirror support
This fixes:
* Bzr repositories pulling from the branch they were created with.
* Bzr's _ensure_mirror() not actually checking that it successfully
mirrored the ref.
-
84872141
by Jonathan Maw
at 2018-07-27T12:24:56Z
git.py: Improve mirror support
When implementing fetching from mirrors, I encountered some problems
with the git source:
1. The mirror URL was using translate_url()'s output, so if a different
alias was used, then fetching from the mirror would go to a different
directory, and be inaccessible.
2. After fixing that, fetching was unable to pull from a URL other than
the one used at repository creation, meaning it wouldn't actually
pull from the mirror.
-
bd51a0b2
by Jonathan Maw
at 2018-07-27T12:24:56Z
testutils: Add a helper to copy a testutils repo
This is helpful if you want to test what happens when you have one repo
that has diverged from another. By copying the repo you're sure they
start with shared history.
This is especially useful when mirroring.
-
909120ab
by Jonathan Maw
at 2018-07-27T12:24:56Z
tests: Add mirrored fetching and tracking tests
-
1cbc2e17
by Jonathan Maw
at 2018-07-27T12:24:56Z
doc: Add documentation for source mirroring
-
202d9d26
by Jonathan Maw
at 2018-07-27T12:24:56Z
NEWS: Add item for source mirroring
-
2b23898d
by Jonathan Maw
at 2018-07-27T12:24:56Z
doc: Add tutorials for setting up git and tar mirrors
-
bd4d0355
by Jonathan Maw
at 2018-07-27T13:08:00Z
Merge branch '328-support-for-downloading-sources-from-mirrors' into 'master'
Resolve "Support for downloading sources from mirrors"
Closes #328
See merge request BuildStream/buildstream!404
-
cb4693b2
by Josh Smith
at 2018-07-27T14:00:26Z
Merge branch 'Qinusty/pytest_cache_gitignore'
-
8a96679a
by Josh Smith
at 2018-07-27T14:01:07Z
Merge branch 'master' of gitlab.com:BuildStream/buildstream
-
57e20a02
by Phillip Smyth
at 2018-07-27T14:52:12Z
Implementing relative workspaces