[Notes] [Git][BuildStream/buildstream][tpollard/483] 14 commits: Fix ostree repository mirroring

[Tom Pollard <gitlab mg gitlab com>]

Title: GitLab

Tom Pollard pushed to branch tpollard/483 at BuildStream / buildstream

Commits:

• c0431f15
  by Valentin David at 2018-08-16T07:43:07Z

  Fix ostree repository mirroring
  
  Ostree mirrors were not sharing the same local repository, so it was
  impossible the request refs from the right local repository when data
  was fetched from a mirror rather than upstream.
  
  Instead of having several repository with one remote each, we now
  have one repository with several remotes.
  
  This fixes #538.
  

• 04f83679
  by Tristan Van Berkom at 2018-08-16T08:54:27Z

  Merge branch 'valentindavid/fallback_mirror_ostree' into 'master'
  
  Fix ostree repository mirroring
  
  Closes #538
  
  See merge request BuildStream/buildstream!658

• 177c4264
  by William Salmon at 2018-08-16T09:52:38Z

  Trying to mitigate a mtime granularity braking the cache tests
  
  This patch mitigates the granularity of the mtimes used on the default
  gitlab runners, Allowing the test suite to pass on these runners.
  

• 01c4ac57
  by Tom Pollard at 2018-08-16T11:01:45Z

  Merge branch 'willsalmon/CacheExpiryTest' into 'master'
  
  Trying to mitigate a file system issue
  
  See merge request BuildStream/buildstream!595

• 0066d701
  by Tiago Gomes at 2018-08-16T12:33:01Z

  Use http instead of https for the gnu ftp mirror
  
  https seems broken on the Debian image:
  
  START   autotools/hello/42930621-fetch.499.log
  START   Fetching https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz
  FAILURE Fetching https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz
  FAILURE tar source at hello.bst [line 16 column 2]: Error mirroring https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz: <urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:720)>
  

• 898aa8a0
  by Tiago Gomes at 2018-08-16T13:43:06Z

  Merge branch 'tiagogomes/disable-https-gnu-repo' into 'master'
  
  Use http instead of https for the gnu ftp mirror
  
  See merge request BuildStream/buildstream!676

• b62c361c
  by Tiago Gomes at 2018-08-16T13:54:54Z

  cascache: use errno module
  
  os.errno does no longer work with Python 3.7
  
  Closes #577.
  

• 1e3e2a93
  by Tiago Gomes at 2018-08-16T15:00:12Z

  Merge branch 'tiagogomes/issue-577' into 'master'
  
  cascache: use errno module
  
  Closes #577
  
  See merge request BuildStream/buildstream!674

• 10f27f3f
  by Tristan Van Berkom at 2018-08-17T06:20:40Z

  _frontend/app.py: Notify session completions
  
  Use the optionally implemented desktop notification method
  to notify when a session completes.
  
  Previously, this used to only notify when an interactive prompt
  appears.
  
  This is an attempt to fix #385
  

• c102b92f
  by Tristan Van Berkom at 2018-08-17T08:52:07Z

  Merge branch 'tristan/notifications' into 'master'
  
  _frontend/app.py: Notify session completions
  
  Closes #385
  
  See merge request BuildStream/buildstream!672

• 9b6fac5a
  by William Salmon at 2018-08-17T09:48:01Z

  Add Error to git and ostree sources configure
  
  Raise a error at configure time if the track and ref properties are
  not present in the sources.
  
  This is to address https://gitlab.com/BuildStream/buildstream/issues/471
  that documented unhelpful behaviour when tracking git sources. However
  the issue was also identified in ostree.
  

• d5ed4f45
  by William Salmon at 2018-08-17T09:49:23Z

  Improve error message for build if there are refs missing
  

• c43af4eb
  by Will Salmon at 2018-08-17T10:51:38Z

  Merge branch 'willsalmon/APIFix' into 'master'
  
  Updating the no track or ref fix to match the buildstream style
  
  See merge request BuildStream/buildstream!628

• 234072de
  by Tom Pollard at 2018-08-17T12:10:49Z

  plugins/git.py: Warn if ref is not in given track
  
  Add a helper function assert_ref_in_track to git.py GitMirror()
  which is used when staging & initing the source workspace. It
  checks the element's ref exists in the track (branch/tag) if it
  has been specified, raising a warning if necessary. The warning makes
  use of the warning token 'REF_NOT_IN_TRACK' from the configurable
  CoreWarnings. If the element has been tracked with bst, it is assumed
  that the value of ref exists in the track as it was generated from it
  & as such is not asserted.
  

16 changed files:

• buildstream/_artifactcache/cascache.py
• buildstream/_frontend/app.py
• buildstream/_pipeline.py
• buildstream/plugins/sources/git.py
• buildstream/plugins/sources/ostree.py
• doc/examples/autotools/project.conf
• tests/artifactcache/expiry.py
• tests/frontend/mirror.py
• tests/frontend/push.py
• tests/frontend/workspace.py
• tests/sources/git.py
• + tests/sources/ostree.py
• + tests/sources/ostree/template/project.conf
• + tests/sources/ostree/template/repofiles/file
• tests/testutils/__init__.py
• + tests/testutils/runner_integration.py

Changes:

buildstream/_artifactcache/cascache.py

---

...	...	@@ -25,6 +25,7 @@ import signal
25	25	import stat
26	26	import tempfile
27	27	import uuid
	28	+import errno
28	29	from urllib.parse import urlparse
29	30
30	31	import grpc
...	...	@@ -82,7 +83,8 @@ class CASCache(ArtifactCache):
82	83
83	84	tree = self.resolve_ref(ref, update_mtime=True)
84	85
85		- dest = os.path.join(self.extractdir, element._get_project().name, element.normal_name, tree.hash)
	86	+ dest = os.path.join(self.extractdir, element._get_project().name,
	87	+ element.normal_name, tree.hash)
86	88	if os.path.isdir(dest):
87	89	# artifact has already been extracted
88	90	return dest
...	...	@@ -100,7 +102,7 @@ class CASCache(ArtifactCache):
100	102	#
101	103	# If rename fails with these errors, another process beat
102	104	# us to it so just ignore.
103		- if e.errno not in [os.errno.ENOTEMPTY, os.errno.EEXIST]:
	105	+ if e.errno not in [errno.ENOTEMPTY, errno.EEXIST]:
104	106	raise ArtifactError("Failed to extract artifact for ref '{}': {}"
105	107	.format(ref, e)) from e
106	108

buildstream/_frontend/app.py

---

...	...	@@ -269,6 +269,9 @@ class App():
269	269	else:
270	270	self._message(MessageType.FAIL, session_name, elapsed=elapsed)
271	271
	272	+ # Notify session failure
	273	+ self._notify("{} failed".format(session_name), "{}".format(e))
	274	+
272	275	if self._started:
273	276	self._print_summary()
274	277
...	...	@@ -286,6 +289,9 @@ class App():
286	289	if self._started:
287	290	self._print_summary()
288	291
	292	+ # Notify session success
	293	+ self._notify("{} succeeded".format(session_name), "")
	294	+
289	295	# init_project()
290	296	#
291	297	# Initialize a new BuildStream project, either with the explicitly passed options,
...	...	@@ -419,6 +425,12 @@ class App():
419	425	# Local Functions #
420	426	############################################################
421	427
	428	+ # Local function for calling the notify() virtual method
	429	+ #
	430	+ def _notify(self, title, text):
	431	+ if self.interactive:
	432	+ self.notify(title, text)
	433	+
422	434	# Local message propagator
423	435	#
424	436	def _message(self, message_type, message, **kwargs):
...	...	@@ -571,8 +583,8 @@ class App():
571	583	while choice not in ['continue', 'quit', 'terminate', 'retry']:
572	584	click.echo(summary, err=True)
573	585
574		- self.notify("BuildStream failure", "{} on element {}"
575		- .format(failure.action_name, element.name))
	586	+ self._notify("BuildStream failure", "{} on element {}"
	587	+ .format(failure.action_name, element.name))
576	588
577	589	try:
578	590	choice = click.prompt("Choice:", default='continue', err=True,

buildstream/_pipeline.py

---

...	...	@@ -359,23 +359,14 @@ class Pipeline():
359	359
360	360	if inconsistent:
361	361	detail = "Exact versions are missing for the following elements:\n\n"
362		-
363		- missingTrack = 0
364	362	for element in inconsistent:
365		- detail += " " + element._get_full_name()
	363	+ detail += " Element: {} is inconsistent\n".format(element._get_full_name())
366	364	for source in element.sources():
367		- if not source._get_consistency() and not source.get_ref():
368		- if hasattr(source, 'tracking') and source.tracking is None:
369		- detail += ": Source {} is missing ref and track. ".format(source._get_full_name()) + \
370		- "Please specify a ref or branch/tag to track."
371		- missingTrack = 1
372		-
373		- detail += "\n"
	365	+ if source._get_consistency() == Consistency.INCONSISTENT:
	366	+ detail += " Source {} is missing ref\n".format(source)
	367	+ detail += '\n'
	368	+ detail += "Try tracking these elements first with `bst track`\n"
374	369
375		- if missingTrack:
376		- detail += "\nThen track these elements with `bst track`\n"
377		- else:
378		- detail += "\nTry tracking these elements first with `bst track`\n"
379	370	raise PipelineError("Inconsistent pipeline", detail=detail, reason="inconsistent-pipeline")
380	371
381	372	#############################################################

buildstream/plugins/sources/git.py

---

...	...	@@ -74,6 +74,9 @@ This plugin provides the following configurable warnings:
74	74
75	75	- 'git:inconsistent-submodule' - A submodule was found to be missing from the underlying git repository.
76	76
	77	+This plugin also utilises the following configurable core plugin warnings:
	78	+
	79	+- 'ref-not-in-track' - The provided ref was not found in the provided track in the element's git repository.
77	80	"""
78	81
79	82	import os
...	...	@@ -87,6 +90,7 @@ from configparser import RawConfigParser
87	90
88	91	from buildstream import Source, SourceError, Consistency, SourceFetcher
89	92	from buildstream import utils
	93	+from buildstream.plugin import CoreWarnings
90	94
91	95	GIT_MODULES = '.gitmodules'
92	96
...	...	@@ -199,7 +203,7 @@ class GitMirror(SourceFetcher):
199	203	cwd=self.mirror)
200	204	return output.rstrip('\n')
201	205
202		- def stage(self, directory):
	206	+ def stage(self, directory, track=None):
203	207	fullpath = os.path.join(directory, self.path)
204	208
205	209	# Using --shared here avoids copying the objects into the checkout, in any
...	...	@@ -213,10 +217,14 @@ class GitMirror(SourceFetcher):
213	217	fail="Failed to checkout git ref {}".format(self.ref),
214	218	cwd=fullpath)
215	219
	220	+ # Check that the user specified ref exists in the track if provided & not already tracked
	221	+ if track:
	222	+ self.assert_ref_in_track(fullpath, track)
	223	+
216	224	# Remove .git dir
217	225	shutil.rmtree(os.path.join(fullpath, ".git"))
218	226
219		- def init_workspace(self, directory):
	227	+ def init_workspace(self, directory, track=None):
220	228	fullpath = os.path.join(directory, self.path)
221	229	url = self.source.translate_url(self.url)
222	230
...	...	@@ -232,6 +240,10 @@ class GitMirror(SourceFetcher):
232	240	fail="Failed to checkout git ref {}".format(self.ref),
233	241	cwd=fullpath)
234	242
	243	+ # Check that the user specified ref exists in the track if provided & not already tracked
	244	+ if track:
	245	+ self.assert_ref_in_track(fullpath, track)
	246	+
235	247	# List the submodules (path/url tuples) present at the given ref of this repo
236	248	def submodule_list(self):
237	249	modules = "{}:{}".format(self.ref, GIT_MODULES)
...	...	@@ -296,6 +308,28 @@ class GitMirror(SourceFetcher):
296	308
297	309	return None
298	310
	311	+ # Assert that ref exists in track, if track has been specified.
	312	+ def assert_ref_in_track(self, fullpath, track):
	313	+ _, branch = self.source.check_output([self.source.host_git, 'branch', '--list', track,
	314	+ '--contains', self.ref],
	315	+ cwd=fullpath,)
	316	+ if branch:
	317	+ return True
	318	+ else:
	319	+ _, tag = self.source.check_output([self.source.host_git, 'tag', '--list', track,
	320	+ '--contains', self.ref],
	321	+ cwd=fullpath,)
	322	+ if tag:
	323	+ return True
	324	+
	325	+ detail = "The ref provided for the element does not exist locally in the provided track branch / tag " + \
	326	+ "'{}'.\nYou may wish to track the element to update the ref from '{}' ".format(track, track) + \
	327	+ "with `bst track`,\nor examine the upstream at '{}' for the specific ref.".format(self.url)
	328	+
	329	+ self.source.warn("{}: expected ref '{}' was not found in given track '{}' for staged repository: '{}'\n"
	330	+ .format(self.source, self.ref, track, self.url),
	331	+ detail=detail, warning_token=CoreWarnings.REF_NOT_IN_TRACK)
	332	+
299	333
300	334	class GitSource(Source):
301	335	# pylint: disable=attribute-defined-outside-init
...	...	@@ -309,6 +343,13 @@ class GitSource(Source):
309	343	self.original_url = self.node_get_member(node, str, 'url')
310	344	self.mirror = GitMirror(self, '', self.original_url, ref)
311	345	self.tracking = self.node_get_member(node, str, 'track', None)
	346	+
	347	+ # At this point we now know if the source has a ref and/or a track.
	348	+ # If it is missing both then we will be unable to track or build.
	349	+ if self.mirror.ref is None and self.tracking is None:
	350	+ raise SourceError("{}: Git sources require a ref and/or track".format(self),
	351	+ reason="missing-track-and-ref")
	352	+
312	353	self.checkout_submodules = self.node_get_member(node, bool, 'checkout-submodules', True)
313	354	self.submodules = []
314	355
...	...	@@ -326,6 +367,7 @@ class GitSource(Source):
326	367	self.submodule_checkout_overrides[path] = checkout
327	368
328	369	self.mark_download_url(self.original_url)
	370	+ self.tracked = False
329	371
330	372	def preflight(self):
331	373	# Check if git is installed, get the binary at the same time
...	...	@@ -389,6 +431,8 @@ class GitSource(Source):
389	431	# Update self.mirror.ref and node.ref from the self.tracking branch
390	432	ret = self.mirror.latest_commit(self.tracking)
391	433
	434	+ # Set tracked attribute, parameter for if self.mirror.assert_ref_in_track is needed
	435	+ self.tracked = True
392	436	return ret
393	437
394	438	def init_workspace(self, directory):
...	...	@@ -396,7 +440,7 @@ class GitSource(Source):
396	440	self.refresh_submodules()
397	441
398	442	with self.timed_activity('Setting up workspace "{}"'.format(directory), silent_nested=True):
399		- self.mirror.init_workspace(directory)
	443	+ self.mirror.init_workspace(directory, track=(self.tracking if not self.tracked else None))
400	444	for mirror in self.submodules:
401	445	mirror.init_workspace(directory)
402	446
...	...	@@ -412,7 +456,7 @@ class GitSource(Source):
412	456	# Stage the main repo in the specified directory
413	457	#
414	458	with self.timed_activity("Staging {}".format(self.mirror.url), silent_nested=True):
415		- self.mirror.stage(directory)
	459	+ self.mirror.stage(directory, track=(self.tracking if not self.tracked else None))
416	460	for mirror in self.submodules:
417	461	if mirror.path in self.submodule_checkout_overrides:
418	462	checkout = self.submodule_checkout_overrides[mirror.path]

buildstream/plugins/sources/ostree.py

---

...	...	@@ -71,7 +71,13 @@ class OSTreeSource(Source):
71	71	self.ref = self.node_get_member(node, str, 'ref', None)
72	72	self.tracking = self.node_get_member(node, str, 'track', None)
73	73	self.mirror = os.path.join(self.get_mirror_directory(),
74		- utils.url_directory_name(self.url))
	74	+ utils.url_directory_name(self.original_url))
	75	+
	76	+ # At this point we now know if the source has a ref and/or a track.
	77	+ # If it is missing both then we will be unable to track or build.
	78	+ if self.ref is None and self.tracking is None:
	79	+ raise SourceError("{}: OSTree sources require a ref and/or track".format(self),
	80	+ reason="missing-track-and-ref")
75	81
76	82	# (optional) Not all repos are signed. But if they are, get the gpg key
77	83	self.gpg_key_path = None
...	...	@@ -104,10 +110,11 @@ class OSTreeSource(Source):
104	110	return None
105	111
106	112	self.ensure()
	113	+ remote_name = self.ensure_remote(self.url)
107	114	with self.timed_activity("Fetching tracking ref '{}' from origin: {}"
108	115	.format(self.tracking, self.url)):
109	116	try:
110		- _ostree.fetch(self.repo, ref=self.tracking, progress=self.progress)
	117	+ _ostree.fetch(self.repo, remote=remote_name, ref=self.tracking, progress=self.progress)
111	118	except OSTreeError as e:
112	119	raise SourceError("{}: Failed to fetch tracking ref '{}' from origin {}\n\n{}"
113	120	.format(self, self.tracking, self.url, e)) from e
...	...	@@ -116,11 +123,12 @@ class OSTreeSource(Source):
116	123
117	124	def fetch(self):
118	125	self.ensure()
	126	+ remote_name = self.ensure_remote(self.url)
119	127	if not _ostree.exists(self.repo, self.ref):
120	128	with self.timed_activity("Fetching remote ref: {} from origin: {}"
121	129	.format(self.ref, self.url)):
122	130	try:
123		- _ostree.fetch(self.repo, ref=self.ref, progress=self.progress)
	131	+ _ostree.fetch(self.repo, remote=remote_name, ref=self.ref, progress=self.progress)
124	132	except OSTreeError as e:
125	133	raise SourceError("{}: Failed to fetch ref '{}' from origin: {}\n\n{}"
126	134	.format(self, self.ref, self.url, e)) from e
...	...	@@ -171,14 +179,22 @@ class OSTreeSource(Source):
171	179	self.status("Creating local mirror for {}".format(self.url))
172	180
173	181	self.repo = _ostree.ensure(self.mirror, True)
174		- gpg_key = None
175		- if self.gpg_key_path:
176		- gpg_key = 'file://' + self.gpg_key_path
177	182
178		- try:
179		- _ostree.configure_remote(self.repo, "origin", self.url, key_url=gpg_key)
180		- except OSTreeError as e:
181		- raise SourceError("{}: Failed to configure origin {}\n\n{}".format(self, self.url, e)) from e
	183	+ def ensure_remote(self, url):
	184	+ if self.original_url == self.url:
	185	+ remote_name = 'origin'
	186	+ else:
	187	+ remote_name = utils.url_directory_name(url)
	188	+
	189	+ gpg_key = None
	190	+ if self.gpg_key_path:
	191	+ gpg_key = 'file://' + self.gpg_key_path
	192	+
	193	+ try:
	194	+ _ostree.configure_remote(self.repo, remote_name, url, key_url=gpg_key)
	195	+ except OSTreeError as e:
	196	+ raise SourceError("{}: Failed to configure origin {}\n\n{}".format(self, self.url, e)) from e
	197	+ return remote_name
182	198
183	199	def progress(self, percent, message):
184	200	self.status(message)

doc/examples/autotools/project.conf

---

...	...	@@ -10,4 +10,4 @@ element-path: elements
10	10	# Define some aliases for the tarballs we download
11	11	aliases:
12	12	alpine: https://gnome7.codethink.co.uk/tarballs/
13		- gnu: https://ftpmirror.gnu.org/gnu/automake/
	13	+ gnu: http://ftpmirror.gnu.org/gnu/automake/

tests/artifactcache/expiry.py

---

	1	+#
	2	+# Copyright (C) 2018 Codethink Limited
	3	+#
	4	+# This program is free software; you can redistribute it and/or
	5	+# modify it under the terms of the GNU Lesser General Public
	6	+# License as published by the Free Software Foundation; either
	7	+# version 2 of the License, or (at your option) any later version.
	8	+#
	9	+# This library is distributed in the hope that it will be useful,
	10	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	11	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	12	+# Lesser General Public License for more details.
	13	+#
	14	+# You should have received a copy of the GNU Lesser General Public
	15	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	16	+#
	17	+# Authors: Tristan Maat <tristan maat codethink co uk>
	18	+#
	19	+
1	20	import os
2	21
3	22	import pytest
...	...	@@ -5,7 +24,7 @@ import pytest
5	24	from buildstream import _yaml
6	25	from buildstream._exceptions import ErrorDomain, LoadErrorReason
7	26
8		-from tests.testutils import cli, create_element_size
	27	+from tests.testutils import cli, create_element_size, wait_for_cache_granularity
9	28
10	29
11	30	DATA_DIR = os.path.join(
...	...	@@ -108,6 +127,8 @@ def test_expiry_order(cli, datafiles, tmpdir):
108	127	res = cli.run(project=project, args=['build', 'target2.bst'])
109	128	res.assert_success()
110	129
	130	+ wait_for_cache_granularity()
	131	+
111	132	# Now extract dep.bst
112	133	res = cli.run(project=project, args=['checkout', 'dep.bst', checkout])
113	134	res.assert_success()

tests/frontend/mirror.py

---

...	...	@@ -466,10 +466,6 @@ def test_mirror_track_upstream_absent(cli, tmpdir, datafiles, kind):
466	466	@pytest.mark.datafiles(DATA_DIR)
467	467	@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
468	468	def test_mirror_from_includes(cli, tmpdir, datafiles, kind):
469		- if kind == 'ostree':
470		- # FIXME: Mirroring fallback fails with ostree
471		- pytest.skip("Bug #538 - ostree mirror fallback breaks assertion")
472		-
473	469	bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
474	470	upstream_repodir = os.path.join(str(tmpdir), 'upstream')
475	471	mirror_repodir = os.path.join(str(tmpdir), 'mirror')

tests/frontend/push.py

---

	1	+#
	2	+# Copyright (C) 2018 Codethink Limited
	3	+# Copyright (C) 2018 Bloomberg Finance LP
	4	+#
	5	+# This program is free software; you can redistribute it and/or
	6	+# modify it under the terms of the GNU Lesser General Public
	7	+# License as published by the Free Software Foundation; either
	8	+# version 2 of the License, or (at your option) any later version.
	9	+#
	10	+# This library is distributed in the hope that it will be useful,
	11	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	12	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	13	+# Lesser General Public License for more details.
	14	+#
	15	+# You should have received a copy of the GNU Lesser General Public
	16	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	17	+#
	18	+# Authors: Tristan Van Berkom <tristan vanberkom codethink co uk>
	19	+# Sam Thursfield <sam thursfield codethink co uk>
	20	+# Jürg Billeter <juerg billeter codethink co uk>
	21	+#
	22	+
1	23	import os
2	24	import pytest
3	25
4	26	from buildstream._exceptions import ErrorDomain
5	27	from tests.testutils import cli, create_artifact_share, create_element_size
6		-from tests.testutils import generate_junction
	28	+from tests.testutils import generate_junction, wait_for_cache_granularity
7	29	from . import configure_project
8	30
9	31
...	...	@@ -327,6 +349,8 @@ def test_recently_pulled_artifact_does_not_expire(cli, datafiles, tmpdir):
327	349	# Ensure element1 is cached locally
328	350	assert cli.get_element_state(project, 'element1.bst') == 'cached'
329	351
	352	+ wait_for_cache_granularity()
	353	+
330	354	# Create and build the element3 (of 5 MB)
331	355	create_element_size('element3.bst', project, element_path, [], int(5e6))
332	356	result = cli.run(project=project, args=['build', 'element3.bst'])

tests/frontend/workspace.py

---

	1	+#
	2	+# Copyright (C) 2018 Codethink Limited
	3	+# Copyright (C) 2018 Bloomberg Finance LP
	4	+#
	5	+# This program is free software; you can redistribute it and/or
	6	+# modify it under the terms of the GNU Lesser General Public
	7	+# License as published by the Free Software Foundation; either
	8	+# version 2 of the License, or (at your option) any later version.
	9	+#
	10	+# This library is distributed in the hope that it will be useful,
	11	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	12	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	13	+# Lesser General Public License for more details.
	14	+#
	15	+# You should have received a copy of the GNU Lesser General Public
	16	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	17	+#
	18	+# Authors: Tristan Van Berkom <tristan vanberkom codethink co uk>
	19	+# Tristan Maat <tristan maat codethink co uk>
	20	+# Chandan Singh <csingh43 bloomberg net>
	21	+# Phillip Smyth <phillip smyth codethink co uk>
	22	+# Jonathan Maw <jonathan maw codethink co uk>
	23	+# Richard Maw <richard maw codethink co uk>
	24	+#
	25	+
1	26	import os
2	27	import pytest
3	28	import shutil
4	29	import subprocess
5	30	from ruamel.yaml.comments import CommentedSet
6		-from tests.testutils import cli, create_repo, ALL_REPO_KINDS
	31	+from tests.testutils import cli, create_repo, ALL_REPO_KINDS, wait_for_cache_granularity
7	32
8	33	from buildstream import _yaml
9	34	from buildstream._exceptions import ErrorDomain, LoadError, LoadErrorReason
...	...	@@ -507,6 +532,8 @@ def test_detect_modifications(cli, tmpdir, datafiles, modification, strict):
507	532	assert cli.get_element_state(project, element_name) == 'cached'
508	533	assert cli.get_element_key(project, element_name) != "{:?<64}".format('')
509	534
	535	+ wait_for_cache_granularity()
	536	+
510	537	# Modify the workspace in various different ways, ensuring we
511	538	# properly detect the changes.
512	539	#

tests/sources/git.py

---

	1	+#
	2	+# Copyright (C) 2018 Codethink Limited
	3	+# Copyright (C) 2018 Bloomberg Finance LP
	4	+#
	5	+# This program is free software; you can redistribute it and/or
	6	+# modify it under the terms of the GNU Lesser General Public
	7	+# License as published by the Free Software Foundation; either
	8	+# version 2 of the License, or (at your option) any later version.
	9	+#
	10	+# This library is distributed in the hope that it will be useful,
	11	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	12	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	13	+# Lesser General Public License for more details.
	14	+#
	15	+# You should have received a copy of the GNU Lesser General Public
	16	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	17	+#
	18	+# Authors: Tristan Van Berkom <tristan vanberkom codethink co uk>
	19	+# Jonathan Maw <jonathan maw codethink co uk>
	20	+# William Salmon <will salmon codethink co uk>
	21	+#
	22	+
1	23	import os
2	24	import pytest
3	25
4	26	from buildstream._exceptions import ErrorDomain
5	27	from buildstream import _yaml
	28	+from buildstream.plugin import CoreWarnings
6	29
7	30	from tests.testutils import cli, create_repo
8	31	from tests.testutils.site import HAVE_GIT
...	...	@@ -383,21 +406,78 @@ def test_submodule_track_no_ref_or_track(cli, tmpdir, datafiles):
383	406	_yaml.dump(element, os.path.join(project, 'target.bst'))
384	407
385	408	# Track will encounter an inconsistent submodule without any ref
386		- result = cli.run(project=project, args=['track', 'target.bst'])
387		- result.assert_main_error(ErrorDomain.STREAM, None)
388		- result.assert_task_error(ErrorDomain.SOURCE, 'track-attempt-no-track')
389		-
390		- # Assert that we are just fine without it, and emit a warning to the user.
391		- assert "FAILURE git source at" in result.stderr
392		- assert "Without a tracking branch ref can not be updated. Please " + \
393		- "provide a ref or a track." in result.stderr
394		-
395		- # Track will encounter an inconsistent submodule without any ref
396		- result = cli.run(project=project, args=['build', 'target.bst'])
397		- result.assert_main_error(ErrorDomain.PIPELINE, 'inconsistent-pipeline')
	409	+ result = cli.run(project=project, args=['show', 'target.bst'])
	410	+ result.assert_main_error(ErrorDomain.SOURCE, "missing-track-and-ref")
398	411	result.assert_task_error(None, None)
399	412
400	413	# Assert that we are just fine without it, and emit a warning to the user.
401	414	assert "Exact versions are missing for the following elements" in result.stderr
402	415	assert "is missing ref and track." in result.stderr
403	416	assert "Then track these elements with `bst track`" in result.stderr
	417	+
	418	+
	419	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	420	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	421	+def test_ref_not_in_track_warn(cli, tmpdir, datafiles):
	422	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	423	+
	424	+ # Create the repo from 'repofiles', create a branch without latest commit
	425	+ repo = create_repo('git', str(tmpdir))
	426	+ ref = repo.create(os.path.join(project, 'repofiles'))
	427	+
	428	+ gitsource = repo.source_config(ref=ref)
	429	+
	430	+ # Overwrite the track value to the added branch
	431	+ gitsource['track'] = 'foo'
	432	+
	433	+ # Write out our test target
	434	+ element = {
	435	+ 'kind': 'import',
	436	+ 'sources': [
	437	+ gitsource
	438	+ ]
	439	+ }
	440	+ _yaml.dump(element, os.path.join(project, 'target.bst'))
	441	+
	442	+ # Assert the warning is raised as ref is not in branch foo.
	443	+ # Assert warning not error to the user, when not set as fatal.
	444	+ result = cli.run(project=project, args=['build', 'target.bst'])
	445	+ assert "The ref provided for the element does not exist locally" in result.stderr
	446	+
	447	+
	448	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	449	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	450	+def test_ref_not_in_track_warn_error(cli, tmpdir, datafiles):
	451	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	452	+
	453	+ # Add fatal-warnings ref-not-in-track to project.conf
	454	+ project_template = {
	455	+ "name": "foo",
	456	+ "fatal-warnings": [CoreWarnings.REF_NOT_IN_TRACK]
	457	+ }
	458	+
	459	+ _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	460	+
	461	+ # Create the repo from 'repofiles', create a branch without latest commit
	462	+ repo = create_repo('git', str(tmpdir))
	463	+ ref = repo.create(os.path.join(project, 'repofiles'))
	464	+
	465	+ gitsource = repo.source_config(ref=ref)
	466	+
	467	+ # Overwrite the track value to the added branch
	468	+ gitsource['track'] = 'foo'
	469	+
	470	+ # Write out our test target
	471	+ element = {
	472	+ 'kind': 'import',
	473	+ 'sources': [
	474	+ gitsource
	475	+ ]
	476	+ }
	477	+ _yaml.dump(element, os.path.join(project, 'target.bst'))
	478	+
	479	+ # Assert that build raises a warning here that is captured
	480	+ # as plugin error, due to the fatal warning being set
	481	+ result = cli.run(project=project, args=['build', 'target.bst'])
	482	+ result.assert_main_error(ErrorDomain.STREAM, None)
	483	+ result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.REF_NOT_IN_TRACK)

tests/sources/ostree.py

---

	1	+#
	2	+# Copyright (C) 2018 Bloomberg Finance LP
	3	+#
	4	+# This program is free software; you can redistribute it and/or
	5	+# modify it under the terms of the GNU Lesser General Public
	6	+# License as published by the Free Software Foundation; either
	7	+# version 2 of the License, or (at your option) any later version.
	8	+#
	9	+# This library is distributed in the hope that it will be useful,
	10	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	11	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	12	+# Lesser General Public License for more details.
	13	+#
	14	+# You should have received a copy of the GNU Lesser General Public
	15	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	16	+#
	17	+# Authors: William Salmon <will salmon codethink co uk>
	18	+#
	19	+
	20	+import os
	21	+import pytest
	22	+
	23	+from buildstream._exceptions import ErrorDomain
	24	+from buildstream import _yaml
	25	+
	26	+from tests.testutils import cli, create_repo
	27	+
	28	+DATA_DIR = os.path.join(
	29	+ os.path.dirname(os.path.realpath(__file__)),
	30	+ 'ostree',
	31	+)
	32	+
	33	+
	34	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	35	+def test_submodule_track_no_ref_or_track(cli, tmpdir, datafiles):
	36	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	37	+
	38	+ # Create the repo from 'repofiles' subdir
	39	+ repo = create_repo('ostree', str(tmpdir))
	40	+ ref = repo.create(os.path.join(project, 'repofiles'))
	41	+
	42	+ # Write out our test target
	43	+ ostreesource = repo.source_config(ref=None)
	44	+ ostreesource.pop('track')
	45	+ element = {
	46	+ 'kind': 'import',
	47	+ 'sources': [
	48	+ ostreesource
	49	+ ]
	50	+ }
	51	+
	52	+ _yaml.dump(element, os.path.join(project, 'target.bst'))
	53	+
	54	+ # Track will encounter an inconsistent submodule without any ref
	55	+ result = cli.run(project=project, args=['show', 'target.bst'])
	56	+ result.assert_main_error(ErrorDomain.SOURCE, "missing-track-and-ref")
	57	+ result.assert_task_error(None, None)

tests/sources/ostree/template/project.conf

---

	1	+# Basic project
	2	+name: foo

tests/sources/ostree/template/repofiles/file

---

tests/testutils/__init__.py

---

	1	+#
	2	+# Copyright (C) 2018 Codethink Limited
	3	+# Copyright (C) 2018 Bloomberg Finance LP
	4	+#
	5	+# This program is free software; you can redistribute it and/or
	6	+# modify it under the terms of the GNU Lesser General Public
	7	+# License as published by the Free Software Foundation; either
	8	+# version 2 of the License, or (at your option) any later version.
	9	+#
	10	+# This library is distributed in the hope that it will be useful,
	11	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	12	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	13	+# Lesser General Public License for more details.
	14	+#
	15	+# You should have received a copy of the GNU Lesser General Public
	16	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	17	+#
	18	+# Authors: Tristan Van Berkom <tristan vanberkom codethink co uk>
	19	+# Tristan Maat <tristan maat codethink co uk>
	20	+# Sam Thursfield <sam thursfield codethink co uk>
	21	+# James Ennis <james ennis codethink co uk>
	22	+# Valentin David <valentin david codethink co uk>
	23	+# William Salmon <will salmon codethink co uk>
	24	+#
	25	+
1	26	from .runcli import cli, cli_integration
2	27	from .repo import create_repo, ALL_REPO_KINDS
3	28	from .artifactshare import create_artifact_share
4	29	from .element_generators import create_element_size
5	30	from .junction import generate_junction
	31	+from .runner_integration import wait_for_cache_granularity

tests/testutils/runner_integration.py

---

	1	+#
	2	+# Copyright (C) 2018 Bloomberg Finance LP
	3	+#
	4	+# This program is free software; you can redistribute it and/or
	5	+# modify it under the terms of the GNU Lesser General Public
	6	+# License as published by the Free Software Foundation; either
	7	+# version 2 of the License, or (at your option) any later version.
	8	+#
	9	+# This library is distributed in the hope that it will be useful,
	10	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	11	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	12	+# Lesser General Public License for more details.
	13	+#
	14	+# You should have received a copy of the GNU Lesser General Public
	15	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	16	+#
	17	+# Authors:
	18	+# Will Salmon <will salmon codethink co uk>
	19	+
	20	+import time
	21	+
	22	+
	23	+def wait_for_cache_granularity():
	24	+ # This isn't called very often so has minimal impact on test runtime.
	25	+ # If this changes it may be worth while adding a more sophisticated approach.
	26	+ """
	27	+ Mitigate the coarse granularity of the gitlab runners mtime
	28	+
	29	+ This function waits for the mtime to increment so that the cache can sort by mtime and
	30	+ get the most recent results.
	31	+ """
	32	+ time.sleep(1.1)

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.