[Notes] [Git][BuildStream/buildstream][willsalmon/580-backport] 20 commits: NEWS: Adding a news entry for the 1.1.6 release candidate

[Will Salmon <gitlab mg gitlab com>]

Title: GitLab

Will Salmon pushed to branch willsalmon/580-backport at BuildStream / buildstream

Commits:

• 970c6696
  by Tristan Van Berkom at 2018-08-14T09:10:18Z

  NEWS: Adding a news entry for the 1.1.6 release candidate
  

• 40963952
  by Mathieu Bridon at 2018-08-14T21:35:24Z

  deps: Specify the minimum version required for blessings
  
  Buildstream uses the Terminal.does_styling attribute, which was only
  added in blessings 1.6.
  
  Blessings helpfully returns an empty string when calling an nonexistent
  attribute on the Terminal object, instead of failing.
  
  Because Fedora has blessings 1.5, Buildstream thought that my terminal
  couldn't handle styling, and just didn't print the status bar, silently.
  
  Specifying the minimum version avoids this silent failure.
  

• 02396c90
  by Tristan Van Berkom at 2018-08-15T07:31:43Z

  Merge branch 'bochecha/blessings' into 'bst-1.2'
  
  deps: Specify the minimum version required for blessings
  
  See merge request BuildStream/buildstream!663

• 0795a4c5
  by Josh Smith at 2018-08-15T14:34:04Z

  examples: Switch ftp.gnu.org to mirror
  
  This should avoid current issues accessing the necessary sources which
  are blocking CI currently.
  

• ac97bc5e
  by Valentin David at 2018-08-15T15:25:53Z

  Merge branch 'Qinusty/gnu-mirror-backport-1.2' into 'bst-1.2'
  
  Qinusty/gnu mirror backport 1.2
  
  See merge request BuildStream/buildstream!668

• d6d502f2
  by Valentin David at 2018-08-16T08:37:28Z

  buildstream/plugins/sources/git.py: Fix clone side effect in getting fetchers
  
  We now delay refreshing submodules until we have a clone, that is after
  we have fetched the main fetcher.
  
  Fixes #537
  

• 31ee84ac
  by Valentin David at 2018-08-16T08:37:28Z

  tests/frontend/mirror.py: Re-enable a test for git
  

• debdd7dd
  by Jonathan Maw at 2018-08-16T08:37:28Z

  tests: Test that fetching passes when upstream is absent
  

• 3d9aac51
  by Jonathan Maw at 2018-08-16T08:37:28Z

  tests: Add a test of the git source with submodules
  

• de5ca03d
  by Valentin David at 2018-08-16T08:37:28Z

  Test git mirroring fallback on submodules when main repo is not mirrored.
  

• 6023004b
  by Valentin David at 2018-08-16T08:37:28Z

  Test we can discovered submodules on fallback mirrored git repositories
  

• b77305d3
  by Valentin David at 2018-08-16T11:14:52Z

  Merge branch 'valentindavid/fallback_mirror_git-1.2' into 'bst-1.2'
  
  Delay refreshing git submodule until we have a clone
  
  See merge request BuildStream/buildstream!666

• 449a692b
  by Tiago Gomes at 2018-08-16T12:03:52Z

  Use http instead of https for the gnu ftp mirror
  
  https seems broken on the Debian image:
  
  START   autotools/hello/42930621-fetch.499.log
  START   Fetching https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz
  FAILURE Fetching https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz
  FAILURE tar source at hello.bst [line 16 column 2]: Error mirroring https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz: <urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:720)>
  

• 3bc8a513
  by Tiago Gomes at 2018-08-16T12:58:32Z

  Merge branch 'tiagogomes/disable-https-gnu-repo-backport' into 'bst-1.2'
  
  Use http instead of https for the gnu ftp mirror
  
  See merge request BuildStream/buildstream!677

• de73cc0b
  by Valentin David at 2018-08-16T13:37:06Z

  Fix ostree repository mirroring
  
  Ostree mirrors were not sharing the same local repository, so it was
  impossible the request refs from the right local repository when data
  was fetched from a mirror rather than upstream.
  
  Instead of having several repository with one remote each, we now
  have one repository with several remotes.
  
  This fixes #538.
  

• 43f4ffba
  by Valentin David at 2018-08-16T14:36:49Z

  Merge branch 'valentindavid/fallback_mirror_ostree-1.2' into 'bst-1.2'
  
  Fix ostree repository mirroring
  
  See merge request BuildStream/buildstream!667

• 1e140f7d
  by Tristan Van Berkom at 2018-08-16T14:38:02Z

  _frontend/app.py: Notify session completions
  
  Use the optionally implemented desktop notification method
  to notify when a session completes.
  
  Previously, this used to only notify when an interactive prompt
  appears.
  
  This is an attempt to fix #385
  

• 87960822
  by Tristan Van Berkom at 2018-08-16T15:36:18Z

  Merge branch 'tristan/notifications-1.2' into 'bst-1.2'
  
  _frontend/app.py: Notify session completions
  
  See merge request BuildStream/buildstream!673

• 836e514d
  by William Salmon at 2018-08-16T16:37:29Z

  Add Error to git and ostree sources configure
  
  Raise a error at configure time if the track and ref properties are
  not present in the sources.
  
  This is to address https://gitlab.com/BuildStream/buildstream/issues/471
  that documented unhelpful behaviour when tracking git sources. However
  the issue was also identified in ostree.
  

• 9e22f69a
  by William Salmon at 2018-08-16T16:37:29Z

  Improve error message for build if there are refs missing
  

14 changed files:

• NEWS
• buildstream/_frontend/app.py
• buildstream/_pipeline.py
• buildstream/plugins/sources/git.py
• buildstream/plugins/sources/ostree.py
• doc/examples/autotools/project.conf
• setup.py
• tests/frontend/mirror.py
• + tests/frontend/project/files/bar
• + tests/frontend/project/files/foo
• tests/sources/git.py
• + tests/sources/ostree.py
• + tests/sources/ostree/template/project.conf
• + tests/sources/ostree/template/repofiles/file

Changes:

NEWS

---

	1	+=================
	2	+buildstream 1.1.6
	3	+=================
	4	+
	5	+ o A lot of bug fixes
	6	+
	7	+
1	8	=================
2	9	buildstream 1.1.5
3	10	=================

buildstream/_frontend/app.py

---

...	...	@@ -270,6 +270,9 @@ class App():
270	270	else:
271	271	self._message(MessageType.FAIL, session_name, elapsed=elapsed)
272	272
	273	+ # Notify session failure
	274	+ self._notify("{} failed".format(session_name), "{}".format(e))
	275	+
273	276	if self._started:
274	277	self._print_summary()
275	278
...	...	@@ -287,6 +290,9 @@ class App():
287	290	if self._started:
288	291	self._print_summary()
289	292
	293	+ # Notify session success
	294	+ self._notify("{} succeeded".format(session_name), "")
	295	+
290	296	# init_project()
291	297	#
292	298	# Initialize a new BuildStream project, either with the explicitly passed options,
...	...	@@ -420,6 +426,12 @@ class App():
420	426	# Local Functions #
421	427	############################################################
422	428
	429	+ # Local function for calling the notify() virtual method
	430	+ #
	431	+ def _notify(self, title, text):
	432	+ if self.interactive:
	433	+ self.notify(title, text)
	434	+
423	435	# Local message propagator
424	436	#
425	437	def _message(self, message_type, message, **kwargs):
...	...	@@ -572,8 +584,8 @@ class App():
572	584	while choice not in ['continue', 'quit', 'terminate', 'retry']:
573	585	click.echo(summary, err=True)
574	586
575		- self.notify("BuildStream failure", "{} on element {}"
576		- .format(failure.action_name, element.name))
	587	+ self._notify("BuildStream failure", "{} on element {}"
	588	+ .format(failure.action_name, element.name))
577	589
578	590	try:
579	591	choice = click.prompt("Choice:", default='continue', err=True,

buildstream/_pipeline.py

---

...	...	@@ -358,10 +358,15 @@ class Pipeline():
358	358	inconsistent.append(element)
359	359
360	360	if inconsistent:
361		- detail = "Exact versions are missing for the following elements\n" + \
362		- "Try tracking these elements first with `bst track`\n\n"
	361	+ detail = "Exact versions are missing for the following elements:\n\n"
363	362	for element in inconsistent:
364		- detail += " " + element._get_full_name() + "\n"
	363	+ detail += " Element: {} is inconsistent\n".format(element._get_full_name())
	364	+ for source in element.sources():
	365	+ if source._get_consistency() == Consistency.INCONSISTENT:
	366	+ detail += " Source {} is missing ref\n".format(source)
	367	+ detail += '\n'
	368	+ detail += "Try tracking these elements first with `bst track`\n"
	369	+
365	370	raise PipelineError("Inconsistent pipeline", detail=detail, reason="inconsistent-pipeline")
366	371
367	372	#############################################################

buildstream/plugins/sources/git.py

---

...	...	@@ -296,6 +296,13 @@ class GitSource(Source):
296	296	self.original_url = self.node_get_member(node, str, 'url')
297	297	self.mirror = GitMirror(self, '', self.original_url, ref)
298	298	self.tracking = self.node_get_member(node, str, 'track', None)
	299	+
	300	+ # At this point we now know if the source has a ref and/or a track.
	301	+ # If it is missing both then we will be unable to track or build.
	302	+ if self.mirror.ref is None and self.tracking is None:
	303	+ raise SourceError("{}: Git sources require a ref and/or track".format(self),
	304	+ reason = "missing-track-and-ref")
	305	+
299	306	self.checkout_submodules = self.node_get_member(node, bool, 'checkout-submodules', True)
300	307	self.submodules = []
301	308
...	...	@@ -404,8 +411,10 @@ class GitSource(Source):
404	411	mirror.stage(directory)
405	412
406	413	def get_source_fetchers(self):
	414	+ yield self.mirror
407	415	self.refresh_submodules()
408		- return [self.mirror] + self.submodules
	416	+ for submodule in self.submodules:
	417	+ yield submodule
409	418
410	419	###########################################################
411	420	# Local Functions #

buildstream/plugins/sources/ostree.py

---

...	...	@@ -71,7 +71,13 @@ class OSTreeSource(Source):
71	71	self.ref = self.node_get_member(node, str, 'ref', None)
72	72	self.tracking = self.node_get_member(node, str, 'track', None)
73	73	self.mirror = os.path.join(self.get_mirror_directory(),
74		- utils.url_directory_name(self.url))
	74	+ utils.url_directory_name(self.original_url))
	75	+
	76	+ # At this point we now know if the source has a ref and/or a track.
	77	+ # If it is missing both then we will be unable to track or build.
	78	+ if self.ref is None and self.tracking is None:
	79	+ raise SourceError("{}: OSTree sources require a ref and/or track".format(self),
	80	+ reason = "missing-track-and-ref")
75	81
76	82	# (optional) Not all repos are signed. But if they are, get the gpg key
77	83	self.gpg_key_path = None
...	...	@@ -104,10 +110,11 @@ class OSTreeSource(Source):
104	110	return None
105	111
106	112	self.ensure()
	113	+ remote_name = self.ensure_remote(self.url)
107	114	with self.timed_activity("Fetching tracking ref '{}' from origin: {}"
108	115	.format(self.tracking, self.url)):
109	116	try:
110		- _ostree.fetch(self.repo, ref=self.tracking, progress=self.progress)
	117	+ _ostree.fetch(self.repo, remote=remote_name, ref=self.tracking, progress=self.progress)
111	118	except OSTreeError as e:
112	119	raise SourceError("{}: Failed to fetch tracking ref '{}' from origin {}\n\n{}"
113	120	.format(self, self.tracking, self.url, e)) from e
...	...	@@ -116,11 +123,12 @@ class OSTreeSource(Source):
116	123
117	124	def fetch(self):
118	125	self.ensure()
	126	+ remote_name = self.ensure_remote(self.url)
119	127	if not _ostree.exists(self.repo, self.ref):
120	128	with self.timed_activity("Fetching remote ref: {} from origin: {}"
121	129	.format(self.ref, self.url)):
122	130	try:
123		- _ostree.fetch(self.repo, ref=self.ref, progress=self.progress)
	131	+ _ostree.fetch(self.repo, remote=remote_name, ref=self.ref, progress=self.progress)
124	132	except OSTreeError as e:
125	133	raise SourceError("{}: Failed to fetch ref '{}' from origin: {}\n\n{}"
126	134	.format(self, self.ref, self.url, e)) from e
...	...	@@ -171,14 +179,22 @@ class OSTreeSource(Source):
171	179	self.status("Creating local mirror for {}".format(self.url))
172	180
173	181	self.repo = _ostree.ensure(self.mirror, True)
174		- gpg_key = None
175		- if self.gpg_key_path:
176		- gpg_key = 'file://' + self.gpg_key_path
177	182
178		- try:
179		- _ostree.configure_remote(self.repo, "origin", self.url, key_url=gpg_key)
180		- except OSTreeError as e:
181		- raise SourceError("{}: Failed to configure origin {}\n\n{}".format(self, self.url, e)) from e
	183	+ def ensure_remote(self, url):
	184	+ if self.original_url == self.url:
	185	+ remote_name = 'origin'
	186	+ else:
	187	+ remote_name = utils.url_directory_name(url)
	188	+
	189	+ gpg_key = None
	190	+ if self.gpg_key_path:
	191	+ gpg_key = 'file://' + self.gpg_key_path
	192	+
	193	+ try:
	194	+ _ostree.configure_remote(self.repo, remote_name, url, key_url=gpg_key)
	195	+ except OSTreeError as e:
	196	+ raise SourceError("{}: Failed to configure origin {}\n\n{}".format(self, self.url, e)) from e
	197	+ return remote_name
182	198
183	199	def progress(self, percent, message):
184	200	self.status(message)

doc/examples/autotools/project.conf

---

...	...	@@ -10,4 +10,4 @@ element-path: elements
10	10	# Define some aliases for the tarballs we download
11	11	aliases:
12	12	alpine: https://gnome7.codethink.co.uk/tarballs/
13		- gnu: https://ftp.gnu.org/gnu/automake/
	13	+ gnu: http://ftpmirror.gnu.org/gnu/automake/

setup.py

---

...	...	@@ -261,7 +261,7 @@ setup(name='BuildStream',
261	261	'ruamel.yaml < 0.15.52',
262	262	'pluginbase',
263	263	'Click',
264		- 'blessings',
	264	+ 'blessings >= 1.6',
265	265	'jinja2 >= 2.10',
266	266	'protobuf >= 3.5',
267	267	'grpcio >= 1.10',

tests/frontend/mirror.py

---

...	...	@@ -139,6 +139,67 @@ def test_mirror_fetch(cli, tmpdir, datafiles, kind):
139	139	result.assert_success()
140	140
141	141
	142	+@pytest.mark.datafiles(DATA_DIR)
	143	+@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
	144	+def test_mirror_fetch_upstream_absent(cli, tmpdir, datafiles, kind):
	145	+ if kind == 'ostree':
	146	+ # FIXME: Mirroring fallback fails with ostree
	147	+ pytest.skip("Bug #538 - ostree mirror fallback breaks assertion")
	148	+
	149	+ bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
	150	+ dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
	151	+ upstream_repodir = os.path.join(str(tmpdir), 'upstream')
	152	+ mirror_repodir = os.path.join(str(tmpdir), 'mirror')
	153	+ project_dir = os.path.join(str(tmpdir), 'project')
	154	+ os.makedirs(project_dir)
	155	+ element_dir = os.path.join(project_dir, 'elements')
	156	+
	157	+ # Create repo objects of the upstream and mirror
	158	+ upstream_repo = create_repo(kind, upstream_repodir)
	159	+ ref = upstream_repo.create(dev_files_path)
	160	+ mirror_repo = upstream_repo.copy(mirror_repodir)
	161	+
	162	+ element = {
	163	+ 'kind': 'import',
	164	+ 'sources': [
	165	+ upstream_repo.source_config(ref=ref)
	166	+ ]
	167	+ }
	168	+
	169	+ element_name = 'test.bst'
	170	+ element_path = os.path.join(element_dir, element_name)
	171	+ full_repo = element['sources'][0]['url']
	172	+ upstream_map, repo_name = os.path.split(full_repo)
	173	+ alias = 'foo-' + kind
	174	+ aliased_repo = alias + ':' + repo_name
	175	+ element['sources'][0]['url'] = aliased_repo
	176	+ full_mirror = mirror_repo.source_config()['url']
	177	+ mirror_map, _ = os.path.split(full_mirror)
	178	+ os.makedirs(element_dir)
	179	+ _yaml.dump(element, element_path)
	180	+
	181	+ project = {
	182	+ 'name': 'test',
	183	+ 'element-path': 'elements',
	184	+ 'aliases': {
	185	+ alias: 'http://www.example.com/'
	186	+ },
	187	+ 'mirrors': [
	188	+ {
	189	+ 'name': 'middle-earth',
	190	+ 'aliases': {
	191	+ alias: [mirror_map + "/"],
	192	+ },
	193	+ },
	194	+ ]
	195	+ }
	196	+ project_file = os.path.join(project_dir, 'project.conf')
	197	+ _yaml.dump(project, project_file)
	198	+
	199	+ result = cli.run(project=project_dir, args=['fetch', element_name])
	200	+ result.assert_success()
	201	+
	202	+
142	203	@pytest.mark.datafiles(DATA_DIR)
143	204	def test_mirror_fetch_multi(cli, tmpdir, datafiles):
144	205	output_file = os.path.join(str(tmpdir), "output.txt")
...	...	@@ -405,14 +466,6 @@ def test_mirror_track_upstream_absent(cli, tmpdir, datafiles, kind):
405	466	@pytest.mark.datafiles(DATA_DIR)
406	467	@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
407	468	def test_mirror_from_includes(cli, tmpdir, datafiles, kind):
408		- if kind == 'git':
409		- # FIXME: Mirroring fallback does not work with git because it tries to
410		- # fetch submodules on upstream.
411		- pytest.skip("Bug #537 - Mirror fallback does not work for git")
412		- if kind == 'ostree':
413		- # FIXME: Mirroring fallback fails with ostree
414		- pytest.skip("Bug #538 - ostree mirror fallback breaks assertion")
415		-
416	469	bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
417	470	upstream_repodir = os.path.join(str(tmpdir), 'upstream')
418	471	mirror_repodir = os.path.join(str(tmpdir), 'mirror')
...	...	@@ -559,3 +612,276 @@ def test_mirror_junction_from_includes(cli, tmpdir, datafiles, kind):
559	612	os.rename('{}.bak'.format(upstream_repo.repo), upstream_repo.repo)
560	613	result = cli.run(project=project_dir, args=['fetch', element_name])
561	614	result.assert_success()
	615	+
	616	+
	617	+@pytest.mark.datafiles(DATA_DIR)
	618	+def test_mirror_git_submodule_fetch(cli, tmpdir, datafiles):
	619	+ # Test that it behaves as expected with submodules, both defined in config
	620	+ # and discovered when fetching.
	621	+ foo_file = os.path.join(str(datafiles), 'files', 'foo')
	622	+ bar_file = os.path.join(str(datafiles), 'files', 'bar')
	623	+ bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
	624	+ dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
	625	+ mirror_dir = os.path.join(str(datafiles), 'mirror')
	626	+
	627	+ defined_subrepo = create_repo('git', str(tmpdir), 'defined_subrepo')
	628	+ defined_mirror_ref = defined_subrepo.create(bin_files_path)
	629	+ defined_mirror = defined_subrepo.copy(mirror_dir)
	630	+ defined_subref = defined_subrepo.add_file(foo_file)
	631	+
	632	+ found_subrepo = create_repo('git', str(tmpdir), 'found_subrepo')
	633	+ found_subref = found_subrepo.create(dev_files_path)
	634	+
	635	+ main_repo = create_repo('git', str(tmpdir))
	636	+ main_mirror_ref = main_repo.create(bin_files_path)
	637	+ main_repo.add_submodule('defined', 'file://' + defined_subrepo.repo)
	638	+ main_repo.add_submodule('found', 'file://' + found_subrepo.repo)
	639	+ main_mirror = main_repo.copy(mirror_dir)
	640	+ main_ref = main_repo.add_file(bar_file)
	641	+
	642	+ project_dir = os.path.join(str(tmpdir), 'project')
	643	+ os.makedirs(project_dir)
	644	+ element_dir = os.path.join(project_dir, 'elements')
	645	+ os.makedirs(element_dir)
	646	+ element = {
	647	+ 'kind': 'import',
	648	+ 'sources': [
	649	+ main_repo.source_config(ref=main_mirror_ref)
	650	+ ]
	651	+ }
	652	+ element_name = 'test.bst'
	653	+ element_path = os.path.join(element_dir, element_name)
	654	+
	655	+ # Alias the main repo
	656	+ full_repo = element['sources'][0]['url']
	657	+ _, repo_name = os.path.split(full_repo)
	658	+ alias = 'foo'
	659	+ aliased_repo = alias + ':' + repo_name
	660	+ element['sources'][0]['url'] = aliased_repo
	661	+
	662	+ # Hide the found subrepo
	663	+ del element['sources'][0]['submodules']['found']
	664	+
	665	+ # Alias the defined subrepo
	666	+ subrepo = element['sources'][0]['submodules']['defined']['url']
	667	+ _, repo_name = os.path.split(subrepo)
	668	+ aliased_repo = alias + ':' + repo_name
	669	+ element['sources'][0]['submodules']['defined']['url'] = aliased_repo
	670	+
	671	+ _yaml.dump(element, element_path)
	672	+
	673	+ full_mirror = main_mirror.source_config()['url']
	674	+ mirror_map, _ = os.path.split(full_mirror)
	675	+ project = {
	676	+ 'name': 'test',
	677	+ 'element-path': 'elements',
	678	+ 'aliases': {
	679	+ alias: 'http://www.example.com/'
	680	+ },
	681	+ 'mirrors': [
	682	+ {
	683	+ 'name': 'middle-earth',
	684	+ 'aliases': {
	685	+ alias: [mirror_map + "/"],
	686	+ },
	687	+ },
	688	+ ]
	689	+ }
	690	+ project_file = os.path.join(project_dir, 'project.conf')
	691	+ _yaml.dump(project, project_file)
	692	+
	693	+ result = cli.run(project=project_dir, args=['fetch', element_name])
	694	+ result.assert_success()
	695	+
	696	+
	697	+@pytest.mark.datafiles(DATA_DIR)
	698	+def test_mirror_fallback_git_only_submodules(cli, tmpdir, datafiles):
	699	+ # Main repo has no mirror or alias.
	700	+ # One submodule is overridden to use a mirror.
	701	+ # There is another submodules not overriden.
	702	+ # Upstream for overriden submodule is down.
	703	+ #
	704	+ # We expect:
	705	+ # - overriden submodule is fetched from mirror.
	706	+ # - other submodule is fetched.
	707	+
	708	+ bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
	709	+ dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
	710	+
	711	+ upstream_bin_repodir = os.path.join(str(tmpdir), 'bin-upstream')
	712	+ mirror_bin_repodir = os.path.join(str(tmpdir), 'bin-mirror')
	713	+ upstream_bin_repo = create_repo('git', upstream_bin_repodir)
	714	+ upstream_bin_repo.create(bin_files_path)
	715	+ mirror_bin_repo = upstream_bin_repo.copy(mirror_bin_repodir)
	716	+
	717	+ dev_repodir = os.path.join(str(tmpdir), 'dev-upstream')
	718	+ dev_repo = create_repo('git', dev_repodir)
	719	+ dev_repo.create(dev_files_path)
	720	+
	721	+ main_files = os.path.join(str(tmpdir), 'main-files')
	722	+ os.makedirs(main_files)
	723	+ with open(os.path.join(main_files, 'README'), 'w') as f:
	724	+ f.write("TEST\n")
	725	+ main_repodir = os.path.join(str(tmpdir), 'main-upstream')
	726	+ main_repo = create_repo('git', main_repodir)
	727	+ main_repo.create(main_files)
	728	+
	729	+ upstream_url = 'file://{}'.format(upstream_bin_repo.repo)
	730	+ main_repo.add_submodule('bin', url=upstream_url)
	731	+ main_repo.add_submodule('dev', url='file://{}'.format(dev_repo.repo))
	732	+ # Unlist 'dev'.
	733	+ del main_repo.submodules['dev']
	734	+
	735	+ main_ref = main_repo.latest_commit()
	736	+
	737	+ upstream_map, repo_name = os.path.split(upstream_url)
	738	+ alias = 'foo'
	739	+ aliased_repo = '{}:{}'.format(alias, repo_name)
	740	+ main_repo.submodules['bin']['url'] = aliased_repo
	741	+
	742	+ full_mirror = mirror_bin_repo.source_config()['url']
	743	+ mirror_map, _ = os.path.split(full_mirror)
	744	+
	745	+ project_dir = os.path.join(str(tmpdir), 'project')
	746	+ os.makedirs(project_dir)
	747	+ element_dir = os.path.join(project_dir, 'elements')
	748	+
	749	+ element = {
	750	+ 'kind': 'import',
	751	+ 'sources': [
	752	+ main_repo.source_config(ref=main_ref, checkout_submodules=True)
	753	+ ]
	754	+ }
	755	+ element_name = 'test.bst'
	756	+ element_path = os.path.join(element_dir, element_name)
	757	+ os.makedirs(element_dir)
	758	+ _yaml.dump(element, element_path)
	759	+
	760	+ project = {
	761	+ 'name': 'test',
	762	+ 'element-path': 'elements',
	763	+ 'aliases': {
	764	+ alias: upstream_map + "/"
	765	+ },
	766	+ 'mirrors': [
	767	+ {
	768	+ 'name': 'middle-earth',
	769	+ 'aliases': {
	770	+ alias: [mirror_map + "/"],
	771	+ }
	772	+ }
	773	+ ]
	774	+ }
	775	+ project_file = os.path.join(project_dir, 'project.conf')
	776	+ _yaml.dump(project, project_file)
	777	+
	778	+ # Now make the upstream unavailable.
	779	+ os.rename(upstream_bin_repo.repo, '{}.bak'.format(upstream_bin_repo.repo))
	780	+ result = cli.run(project=project_dir, args=['fetch', element_name])
	781	+ result.assert_success()
	782	+
	783	+ result = cli.run(project=project_dir, args=['build', element_name])
	784	+ result.assert_success()
	785	+
	786	+ checkout = os.path.join(str(tmpdir), 'checkout')
	787	+ result = cli.run(project=project_dir, args=['checkout', element_name, checkout])
	788	+ result.assert_success()
	789	+
	790	+ assert os.path.exists(os.path.join(checkout, 'bin', 'bin', 'hello'))
	791	+ assert os.path.exists(os.path.join(checkout, 'dev', 'include', 'pony.h'))
	792	+
	793	+
	794	+@pytest.mark.datafiles(DATA_DIR)
	795	+def test_mirror_fallback_git_with_submodules(cli, tmpdir, datafiles):
	796	+ # Main repo has mirror. But does not list submodules.
	797	+ #
	798	+ # We expect:
	799	+ # - we will fetch submodules anyway
	800	+
	801	+ bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
	802	+ dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
	803	+
	804	+ bin_repodir = os.path.join(str(tmpdir), 'bin-repo')
	805	+ bin_repo = create_repo('git', bin_repodir)
	806	+ bin_repo.create(bin_files_path)
	807	+
	808	+ dev_repodir = os.path.join(str(tmpdir), 'dev-repo')
	809	+ dev_repo = create_repo('git', dev_repodir)
	810	+ dev_repo.create(dev_files_path)
	811	+
	812	+ main_files = os.path.join(str(tmpdir), 'main-files')
	813	+ os.makedirs(main_files)
	814	+ with open(os.path.join(main_files, 'README'), 'w') as f:
	815	+ f.write("TEST\n")
	816	+ upstream_main_repodir = os.path.join(str(tmpdir), 'main-upstream')
	817	+ upstream_main_repo = create_repo('git', upstream_main_repodir)
	818	+ upstream_main_repo.create(main_files)
	819	+
	820	+ upstream_main_repo.add_submodule('bin', url='file://{}'.format(bin_repo.repo))
	821	+ upstream_main_repo.add_submodule('dev', url='file://{}'.format(dev_repo.repo))
	822	+ # Unlist submodules.
	823	+ del upstream_main_repo.submodules['bin']
	824	+ del upstream_main_repo.submodules['dev']
	825	+
	826	+ upstream_main_ref = upstream_main_repo.latest_commit()
	827	+
	828	+ mirror_main_repodir = os.path.join(str(tmpdir), 'main-mirror')
	829	+ mirror_main_repo = upstream_main_repo.copy(mirror_main_repodir)
	830	+
	831	+ upstream_url = mirror_main_repo.source_config()['url']
	832	+
	833	+ upstream_map, repo_name = os.path.split(upstream_url)
	834	+ alias = 'foo'
	835	+ aliased_repo = '{}:{}'.format(alias, repo_name)
	836	+
	837	+ full_mirror = mirror_main_repo.source_config()['url']
	838	+ mirror_map, _ = os.path.split(full_mirror)
	839	+
	840	+ project_dir = os.path.join(str(tmpdir), 'project')
	841	+ os.makedirs(project_dir)
	842	+ element_dir = os.path.join(project_dir, 'elements')
	843	+
	844	+ element = {
	845	+ 'kind': 'import',
	846	+ 'sources': [
	847	+ upstream_main_repo.source_config(ref=upstream_main_ref, checkout_submodules=True)
	848	+ ]
	849	+ }
	850	+ element['sources'][0]['url'] = aliased_repo
	851	+ element_name = 'test.bst'
	852	+ element_path = os.path.join(element_dir, element_name)
	853	+ os.makedirs(element_dir)
	854	+ _yaml.dump(element, element_path)
	855	+
	856	+ project = {
	857	+ 'name': 'test',
	858	+ 'element-path': 'elements',
	859	+ 'aliases': {
	860	+ alias: upstream_map + "/"
	861	+ },
	862	+ 'mirrors': [
	863	+ {
	864	+ 'name': 'middle-earth',
	865	+ 'aliases': {
	866	+ alias: [mirror_map + "/"],
	867	+ }
	868	+ }
	869	+ ]
	870	+ }
	871	+ project_file = os.path.join(project_dir, 'project.conf')
	872	+ _yaml.dump(project, project_file)
	873	+
	874	+ # Now make the upstream unavailable.
	875	+ os.rename(upstream_main_repo.repo, '{}.bak'.format(upstream_main_repo.repo))
	876	+ result = cli.run(project=project_dir, args=['fetch', element_name])
	877	+ result.assert_success()
	878	+
	879	+ result = cli.run(project=project_dir, args=['build', element_name])
	880	+ result.assert_success()
	881	+
	882	+ checkout = os.path.join(str(tmpdir), 'checkout')
	883	+ result = cli.run(project=project_dir, args=['checkout', element_name, checkout])
	884	+ result.assert_success()
	885	+
	886	+ assert os.path.exists(os.path.join(checkout, 'bin', 'bin', 'hello'))
	887	+ assert os.path.exists(os.path.join(checkout, 'dev', 'include', 'pony.h'))

tests/frontend/project/files/bar

---

tests/frontend/project/files/foo

---

tests/sources/git.py

---

	1	+#
	2	+# Copyright (C) 2018 Codethink Limited
	3	+# Copyright (C) 2018 Bloomberg Finance LP
	4	+#
	5	+# This program is free software; you can redistribute it and/or
	6	+# modify it under the terms of the GNU Lesser General Public
	7	+# License as published by the Free Software Foundation; either
	8	+# version 2 of the License, or (at your option) any later version.
	9	+#
	10	+# This library is distributed in the hope that it will be useful,
	11	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	12	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	13	+# Lesser General Public License for more details.
	14	+#
	15	+# You should have received a copy of the GNU Lesser General Public
	16	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	17	+#
	18	+# Authors: Tristan Van Berkom <tristan vanberkom codethink co uk>
	19	+# Jonathan Maw <jonathan maw codethink co uk>
	20	+# William Salmon <will salmon codethink co uk>
	21	+#
	22	+
1	23	import os
2	24	import pytest
3	25
...	...	@@ -359,3 +381,30 @@ def test_submodule_track_ignore_inconsistent(cli, tmpdir, datafiles):
359	381
360	382	# Assert that we are just fine without it, and emit a warning to the user.
361	383	assert "Ignoring inconsistent submodule" in result.stderr
	384	+
	385	+
	386	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	387	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	388	+def test_submodule_track_no_ref_or_track(cli, tmpdir, datafiles):
	389	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	390	+
	391	+ # Create the repo from 'repofiles' subdir
	392	+ repo = create_repo('git', str(tmpdir))
	393	+ ref = repo.create(os.path.join(project, 'repofiles'))
	394	+
	395	+ # Write out our test target
	396	+ gitsource = repo.source_config(ref=None)
	397	+ gitsource.pop('track')
	398	+ element = {
	399	+ 'kind': 'import',
	400	+ 'sources': [
	401	+ gitsource
	402	+ ]
	403	+ }
	404	+
	405	+ _yaml.dump(element, os.path.join(project, 'target.bst'))
	406	+
	407	+ # Track will encounter an inconsistent submodule without any ref
	408	+ result = cli.run(project=project, args=['show', 'target.bst'])
	409	+ result.assert_main_error(ErrorDomain.SOURCE, "missing-track-and-ref")
	410	+ result.assert_task_error(None, None)

tests/sources/ostree.py

---

	1	+#
	2	+# Copyright (C) 2018 Bloomberg Finance LP
	3	+#
	4	+# This program is free software; you can redistribute it and/or
	5	+# modify it under the terms of the GNU Lesser General Public
	6	+# License as published by the Free Software Foundation; either
	7	+# version 2 of the License, or (at your option) any later version.
	8	+#
	9	+# This library is distributed in the hope that it will be useful,
	10	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	11	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	12	+# Lesser General Public License for more details.
	13	+#
	14	+# You should have received a copy of the GNU Lesser General Public
	15	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	16	+#
	17	+# Authors: William Salmon <will salmon codethink co uk>
	18	+#
	19	+
	20	+import os
	21	+import pytest
	22	+
	23	+from buildstream._exceptions import ErrorDomain
	24	+from buildstream import _yaml
	25	+
	26	+from tests.testutils import cli, create_repo
	27	+
	28	+DATA_DIR = os.path.join(
	29	+ os.path.dirname(os.path.realpath(__file__)),
	30	+ 'ostree',
	31	+)
	32	+
	33	+
	34	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	35	+def test_submodule_track_no_ref_or_track(cli, tmpdir, datafiles):
	36	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	37	+
	38	+ # Create the repo from 'repofiles' subdir
	39	+ repo = create_repo('ostree', str(tmpdir))
	40	+ ref = repo.create(os.path.join(project, 'repofiles'))
	41	+
	42	+ # Write out our test target
	43	+ ostreesource = repo.source_config(ref=None)
	44	+ ostreesource.pop('track')
	45	+ element = {
	46	+ 'kind': 'import',
	47	+ 'sources': [
	48	+ ostreesource
	49	+ ]
	50	+ }
	51	+
	52	+ _yaml.dump(element, os.path.join(project, 'target.bst'))
	53	+
	54	+ # Track will encounter an inconsistent submodule without any ref
	55	+ result = cli.run(project=project, args=['show', 'target.bst'])
	56	+ result.assert_main_error(ErrorDomain.SOURCE, "missing-track-and-ref")
	57	+ result.assert_task_error(None, None)

tests/sources/ostree/template/project.conf

---

	1	+# Basic project
	2	+name: foo

tests/sources/ostree/template/repofiles/file

---

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.