[Notes] [Git][BuildStream/buildstream][jjardon/fedora_29] 23 commits: _artifactcache.py: don't leak the project specific remote caches

[Javier Jardón <gitlab mg gitlab com>]

Title: GitLab

Javier Jardón pushed to branch jjardon/fedora_29 at BuildStream / buildstream

Commits:

• 1ee4a4ba
  by Abderrahim Kitouni at 2019-02-08T16:42:23Z

  _artifactcache.py: don't leak the project specific remote caches
  
  the code for initializing remotes added the project specific remote
  caches to the global list instead of making a copy.
  
  Fixes #618
  

• 24c0de16
  by Abderrahim Kitouni at 2019-02-08T16:42:23Z

  _project.py: use artifact caches from the parent project for junctions
  
  This makes a junction use the artifact cache of the parent project
  before the ones defined for the junction
  
  Fixes #401
  

• a937f99a
  by Javier Jardón at 2019-02-08T20:29:05Z

  Merge branch 'abderrahim/artifact-cache-junction' into 'master'
  
  Use artifact cache specs from the parent project before those defined in junctions
  
  Closes #618 and #401
  
  See merge request BuildStream/buildstream!1113

• da1560e4
  by Javier Jardón at 2019-02-09T09:10:01Z

  .gitlab-ci.yml: Remove testing on Fedora 27
  
  Fedora 27 is EOL since 2018-11-30
  
  See https://fedoraproject.org/wiki/End_of_life
  

• 79fbab61
  by Jürg Billeter at 2019-02-09T10:30:20Z

  Merge branch 'jjardon/fedora_27' into 'master'
  
  .gitlab-ci.yml: Remove testing on Fedora 27
  
  See merge request BuildStream/buildstream!1136

• 354c563d
  by Jürg Billeter at 2019-02-10T20:03:46Z

  import.py: Validate config node
  
  Fixes #662.
  

• 5e1be71f
  by Jürg Billeter at 2019-02-11T05:10:56Z

  Merge branch 'juerg/import' into 'master'
  
  import.py: Validate config node
  
  Closes #662
  
  See merge request BuildStream/buildstream!1141

• 99e1be45
  by Jürg Billeter at 2019-02-11T05:12:25Z

  local.py: Do not follow symlinks in local directories
  
  isdir() follows symlinks on the host, resulting in potential host
  contamination. This change reorders the file checks to avoid this issue.
  

• f95e222e
  by Jürg Billeter at 2019-02-11T05:12:25Z

  sandbox/sandbox.py: Do not follow symlinks in _has_command()
  
  This is required to ensure symlinks are not resolved on the host.
  

• 89973fb3
  by Jürg Billeter at 2019-02-11T05:12:25Z

  utils.py: Remove list_dirs parameter from list_relative_paths()
  
  list_dirs was always True in the BuildStream code base. There was also a
  bug in the list_dirs=False code path as it did not return symlinks in
  `dirnames`.
  
  This is an API break, however, there are no known external callers.
  

• d1da3fb0
  by Jürg Billeter at 2019-02-11T05:12:25Z

  utils.py: Fix sorting of symlinks to directories
  
  os.walk() resolves symlinks to check whether they point to a directory
  even when followlinks is set to False. We already work around that
  broken behavior by extracting symlinks from `dirnames`. However, the
  sort order was still incorrect as we returned symlinks in dirnames
  before files and other symlinks. This change fixes this, sorting all
  files and symlinks in a single list.
  

• 7cf67ed3
  by Jürg Billeter at 2019-02-11T05:12:25Z

  _casbaseddirectory.py: Do not mimic os.walk() in list_relative_paths()
  
  This matches the change in utils.list_relative_paths() that now sorts
  all symlinks as files, instead of following the broken behavior of
  os.walk().
  

• 7ec0bb5e
  by Jürg Billeter at 2019-02-11T05:44:20Z

  tests/sources/local.py: Add directory symlink test
  

• c07cc967
  by Jürg Billeter at 2019-02-11T07:13:28Z

  Merge branch 'juerg/symlinks' into 'master'
  
  Symlink fixes
  
  See merge request BuildStream/buildstream!1138

• 9e3d3e05
  by Angelos Evripiotis at 2019-02-11T07:14:50Z

  contributing: snakeviz replaces pyflame+flamegraph
  
  Replace the instructions for pyflame+flamegraph with simpler ones for
  snakeviz. For our general use-case this seems to be easier and better.
  Usage of this tool was demonstrated at the 2019 BuildStream Gathering in
  January by Daniel Silverstone, when presenting the aggregate results of
  profiling on many target environments.
  
  Here is the relevant mailing list thread:
  
  "Profiling before the gathering"
  https://mail.gnome.org/archives/buildstream-list/2019-January/msg00057.html
  

• bb6a692d
  by Jürg Billeter at 2019-02-11T09:20:34Z

  Merge branch 'snakeviz' into 'master'
  
  contributing: snakeviz replaces pyflame+flamegraph
  
  See merge request BuildStream/buildstream!1129

• 1ed63e54
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: better provenance on recursive include
  
  Use the provenance of the include block, instead of the whole node.
  

• 02e48209
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: better error on missing include
  
  Previously, a missing include would result in an error like this:
  
      Could not find file at not-a-file.include
  
  Note that the file containing the include was not mentioned.
  
  Now we get an error like this instead:
  
      element.bst [line 7 column 5]: Include block references a file that
      could not be found: 'not-a-file.include'.
  

• 4336e3bf
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: better error on including directory
  
  Previously, include a directory result in an error like this:
  
      mydir is a directory. bst command expects a .bst file.
  
  Note that the file containing the include was not mentioned.
  
  Now we get an error like this instead:
  
      element.bst [line 12 column 0]: Include block references a
      directory instead of a file: 'mydir'.
  

• 3f6c5000
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: re-use file_path variable
  
  Avoid an unnecessary call to os.path.join().
  

• adde0c94
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  tests/format/include: remove unused tmpdir's
  
  Don't create and remove temp dirs unnecessarily when they are not used,
  looks like these were just copy-pastes without intended side-effects.
  

• a66f8379
  by Jürg Billeter at 2019-02-11T13:52:54Z

  Merge branch 'aevri/include-error' into 'master'
  
  More user-friendly reporting on include errors
  
  See merge request BuildStream/buildstream!891

• 48c0a59b
  by Javier Jardón at 2019-02-11T16:23:48Z

  .gitlab-ci.yml: Test with current fedora release: 29
  

12 changed files:

• .gitlab-ci.yml
• CONTRIBUTING.rst
• buildstream/_artifactcache.py
• buildstream/_includes.py
• buildstream/_project.py
• buildstream/plugins/elements/import.py
• buildstream/plugins/sources/local.py
• buildstream/sandbox/sandbox.py
• buildstream/storage/_casbaseddirectory.py
• buildstream/utils.py
• tests/format/include.py
• tests/sources/local.py

Changes:

.gitlab-ci.yml

---

...	...	@@ -49,14 +49,14 @@ tests-debian-9:
49	49	image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-debian:9-master-46405991
50	50	<<: *tests
51	51
52		-tests-fedora-27:
53		- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:27-master-46405991
54		- <<: *tests
55		-
56	52	tests-fedora-28:
57	53	image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:28-master-46405991
58	54	<<: *tests
59	55
	56	+tests-fedora-29:
	57	+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:29-jjardon-fedora29-46745295
	58	+ <<: *tests
	59	+
60	60	tests-ubuntu-18.04:
61	61	image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-ubuntu:18.04-master-46405991
62	62	<<: *tests
...	...	@@ -75,8 +75,8 @@ tests-centos-7.6:
75	75	<<: *tests
76	76	image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-centos:7.6.1810-master-46405991
77	77
78		-overnight-fedora-28-aarch64:
79		- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:aarch64-28-master-46405991
	78	+overnight-fedora-29-aarch64:
	79	+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:aarch64-29-master-46405991
80	80	tags:
81	81	- aarch64
82	82	<<: *tests
...	...	@@ -95,7 +95,7 @@ overnight-fedora-28-aarch64:
95	95	tests-unix:
96	96	# Use fedora here, to a) run a test on fedora and b) ensure that we
97	97	# can get rid of ostree - this is not possible with debian-8
98		- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:27-master-46405991
	98	+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:29-jjardon-fedora29-46745295
99	99	<<: *tests
100	100	variables:
101	101	BST_FORCE_BACKEND: "unix"
...	...	@@ -113,7 +113,7 @@ tests-unix:
113	113
114	114	tests-fedora-missing-deps:
115	115	# Ensure that tests behave nicely while missing bwrap and ostree
116		- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:28-master-46405991
	116	+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:29-jjardon-fedora29-46745295
117	117	<<: *tests
118	118
119	119	script:
...	...	@@ -132,7 +132,7 @@ tests-fedora-update-deps:
132	132	# Check if the tests pass after updating requirements to their latest
133	133	# allowed version.
134	134	allow_failure: true
135		- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:28-master-46405991
	135	+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:29-jjardon-fedora29-46745295
136	136	<<: *tests
137	137
138	138	script:
...	...	@@ -291,8 +291,8 @@ coverage:
291	291	- cp -a .coverage-reports/ ./coverage-report
292	292	dependencies:
293	293	- tests-debian-9
294		- - tests-fedora-27
295	294	- tests-fedora-28
	295	+ - tests-fedora-29
296	296	- tests-fedora-missing-deps
297	297	- tests-ubuntu-18.04
298	298	- tests-unix

CONTRIBUTING.rst

---

...	...	@@ -1707,26 +1707,13 @@ You can then analyze the results interactively using the 'pstats' module:
1707	1707	For more detailed documentation of cProfile and 'pstats', see:
1708	1708	https://docs.python.org/3/library/profile.html.
1709	1709
1710		-For a richer visualisation of the callstack you can try `Pyflame
1711		-<https://github.com/uber/pyflame>`_. Once you have followed the instructions in
1712		-Pyflame's README to install the tool, you can profile `bst` commands as in the
1713		-following example:
	1710	+For a richer and interactive visualisation of the `.cprofile` files, you can
	1711	+try `snakeviz <http://jiffyclub.github.io/snakeviz/#interpreting-results>`_.
	1712	+You can install it with `pip install snakeviz`. Here is an example invocation:
1714	1713
1715		- pyflame --output bst.flame --trace bst --help
1716		-
1717		-You may see an `Unexpected ptrace(2) exception:` error. Note that the `bst`
1718		-operation will continue running in the background in this case, you will need
1719		-to wait for it to complete or kill it. Once this is done, rerun the above
1720		-command which appears to fix the issue.
1721		-
1722		-Once you have output from pyflame, you can use the ``flamegraph.pl`` script
1723		-from the `Flamegraph project <https://github.com/brendangregg/FlameGraph>`_
1724		-to generate an .svg image:
1725		-
1726		- ./flamegraph.pl bst.flame > bst-flamegraph.svg
1727		-
1728		-The generated SVG file can then be viewed in your preferred web browser.
	1714	+ snakeviz bst.cprofile
1729	1715
	1716	+It will then start a webserver and launch a browser to the relevant page.
1730	1717
1731	1718	Profiling specific parts of BuildStream with BST_PROFILE
1732	1719	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

buildstream/_artifactcache.py

---

...	...	@@ -467,7 +467,7 @@ class ArtifactCache():
467	467	# on_failure (callable): Called if we fail to contact one of the caches.
468	468	#
469	469	def initialize_remotes(self, *, on_failure=None):
470		- remote_specs = self.global_remote_specs
	470	+ remote_specs = list(self.global_remote_specs)
471	471
472	472	for project in self.project_remote_specs:
473	473	remote_specs += self.project_remote_specs[project]
...	...	@@ -1046,8 +1046,5 @@ class ArtifactCache():
1046	1046	# A list of ArtifactCacheSpec instances describing the remote artifact caches.
1047	1047	#
1048	1048	def _configured_remote_artifact_cache_specs(context, project):
1049		- project_overrides = context.get_overrides(project.name)
1050		- project_extra_specs = ArtifactCache.specs_from_config_node(project_overrides)
1051		-
1052	1049	return list(utils._deduplicate(
1053		- project_extra_specs + project.artifact_cache_specs + context.artifact_cache_specs))
	1050	+ project.artifact_cache_specs + context.artifact_cache_specs))

buildstream/_includes.py

---

...	...	@@ -40,19 +40,34 @@ class Includes:
40	40	includes = [_yaml.node_get(node, str, '(@)')]
41	41	else:
42	42	includes = _yaml.node_get(node, list, '(@)', default_value=None)
	43	+
	44	+ include_provenance = None
43	45	if '(@)' in node:
	46	+ include_provenance = _yaml.node_get_provenance(node, key='(@)')
44	47	del node['(@)']
45	48
46	49	if includes:
47	50	for include in reversed(includes):
48	51	if only_local and ':' in include:
49	52	continue
50		- include_node, file_path, sub_loader = self._include_file(include,
51		- current_loader)
	53	+ try:
	54	+ include_node, file_path, sub_loader = self._include_file(include,
	55	+ current_loader)
	56	+ except LoadError as e:
	57	+ if e.reason == LoadErrorReason.MISSING_FILE:
	58	+ message = "{}: Include block references a file that could not be found: '{}'.".format(
	59	+ include_provenance, include)
	60	+ raise LoadError(LoadErrorReason.MISSING_FILE, message) from e
	61	+ elif e.reason == LoadErrorReason.LOADING_DIRECTORY:
	62	+ message = "{}: Include block references a directory instead of a file: '{}'.".format(
	63	+ include_provenance, include)
	64	+ raise LoadError(LoadErrorReason.LOADING_DIRECTORY, message) from e
	65	+ else:
	66	+ raise
	67	+
52	68	if file_path in included:
53		- provenance = _yaml.node_get_provenance(node)
54	69	raise LoadError(LoadErrorReason.RECURSIVE_INCLUDE,
55		- "{}: trying to recursively include {}". format(provenance,
	70	+ "{}: trying to recursively include {}". format(include_provenance,
56	71	file_path))
57	72	# Because the included node will be modified, we need
58	73	# to copy it so that we do not modify the toplevel
...	...	@@ -101,7 +116,7 @@ class Includes:
101	116	file_path = os.path.join(directory, include)
102	117	key = (current_loader, file_path)
103	118	if key not in self._loaded:
104		- self._loaded[key] = _yaml.load(os.path.join(directory, include),
	119	+ self._loaded[key] = _yaml.load(file_path,
105	120	shortname=shortname,
106	121	project=project,
107	122	copy_tree=self._copy_tree)

buildstream/_project.py

---

...	...	@@ -549,7 +549,15 @@ class Project():
549	549	#
550	550
551	551	# Load artifacts pull/push configuration for this project
552		- self.artifact_cache_specs = ArtifactCache.specs_from_config_node(config, self.directory)
	552	+ project_specs = ArtifactCache.specs_from_config_node(config, self.directory)
	553	+ override_specs = ArtifactCache.specs_from_config_node(
	554	+ self._context.get_overrides(self.name), self.directory)
	555	+
	556	+ self.artifact_cache_specs = override_specs + project_specs
	557	+
	558	+ if self.junction:
	559	+ parent = self.junction._get_project()
	560	+ self.artifact_cache_specs = parent.artifact_cache_specs + self.artifact_cache_specs
553	561
554	562	# Load remote-execution configuration for this project
555	563	project_specs = SandboxRemote.specs_from_config_node(config, self.directory)

buildstream/plugins/elements/import.py

---

...	...	@@ -42,6 +42,10 @@ class ImportElement(Element):
42	42	BST_VIRTUAL_DIRECTORY = True
43	43
44	44	def configure(self, node):
	45	+ self.node_validate(node, [
	46	+ 'source', 'target'
	47	+ ])
	48	+
45	49	self.source = self.node_subst_member(node, 'source')
46	50	self.target = self.node_subst_member(node, 'target')
47	51

buildstream/plugins/sources/local.py

---

...	...	@@ -97,7 +97,7 @@ class LocalSource(Source):
97	97	with self.timed_activity("Staging local files at {}".format(self.path)):
98	98
99	99	if os.path.isdir(self.fullpath):
100		- files = list(utils.list_relative_paths(self.fullpath, list_dirs=True))
	100	+ files = list(utils.list_relative_paths(self.fullpath))
101	101	utils.copy_files(self.fullpath, directory, files=files)
102	102	else:
103	103	destfile = os.path.join(directory, os.path.basename(self.path))
...	...	@@ -133,11 +133,11 @@ def unique_key(filename):
133	133
134	134	# Return some hard coded things for files which
135	135	# have no content to calculate a key for
136		- if os.path.isdir(filename):
137		- return "0"
138		- elif os.path.islink(filename):
	136	+ if os.path.islink(filename):
139	137	# For a symbolic link, use the link target as its unique identifier
140	138	return os.readlink(filename)
	139	+ elif os.path.isdir(filename):
	140	+ return "0"
141	141
142	142	return utils.sha256sum(filename)
143	143

buildstream/sandbox/sandbox.py

---

...	...	@@ -525,11 +525,11 @@ class Sandbox():
525	525	# (bool): Whether a command exists inside the sandbox.
526	526	def _has_command(self, command, env=None):
527	527	if os.path.isabs(command):
528		- return os.path.exists(os.path.join(
	528	+ return os.path.lexists(os.path.join(
529	529	self._root, command.lstrip(os.sep)))
530	530
531	531	for path in env.get('PATH').split(':'):
532		- if os.path.exists(os.path.join(
	532	+ if os.path.lexists(os.path.join(
533	533	self._root, path.lstrip(os.sep), command)):
534	534	return True
535	535

buildstream/storage/_casbaseddirectory.py

---

...	...	@@ -795,24 +795,11 @@ class CasBasedDirectory(Directory):
795	795	Return value: List(str) - list of all paths
796	796	"""
797	797
798		- symlink_list = filter(lambda i: isinstance(i[1].pb_object, remote_execution_pb2.SymlinkNode),
799		- self.index.items())
800		- file_list = list(filter(lambda i: isinstance(i[1].pb_object, remote_execution_pb2.FileNode),
	798	+ file_list = list(filter(lambda i: not isinstance(i[1].buildstream_object, CasBasedDirectory),
801	799	self.index.items()))
802	800	directory_list = filter(lambda i: isinstance(i[1].buildstream_object, CasBasedDirectory),
803	801	self.index.items())
804	802
805		- # We need to mimic the behaviour of os.walk, in which symlinks
806		- # to directories count as directories and symlinks to file or
807		- # broken symlinks count as files. os.walk doesn't follow
808		- # symlinks, so we don't recurse.
809		- for (k, v) in sorted(symlink_list):
810		- target = self._resolve(k, absolute_symlinks_resolve=True)
811		- if isinstance(target, CasBasedDirectory):
812		- yield os.path.join(relpath, k)
813		- else:
814		- file_list.append((k, v))
815		-
816	803	if file_list == [] and relpath != "":
817	804	yield relpath
818	805	else:

buildstream/utils.py

---

...	...	@@ -111,7 +111,7 @@ class FileListResult():
111	111	return ret
112	112
113	113
114		-def list_relative_paths(directory, *, list_dirs=True):
	114	+def list_relative_paths(directory):
115	115	"""A generator for walking directory relative paths
116	116
117	117	This generator is useful for checking the full manifest of
...	...	@@ -125,13 +125,26 @@ def list_relative_paths(directory, *, list_dirs=True):
125	125
126	126	Args:
127	127	directory (str): The directory to list files in
128		- list_dirs (bool): Whether to list directories
129	128
130	129	Yields:
131	130	Relative filenames in `directory`
132	131	"""
133	132	for (dirpath, dirnames, filenames) in os.walk(directory):
134	133
	134	+ # os.walk does not decend into symlink directories, which
	135	+ # makes sense because otherwise we might have redundant
	136	+ # directories, or end up descending into directories outside
	137	+ # of the walk() directory.
	138	+ #
	139	+ # But symlinks to directories are still identified as
	140	+ # subdirectories in the walked `dirpath`, so we extract
	141	+ # these symlinks from `dirnames` and add them to `filenames`.
	142	+ #
	143	+ for d in dirnames:
	144	+ fullpath = os.path.join(dirpath, d)
	145	+ if os.path.islink(fullpath):
	146	+ filenames.append(d)
	147	+
135	148	# Modifying the dirnames directly ensures that the os.walk() generator
136	149	# allows us to specify the order in which they will be iterated.
137	150	dirnames.sort()
...	...	@@ -143,25 +156,10 @@ def list_relative_paths(directory, *, list_dirs=True):
143	156	# `directory`, prefer to have no prefix in that case.
144	157	basepath = relpath if relpath != '.' and dirpath != directory else ''
145	158
146		- # os.walk does not decend into symlink directories, which
147		- # makes sense because otherwise we might have redundant
148		- # directories, or end up descending into directories outside
149		- # of the walk() directory.
150		- #
151		- # But symlinks to directories are still identified as
152		- # subdirectories in the walked `dirpath`, so we extract
153		- # these symlinks from `dirnames`
154		- #
155		- if list_dirs:
156		- for d in dirnames:
157		- fullpath = os.path.join(dirpath, d)
158		- if os.path.islink(fullpath):
159		- yield os.path.join(basepath, d)
160		-
161	159	# We've decended into an empty directory, in this case we
162	160	# want to include the directory itself, but not in any other
163	161	# case.
164		- if list_dirs and not filenames:
	162	+ if not filenames:
165	163	yield relpath
166	164
167	165	# List the filenames in the walked directory

tests/format/include.py

---

1	1	import os
	2	+import textwrap
2	3	import pytest
3	4	from buildstream import _yaml
4	5	from buildstream._exceptions import ErrorDomain, LoadErrorReason
...	...	@@ -27,6 +28,46 @@ def test_include_project_file(cli, datafiles):
27	28	assert loaded['included'] == 'True'
28	29
29	30
	31	+def test_include_missing_file(cli, tmpdir):
	32	+ tmpdir.join('project.conf').write('{"name": "test"}')
	33	+ element = tmpdir.join('include_missing_file.bst')
	34	+
	35	+ # Normally we would use dicts and _yaml.dump to write such things, but here
	36	+ # we want to be sure of a stable line and column number.
	37	+ element.write(textwrap.dedent("""
	38	+ kind: manual
	39	+
	40	+ "(@)":
	41	+ - nosuch.yaml
	42	+ """).strip())
	43	+
	44	+ result = cli.run(project=str(tmpdir), args=['show', str(element.basename)])
	45	+ result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.MISSING_FILE)
	46	+ # Make sure the root cause provenance is in the output.
	47	+ assert 'line 4 column 2' in result.stderr
	48	+
	49	+
	50	+def test_include_dir(cli, tmpdir):
	51	+ tmpdir.join('project.conf').write('{"name": "test"}')
	52	+ tmpdir.mkdir('subdir')
	53	+ element = tmpdir.join('include_dir.bst')
	54	+
	55	+ # Normally we would use dicts and _yaml.dump to write such things, but here
	56	+ # we want to be sure of a stable line and column number.
	57	+ element.write(textwrap.dedent("""
	58	+ kind: manual
	59	+
	60	+ "(@)":
	61	+ - subdir/
	62	+ """).strip())
	63	+
	64	+ result = cli.run(project=str(tmpdir), args=['show', str(element.basename)])
	65	+ result.assert_main_error(
	66	+ ErrorDomain.LOAD, LoadErrorReason.LOADING_DIRECTORY)
	67	+ # Make sure the root cause provenance is in the output.
	68	+ assert 'line 4 column 2' in result.stderr
	69	+
	70	+
30	71	@pytest.mark.datafiles(DATA_DIR)
31	72	def test_include_junction_file(cli, tmpdir, datafiles):
32	73	project = os.path.join(str(datafiles), 'junction')
...	...	@@ -47,7 +88,7 @@ def test_include_junction_file(cli, tmpdir, datafiles):
47	88
48	89
49	90	@pytest.mark.datafiles(DATA_DIR)
50		-def test_include_junction_options(cli, tmpdir, datafiles):
	91	+def test_include_junction_options(cli, datafiles):
51	92	project = os.path.join(str(datafiles), 'options')
52	93
53	94	result = cli.run(project=project, args=[
...	...	@@ -128,7 +169,7 @@ def test_junction_element_not_partial_project_file(cli, tmpdir, datafiles):
128	169
129	170
130	171	@pytest.mark.datafiles(DATA_DIR)
131		-def test_include_element_overrides(cli, tmpdir, datafiles):
	172	+def test_include_element_overrides(cli, datafiles):
132	173	project = os.path.join(str(datafiles), 'overrides')
133	174
134	175	result = cli.run(project=project, args=[
...	...	@@ -143,7 +184,7 @@ def test_include_element_overrides(cli, tmpdir, datafiles):
143	184
144	185
145	186	@pytest.mark.datafiles(DATA_DIR)
146		-def test_include_element_overrides_composition(cli, tmpdir, datafiles):
	187	+def test_include_element_overrides_composition(cli, datafiles):
147	188	project = os.path.join(str(datafiles), 'overrides')
148	189
149	190	result = cli.run(project=project, args=[
...	...	@@ -158,7 +199,7 @@ def test_include_element_overrides_composition(cli, tmpdir, datafiles):
158	199
159	200
160	201	@pytest.mark.datafiles(DATA_DIR)
161		-def test_include_element_overrides_sub_include(cli, tmpdir, datafiles):
	202	+def test_include_element_overrides_sub_include(cli, datafiles):
162	203	project = os.path.join(str(datafiles), 'sub-include')
163	204
164	205	result = cli.run(project=project, args=[
...	...	@@ -192,7 +233,7 @@ def test_junction_do_not_use_included_overrides(cli, tmpdir, datafiles):
192	233
193	234
194	235	@pytest.mark.datafiles(DATA_DIR)
195		-def test_conditional_in_fragment(cli, tmpdir, datafiles):
	236	+def test_conditional_in_fragment(cli, datafiles):
196	237	project = os.path.join(str(datafiles), 'conditional')
197	238
198	239	result = cli.run(project=project, args=[
...	...	@@ -222,7 +263,7 @@ def test_inner(cli, datafiles):
222	263
223	264
224	265	@pytest.mark.datafiles(DATA_DIR)
225		-def test_recusive_include(cli, tmpdir, datafiles):
	266	+def test_recursive_include(cli, datafiles):
226	267	project = os.path.join(str(datafiles), 'recursive')
227	268
228	269	result = cli.run(project=project, args=[
...	...	@@ -231,6 +272,7 @@ def test_recusive_include(cli, tmpdir, datafiles):
231	272	'--format', '%{vars}',
232	273	'element.bst'])
233	274	result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.RECURSIVE_INCLUDE)
	275	+ assert 'line 2 column 2' in result.stderr
234	276
235	277
236	278	@pytest.mark.datafiles(DATA_DIR)

tests/sources/local.py

---

...	...	@@ -136,3 +136,23 @@ def test_stage_file_exists(cli, tmpdir, datafiles):
136	136	result = cli.run(project=project, args=['build', 'target.bst'])
137	137	result.assert_main_error(ErrorDomain.STREAM, None)
138	138	result.assert_task_error(ErrorDomain.SOURCE, 'ensure-stage-dir-fail')
	139	+
	140	+
	141	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'directory'))
	142	+def test_stage_directory_symlink(cli, tmpdir, datafiles):
	143	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	144	+ checkoutdir = os.path.join(str(tmpdir), "checkout")
	145	+
	146	+ symlink = os.path.join(project, 'files', 'symlink-to-subdir')
	147	+ os.symlink('subdir', symlink)
	148	+
	149	+ # Build, checkout
	150	+ result = cli.run(project=project, args=['build', 'target.bst'])
	151	+ result.assert_success()
	152	+ result = cli.run(project=project, args=['artifact', 'checkout', 'target.bst', '--directory', checkoutdir])
	153	+ result.assert_success()
	154	+
	155	+ # Check that the checkout contains the expected directory and directory symlink
	156	+ assert(os.path.exists(os.path.join(checkoutdir, 'subdir', 'anotherfile.txt')))
	157	+ assert(os.path.exists(os.path.join(checkoutdir, 'symlink-to-subdir', 'anotherfile.txt')))
	158	+ assert(os.path.islink(os.path.join(checkoutdir, 'symlink-to-subdir')))

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.