[Notes] [Git][BuildStream/buildstream][juerg/command-batching] 22 commits: utils.py: Ensure move_atomic booleans are keyword only arguments

[Jürg Billeter <gitlab mg gitlab com>]

Title: GitLab

Jürg Billeter pushed to branch juerg/command-batching at BuildStream / buildstream

Commits:

• a50c598e
  by Benjamin Schubert at 2018-11-21T09:24:15Z

  utils.py: Ensure move_atomic booleans are keyword only arguments
  
  This is to be coherent with the rest of the codebase
  

• b2f604b8
  by Benjamin Schubert at 2018-11-21T09:24:15Z

  utils.py: improve documentation of move_atomic
  

• daca0c01
  by Benjamin Schubert at 2018-11-21T09:52:42Z

  Merge branch 'bschubert/mr938-comments' into 'master'
  
  Followup on MR 938, addressing additional comments
  
  See merge request BuildStream/buildstream!958

• 050cd1bd
  by Tom Pollard at 2018-11-21T11:37:48Z

  tests/plugin/pipeline.py: Avoid using host user conf
  
  create_pipeline was passing no config parameter to context.load,
  leading to _context.py loading in the host's buildstream user conf
  when running tests locally, potentially leading to unwanted errors.
  

• 58353aad
  by Jürg Billeter at 2018-11-21T14:29:49Z

  Merge branch 'tpollard/pipelinehostconfig' into 'master'
  
  tests/plugin/pipeline.py: Avoid using host user conf
  
  See merge request BuildStream/buildstream!961

• 2022c3e8
  by William Salmon at 2018-11-21T15:53:03Z

  tests/frontend/workspace.py: Refactor support function
  
  The Refactor now makes it possible to test for the forth coming multi
  element workspace open commands and to test for failure.
  

• 7bdd44b2
  by William Salmon at 2018-11-21T15:53:03Z

  Workspace CLI update
  
  This is to update the workspace CLI to as agreed on the mailing list
  https://mail.gnome.org/archives/buildstream-list/2018-September/msg00046.html
  
  This patch also introduces the default workspace directory.
  

• 6612e708
  by William Salmon at 2018-11-21T15:53:03Z

  Added a news item for workspace open changes
  

• 4d6e2cd3
  by Will Salmon at 2018-11-21T16:20:26Z

  Merge branch 'willsalmon/defaultWorkspaces' into 'master'
  
  Updated Workspace CLI
  
  See merge request BuildStream/buildstream!897

• 07368d8d
  by Jürg Billeter at 2018-11-21T16:24:25Z

  tests/integration/manual.py: Add test for command logging
  

• 355b8920
  by Jürg Billeter at 2018-11-21T16:24:25Z

  sandbox: Deduplicate code to process cwd, env, and command arguments
  

• b7be26c2
  by Jürg Billeter at 2018-11-21T16:24:25Z

  sandbox/sandbox.py: Add command batching API
  
  This adds the batch() context manager.
  

• 2aa7ab2a
  by Jürg Billeter at 2018-11-21T16:24:25Z

  element.py: Pass element object to sandbox
  
  This allows the sandbox to output messages using the plugin id.
  

• be1663ba
  by Jürg Billeter at 2018-11-21T16:24:25Z

  element.py: Cache artifact also on SandboxCommandError
  

• bccb61a3
  by Jürg Billeter at 2018-11-21T16:24:25Z

  element.py: Defer workspace.prepared setting in case of pending commands
  

• f47294a0
  by Jürg Billeter at 2018-11-21T16:24:25Z

  element.py: Support command batching for integration commands
  

• 0c6079df
  by Jürg Billeter at 2018-11-21T16:24:25Z

  element.py: Add configure_prepare_assemble_batch() method
  
  This allows batching of all sandbox commands aross prepare() and
  assemble().
  

• 0df528f4
  by Jürg Billeter at 2018-11-21T16:24:25Z

  buildelement.py: Support batching for integration and build commands
  

• 4f0be43b
  by Jürg Billeter at 2018-11-21T16:24:25Z

  scriptelement.py: Support batching for integration and build commands
  

• 5f05cee6
  by Jürg Billeter at 2018-11-21T16:24:25Z

  compose.py: Support batching for integration commands
  

• 42568301
  by Jürg Billeter at 2018-11-21T16:24:25Z

  sandbox/_sandboxremote.py: Implement command batching
  
  Execute batched commands in a single shell script.
  

• 991fe02d
  by Jürg Billeter at 2018-11-21T16:24:42Z

  NEWS: Add entry for command batching
  

30 changed files:

• NEWS
• buildstream/__init__.py
• buildstream/_context.py
• buildstream/_frontend/cli.py
• buildstream/_stream.py
• buildstream/buildelement.py
• buildstream/data/userconfig.yaml
• buildstream/element.py
• buildstream/plugins/elements/compose.py
• buildstream/sandbox/__init__.py
• buildstream/sandbox/_sandboxbwrap.py
• buildstream/sandbox/_sandboxchroot.py
• buildstream/sandbox/_sandboxdummy.py
• buildstream/sandbox/_sandboxremote.py
• buildstream/sandbox/sandbox.py
• buildstream/scriptelement.py
• buildstream/utils.py
• doc/sessions/developing.run
• doc/sessions/junctions.run
• tests/examples/developing.py
• tests/examples/junctions.py
• tests/frontend/buildcheckout.py
• tests/frontend/cross_junction_workspace.py
• tests/frontend/workspace.py
• tests/integration/manual.py
• tests/integration/sandbox-bwrap.py
• tests/integration/shell.py
• tests/integration/workspace.py
• tests/plugins/filter.py
• tests/plugins/pipeline.py

Changes:

NEWS

---

...	...	@@ -63,6 +63,13 @@ buildstream 1.3.1
63	63
64	64	o Added new `bst source-checkout` command to checkout sources of an element.
65	65
	66	+ o `bst workspace open` now supports the creation of multiple elements and
	67	+ allows the user to set a default location for their creation. This has meant
	68	+ that the new CLI is no longer backwards compatible with buildstream 1.2.
	69	+
	70	+ o Add sandbox API for command batching and use it for build, script, and
	71	+ compose elements.
	72	+
66	73
67	74	=================
68	75	buildstream 1.1.5

buildstream/__init__.py

---

...	...	@@ -27,7 +27,7 @@ if "_BST_COMPLETION" not in os.environ:
27	27	del get_versions
28	28
29	29	from .utils import UtilError, ProgramNotFoundError
30		- from .sandbox import Sandbox, SandboxFlags
	30	+ from .sandbox import Sandbox, SandboxFlags, SandboxCommandError
31	31	from .types import Scope, Consistency
32	32	from .plugin import Plugin
33	33	from .source import Source, SourceError, SourceFetcher

buildstream/_context.py

---

...	...	@@ -59,6 +59,9 @@ class Context():
59	59	# The directory where build sandboxes will be created
60	60	self.builddir = None
61	61
	62	+ # Default root location for workspaces
	63	+ self.workspacedir = None
	64	+
62	65	# The local binary artifact cache directory
63	66	self.artifactdir = None
64	67
...	...	@@ -177,10 +180,10 @@ class Context():
177	180	_yaml.node_validate(defaults, [
178	181	'sourcedir', 'builddir', 'artifactdir', 'logdir',
179	182	'scheduler', 'artifacts', 'logging', 'projects',
180		- 'cache', 'prompt'
	183	+ 'cache', 'prompt', 'workspacedir',
181	184	])
182	185
183		- for directory in ['sourcedir', 'builddir', 'artifactdir', 'logdir']:
	186	+ for directory in ['sourcedir', 'builddir', 'artifactdir', 'logdir', 'workspacedir']:
184	187	# Allow the ~ tilde expansion and any environment variables in
185	188	# path specification in the config files.
186	189	#

buildstream/_frontend/cli.py

---

...	...	@@ -707,31 +707,23 @@ def workspace():
707	707	@click.option('--no-checkout', default=False, is_flag=True,
708	708	help="Do not checkout the source, only link to the given directory")
709	709	@click.option('--force', '-f', default=False, is_flag=True,
710		- help="Overwrite files existing in checkout directory")
	710	+ help="The workspace will be created even if the directory in which it will be created is not empty " +
	711	+ "or if a workspace for that element already exists")
711	712	@click.option('--track', 'track_', default=False, is_flag=True,
712	713	help="Track and fetch new source references before checking out the workspace")
713		-@click.argument('element',
714		- type=click.Path(readable=False))
715		-@click.argument('directory', type=click.Path(file_okay=False))
	714	+@click.option('--directory', type=click.Path(file_okay=False), default=None,
	715	+ help="Only for use when a single Element is given: Set the directory to use to create the workspace")
	716	+@click.argument('elements', nargs=-1, type=click.Path(readable=False), required=True)
716	717	@click.pass_obj
717		-def workspace_open(app, no_checkout, force, track_, element, directory):
	718	+def workspace_open(app, no_checkout, force, track_, directory, elements):
718	719	"""Open a workspace for manual source modification"""
719	720
720		- if os.path.exists(directory):
721		-
722		- if not os.path.isdir(directory):
723		- click.echo("Checkout directory is not a directory: {}".format(directory), err=True)
724		- sys.exit(-1)
725		-
726		- if not (no_checkout or force) and os.listdir(directory):
727		- click.echo("Checkout directory is not empty: {}".format(directory), err=True)
728		- sys.exit(-1)
729		-
730	721	with app.initialized():
731		- app.stream.workspace_open(element, directory,
	722	+ app.stream.workspace_open(elements,
732	723	no_checkout=no_checkout,
733	724	track_first=track_,
734		- force=force)
	725	+ force=force,
	726	+ custom_dir=directory)
735	727
736	728
737	729	##################################################################

buildstream/_stream.py

---

...	...	@@ -464,44 +464,30 @@ class Stream():
464	464	# Open a project workspace
465	465	#
466	466	# Args:
467		- # target (str): The target element to open the workspace for
468		- # directory (str): The directory to stage the source in
	467	+ # targets (list): List of target elements to open workspaces for
469	468	# no_checkout (bool): Whether to skip checking out the source
470	469	# track_first (bool): Whether to track and fetch first
471	470	# force (bool): Whether to ignore contents in an existing directory
	471	+ # custom_dir (str): Custom location to create a workspace or false to use default location.
472	472	#
473		- def workspace_open(self, target, directory, *,
	473	+ def workspace_open(self, targets, *,
474	474	no_checkout,
475	475	track_first,
476		- force):
	476	+ force,
	477	+ custom_dir):
	478	+ # This function is a little funny but it is trying to be as atomic as possible.
477	479
478	480	if track_first:
479		- track_targets = (target,)
	481	+ track_targets = targets
480	482	else:
481	483	track_targets = ()
482	484
483		- elements, track_elements = self._load((target,), track_targets,
	485	+ elements, track_elements = self._load(targets, track_targets,
484	486	selection=PipelineSelection.REDIRECT,
485	487	track_selection=PipelineSelection.REDIRECT)
486		- target = elements[0]
487		- directory = os.path.abspath(directory)
488		-
489		- if not list(target.sources()):
490		- build_depends = [x.name for x in target.dependencies(Scope.BUILD, recurse=False)]
491		- if not build_depends:
492		- raise StreamError("The given element has no sources")
493		- detail = "Try opening a workspace on one of its dependencies instead:\n"
494		- detail += " \n".join(build_depends)
495		- raise StreamError("The given element has no sources", detail=detail)
496	488
497	489	workspaces = self._context.get_workspaces()
498	490
499		- # Check for workspace config
500		- workspace = workspaces.get_workspace(target._get_full_name())
501		- if workspace and not force:
502		- raise StreamError("Workspace '{}' is already defined at: {}"
503		- .format(target.name, workspace.get_absolute_path()))
504		-
505	491	# If we're going to checkout, we need at least a fetch,
506	492	# if we were asked to track first, we're going to fetch anyway.
507	493	#
...	...	@@ -511,29 +497,88 @@ class Stream():
511	497	track_elements = elements
512	498	self._fetch(elements, track_elements=track_elements)
513	499
514		- if not no_checkout and target._get_consistency() != Consistency.CACHED:
515		- raise StreamError("Could not stage uncached source. " +
516		- "Use `--track` to track and " +
517		- "fetch the latest version of the " +
518		- "source.")
519		-
520		- if workspace:
521		- workspaces.delete_workspace(target._get_full_name())
522		- workspaces.save_config()
523		- shutil.rmtree(directory)
524		- try:
525		- os.makedirs(directory, exist_ok=True)
526		- except OSError as e:
527		- raise StreamError("Failed to create workspace directory: {}".format(e)) from e
	500	+ expanded_directories = []
	501	+ # To try to be more atomic, loop through the elements and raise any errors we can early
	502	+ for target in elements:
	503	+
	504	+ if not list(target.sources()):
	505	+ build_depends = [x.name for x in target.dependencies(Scope.BUILD, recurse=False)]
	506	+ if not build_depends:
	507	+ raise StreamError("The element {} has no sources".format(target.name))
	508	+ detail = "Try opening a workspace on one of its dependencies instead:\n"
	509	+ detail += " \n".join(build_depends)
	510	+ raise StreamError("The element {} has no sources".format(target.name), detail=detail)
	511	+
	512	+ # Check for workspace config
	513	+ workspace = workspaces.get_workspace(target._get_full_name())
	514	+ if workspace and not force:
	515	+ raise StreamError("Element '{}' already has workspace defined at: {}"
	516	+ .format(target.name, workspace.get_absolute_path()))
	517	+
	518	+ if not no_checkout and target._get_consistency() != Consistency.CACHED:
	519	+ raise StreamError("Could not stage uncached source. For {} ".format(target.name) +
	520	+ "Use `--track` to track and " +
	521	+ "fetch the latest version of the " +
	522	+ "source.")
	523	+
	524	+ if not custom_dir:
	525	+ directory = os.path.abspath(os.path.join(self._context.workspacedir, target.name))
	526	+ if directory[-4:] == '.bst':
	527	+ directory = directory[:-4]
	528	+ expanded_directories.append(directory)
	529	+
	530	+ if custom_dir:
	531	+ if len(elements) != 1:
	532	+ raise StreamError("Exactly one element can be given if --directory is used",
	533	+ reason='directory-with-multiple-elements')
	534	+ expanded_directories = [custom_dir, ]
	535	+ else:
	536	+ # If this fails it is a bug in what ever calls this, usually cli.py and so can not be tested for via the
	537	+ # run bst test mechanism.
	538	+ assert len(elements) == len(expanded_directories)
	539	+
	540	+ for target, directory in zip(elements, expanded_directories):
	541	+ if os.path.exists(directory):
	542	+ if not os.path.isdir(directory):
	543	+ raise StreamError("For element '{}', Directory path is not a directory: {}"
	544	+ .format(target.name, directory), reason='bad-directory')
	545	+
	546	+ if not (no_checkout or force) and os.listdir(directory):
	547	+ raise StreamError("For element '{}', Directory path is not empty: {}"
	548	+ .format(target.name, directory), reason='bad-directory')
	549	+
	550	+ # So far this function has tried to catch as many issues as possible with out making any changes
	551	+ # Now it dose the bits that can not be made atomic.
	552	+ targetGenerator = zip(elements, expanded_directories)
	553	+ for target, directory in targetGenerator:
	554	+ self._message(MessageType.INFO, "Creating workspace for element {}"
	555	+ .format(target.name))
	556	+
	557	+ workspace = workspaces.get_workspace(target._get_full_name())
	558	+ if workspace:
	559	+ workspaces.delete_workspace(target._get_full_name())
	560	+ workspaces.save_config()
	561	+ shutil.rmtree(directory)
	562	+ try:
	563	+ os.makedirs(directory, exist_ok=True)
	564	+ except OSError as e:
	565	+ todo_elements = " ".join([str(target.name) for target, directory_dict in targetGenerator])
	566	+ if todo_elements:
	567	+ # This output should make creating the remaining workspaces as easy as possible.
	568	+ todo_elements = "\nDid not try to create workspaces for " + todo_elements
	569	+ raise StreamError("Failed to create workspace directory: {}".format(e) + todo_elements) from e
528	570
529		- workspaces.create_workspace(target._get_full_name(), directory)
	571	+ workspaces.create_workspace(target._get_full_name(), directory)
530	572
531		- if not no_checkout:
532		- with target.timed_activity("Staging sources to {}".format(directory)):
533		- target._open_workspace()
	573	+ if not no_checkout:
	574	+ with target.timed_activity("Staging sources to {}".format(directory)):
	575	+ target._open_workspace()
534	576
535		- workspaces.save_config()
536		- self._message(MessageType.INFO, "Saved workspace configuration")
	577	+ # Saving the workspace once it is set up means that if the next workspace fails to be created before
	578	+ # the configuration gets saved. The successfully created workspace still gets saved.
	579	+ workspaces.save_config()
	580	+ self._message(MessageType.INFO, "Created a workspace for element: {}"
	581	+ .format(target._get_full_name()))
537	582
538	583	# workspace_close
539	584	#

buildstream/buildelement.py

---

...	...	@@ -127,7 +127,7 @@ artifact collection purposes.
127	127	"""
128	128
129	129	import os
130		-from . import Element, Scope, ElementError
	130	+from . import Element, Scope
131	131	from . import SandboxFlags
132	132
133	133
...	...	@@ -207,6 +207,10 @@ class BuildElement(Element):
207	207	# Setup environment
208	208	sandbox.set_environment(self.get_environment())
209	209
	210	+ # Enable command batching across prepare() and assemble()
	211	+ self.configure_prepare_assemble_batch(SandboxFlags.ROOT_READ_ONLY,
	212	+ collect=self.get_variable('install-root'))
	213	+
210	214	def stage(self, sandbox):
211	215
212	216	# Stage deps in the sandbox root
...	...	@@ -215,7 +219,7 @@ class BuildElement(Element):
215	219
216	220	# Run any integration commands provided by the dependencies
217	221	# once they are all staged and ready
218		- with self.timed_activity("Integrating sandbox"):
	222	+ with sandbox.batch(0, label="Integrating sandbox"):
219	223	for dep in self.dependencies(Scope.BUILD):
220	224	dep.integrate(sandbox)
221	225
...	...	@@ -223,14 +227,13 @@ class BuildElement(Element):
223	227	self.stage_sources(sandbox, self.get_variable('build-root'))
224	228
225	229	def assemble(self, sandbox):
226		-
227	230	# Run commands
228	231	for command_name in _command_steps:
229	232	commands = self.__commands[command_name]
230	233	if not commands or command_name == 'configure-commands':
231	234	continue
232	235
233		- with self.timed_activity("Running {}".format(command_name)):
	236	+ with sandbox.batch(SandboxFlags.ROOT_READ_ONLY, label="Running {}".format(command_name)):
234	237	for cmd in commands:
235	238	self.__run_command(sandbox, cmd, command_name)
236	239
...	...	@@ -254,7 +257,7 @@ class BuildElement(Element):
254	257	def prepare(self, sandbox):
255	258	commands = self.__commands['configure-commands']
256	259	if commands:
257		- with self.timed_activity("Running configure-commands"):
	260	+ with sandbox.batch(SandboxFlags.ROOT_READ_ONLY, label="Running configure-commands"):
258	261	for cmd in commands:
259	262	self.__run_command(sandbox, cmd, 'configure-commands')
260	263
...	...	@@ -282,13 +285,9 @@ class BuildElement(Element):
282	285	return commands
283	286
284	287	def __run_command(self, sandbox, cmd, cmd_name):
285		- self.status("Running {}".format(cmd_name), detail=cmd)
286		-
287	288	# Note the -e switch to 'sh' means to exit with an error
288	289	# if any untested command fails.
289	290	#
290		- exitcode = sandbox.run(['sh', '-c', '-e', cmd + '\n'],
291		- SandboxFlags.ROOT_READ_ONLY)
292		- if exitcode != 0:
293		- raise ElementError("Command '{}' failed with exitcode {}".format(cmd, exitcode),
294		- collect=self.get_variable('install-root'))
	291	+ sandbox.run(['sh', '-c', '-e', cmd + '\n'],
	292	+ SandboxFlags.ROOT_READ_ONLY,
	293	+ label=cmd)

buildstream/data/userconfig.yaml

---

...	...	@@ -22,6 +22,9 @@ artifactdir: ${XDG_CACHE_HOME}/buildstream/artifacts
22	22	# Location to store build logs
23	23	logdir: ${XDG_CACHE_HOME}/buildstream/logs
24	24
	25	+# Default root location for workspaces, blank for no default set.
	26	+workspacedir: .
	27	+
25	28	#
26	29	# Cache
27	30	#

buildstream/element.py

---

...	...	@@ -78,6 +78,7 @@ import stat
78	78	import copy
79	79	from collections import OrderedDict
80	80	from collections.abc import Mapping
	81	+import contextlib
81	82	from contextlib import contextmanager
82	83	import tempfile
83	84	import shutil
...	...	@@ -89,7 +90,7 @@ from ._exceptions import BstError, LoadError, LoadErrorReason, ImplError, \
89	90	ErrorDomain
90	91	from .utils import UtilError
91	92	from . import Plugin, Consistency, Scope
92		-from . import SandboxFlags
	93	+from . import SandboxFlags, SandboxCommandError
93	94	from . import utils
94	95	from . import _cachekey
95	96	from . import _signals
...	...	@@ -217,6 +218,10 @@ class Element(Plugin):
217	218	self.__build_result = None # The result of assembling this Element (success, description, detail)
218	219	self._build_log_path = None # The path of the build log for this Element
219	220
	221	+ self.__prepare_assemble_batch = False # Whether batching across prepare()/assemble() is configured
	222	+ self.__prepare_assemble_batch_flags = 0 # Sandbox flags for batching across prepare()/assemble()
	223	+ self.__prepare_assemble_batch_collect = None # Collect dir for batching across prepare()/assemble()
	224	+
220	225	# hash tables of loaded artifact metadata, hashed by key
221	226	self.__metadata_keys = {} # Strong and weak keys for this key
222	227	self.__metadata_dependencies = {} # Dictionary of dependency strong keys
...	...	@@ -770,13 +775,13 @@ class Element(Plugin):
770	775	environment = self.get_environment()
771	776
772	777	if bstdata is not None:
773		- commands = self.node_get_member(bstdata, list, 'integration-commands', [])
774		- for i in range(len(commands)):
775		- cmd = self.node_subst_list_element(bstdata, 'integration-commands', [i])
776		- self.status("Running integration command", detail=cmd)
777		- exitcode = sandbox.run(['sh', '-e', '-c', cmd], 0, env=environment, cwd='/')
778		- if exitcode != 0:
779		- raise ElementError("Command '{}' failed with exitcode {}".format(cmd, exitcode))
	778	+ with sandbox.batch(0):
	779	+ commands = self.node_get_member(bstdata, list, 'integration-commands', [])
	780	+ for i in range(len(commands)):
	781	+ cmd = self.node_subst_list_element(bstdata, 'integration-commands', [i])
	782	+
	783	+ sandbox.run(['sh', '-e', '-c', cmd], 0, env=environment, cwd='/',
	784	+ label=cmd)
780	785
781	786	def stage_sources(self, sandbox, directory):
782	787	"""Stage this element's sources to a directory in the sandbox
...	...	@@ -863,6 +868,24 @@ class Element(Plugin):
863	868
864	869	return None
865	870
	871	+ def configure_prepare_assemble_batch(self, flags, *, collect=None):
	872	+ """ Configure command batching across prepare() and assemble()
	873	+
	874	+ Args:
	875	+ flags (:class:`.SandboxFlags`): The sandbox flags for the command batch
	876	+ collect (str): An optional directory containing partial install contents
	877	+ on command failure.
	878	+
	879	+ This may be called in :func:`Element.configure_sandbox() <buildstream.element.Element.configure_sandbox>`
	880	+ to enable batching of all sandbox commands issued in prepare() and assemble().
	881	+ """
	882	+ if self.__prepare_assemble_batch:
	883	+ raise ElementError("{}: Command batching for prepare/assemble is already configured".format(self))
	884	+
	885	+ self.__prepare_assemble_batch = True
	886	+ self.__prepare_assemble_batch_flags = flags
	887	+ self.__prepare_assemble_batch_collect = collect
	888	+
866	889	#############################################################
867	890	# Private Methods used in BuildStream #
868	891	#############################################################
...	...	@@ -1323,7 +1346,7 @@ class Element(Plugin):
1323	1346	bare_directory=bare_directory) as sandbox:
1324	1347
1325	1348	# Configure always comes first, and we need it.
1326		- self.configure_sandbox(sandbox)
	1349	+ self.__configure_sandbox(sandbox)
1327	1350
1328	1351	# Stage something if we need it
1329	1352	if not directory:
...	...	@@ -1556,15 +1579,24 @@ class Element(Plugin):
1556	1579	# Call the abstract plugin methods
1557	1580	try:
1558	1581	# Step 1 - Configure
1559		- self.configure_sandbox(sandbox)
	1582	+ self.__configure_sandbox(sandbox)
1560	1583	# Step 2 - Stage
1561	1584	self.stage(sandbox)
1562		- # Step 3 - Prepare
1563		- self.__prepare(sandbox)
1564		- # Step 4 - Assemble
1565		- collect = self.assemble(sandbox) # pylint: disable=assignment-from-no-return
	1585	+
	1586	+ if self.__prepare_assemble_batch:
	1587	+ cm = sandbox.batch(self.__prepare_assemble_batch_flags,
	1588	+ collect=self.__prepare_assemble_batch_collect)
	1589	+ else:
	1590	+ cm = contextlib.suppress()
	1591	+
	1592	+ with cm:
	1593	+ # Step 3 - Prepare
	1594	+ self.__prepare(sandbox)
	1595	+ # Step 4 - Assemble
	1596	+ collect = self.assemble(sandbox) # pylint: disable=assignment-from-no-return
	1597	+
1566	1598	self.__set_build_result(success=True, description="succeeded")
1567		- except ElementError as e:
	1599	+ except (ElementError, SandboxCommandError) as e:
1568	1600	# Shelling into a sandbox is useful to debug this error
1569	1601	e.sandbox = True
1570	1602
...	...	@@ -2059,6 +2091,15 @@ class Element(Plugin):
2059	2091	def __can_build_incrementally(self):
2060	2092	return bool(self._get_workspace())
2061	2093
	2094	+ # __configure_sandbox():
	2095	+ #
	2096	+ # Internal method for calling public abstract configure_sandbox() method.
	2097	+ #
	2098	+ def __configure_sandbox(self, sandbox):
	2099	+ self.__prepare_assemble_batch = False
	2100	+
	2101	+ self.configure_sandbox(sandbox)
	2102	+
2062	2103	# __prepare():
2063	2104	#
2064	2105	# Internal method for calling public abstract prepare() method.
...	...	@@ -2074,7 +2115,12 @@ class Element(Plugin):
2074	2115	self.prepare(sandbox)
2075	2116
2076	2117	if workspace:
2077		- workspace.prepared = True
	2118	+ def mark_workspace_prepared():
	2119	+ workspace.prepared = True
	2120	+
	2121	+ # Defer workspace.prepared setting until pending batch commands
	2122	+ # have been executed.
	2123	+ sandbox._callback(mark_workspace_prepared)
2078	2124
2079	2125	def __is_cached(self, keystrength):
2080	2126	if keystrength is None:
...	...	@@ -2157,6 +2203,7 @@ class Element(Plugin):
2157	2203
2158	2204	sandbox = SandboxRemote(context, project,
2159	2205	directory,
	2206	+ plugin=self,
2160	2207	stdout=stdout,
2161	2208	stderr=stderr,
2162	2209	config=config,
...	...	@@ -2175,6 +2222,7 @@ class Element(Plugin):
2175	2222
2176	2223	sandbox = platform.create_sandbox(context, project,
2177	2224	directory,
	2225	+ plugin=self,
2178	2226	stdout=stdout,
2179	2227	stderr=stderr,
2180	2228	config=config,

buildstream/plugins/elements/compose.py

---

...	...	@@ -122,8 +122,9 @@ class ComposeElement(Element):
122	122	snapshot = set(vbasedir.list_relative_paths())
123	123	vbasedir.mark_unmodified()
124	124
125		- for dep in self.dependencies(Scope.BUILD):
126		- dep.integrate(sandbox)
	125	+ with sandbox.batch(0):
	126	+ for dep in self.dependencies(Scope.BUILD):
	127	+ dep.integrate(sandbox)
127	128
128	129	if require_split:
129	130	# Calculate added, modified and removed files

buildstream/sandbox/__init__.py

---

...	...	@@ -17,6 +17,6 @@
17	17	# Authors:
18	18	# Tristan Maat <tristan maat codethink co uk>
19	19
20		-from .sandbox import Sandbox, SandboxFlags
	20	+from .sandbox import Sandbox, SandboxFlags, SandboxCommandError
21	21	from ._sandboxremote import SandboxRemote
22	22	from ._sandboxdummy import SandboxDummy

buildstream/sandbox/_sandboxbwrap.py

---

...	...	@@ -58,22 +58,12 @@ class SandboxBwrap(Sandbox):
58	58	self.die_with_parent_available = kwargs['die_with_parent_available']
59	59	self.json_status_available = kwargs['json_status_available']
60	60
61		- def run(self, command, flags, *, cwd=None, env=None):
	61	+ def _run(self, command, flags, *, cwd, env):
62	62	stdout, stderr = self._get_output()
63	63
64	64	# Allowable access to underlying storage as we're part of the sandbox
65	65	root_directory = self.get_virtual_directory()._get_underlying_directory()
66	66
67		- # Fallback to the sandbox default settings for
68		- # the cwd and env.
69		- #
70		- cwd = self._get_work_directory(cwd=cwd)
71		- env = self._get_environment(cwd=cwd, env=env)
72		-
73		- # Convert single-string argument to a list
74		- if isinstance(command, str):
75		- command = [command]
76		-
77	67	if not self._has_command(command[0], env):
78	68	raise SandboxError("Staged artifacts do not provide command "
79	69	"'{}'".format(command[0]),

buildstream/sandbox/_sandboxchroot.py

---

...	...	@@ -49,17 +49,7 @@ class SandboxChroot(Sandbox):
49	49
50	50	self.mount_map = None
51	51
52		- def run(self, command, flags, *, cwd=None, env=None):
53		-
54		- # Fallback to the sandbox default settings for
55		- # the cwd and env.
56		- #
57		- cwd = self._get_work_directory(cwd=cwd)
58		- env = self._get_environment(cwd=cwd, env=env)
59		-
60		- # Convert single-string argument to a list
61		- if isinstance(command, str):
62		- command = [command]
	52	+ def _run(self, command, flags, *, cwd, env):
63	53
64	54	if not self._has_command(command[0], env):
65	55	raise SandboxError("Staged artifacts do not provide command "

buildstream/sandbox/_sandboxdummy.py

---

...	...	@@ -25,17 +25,7 @@ class SandboxDummy(Sandbox):
25	25	super().__init__(*args, **kwargs)
26	26	self._reason = kwargs.get("dummy_reason", "no reason given")
27	27
28		- def run(self, command, flags, *, cwd=None, env=None):
29		-
30		- # Fallback to the sandbox default settings for
31		- # the cwd and env.
32		- #
33		- cwd = self._get_work_directory(cwd=cwd)
34		- env = self._get_environment(cwd=cwd, env=env)
35		-
36		- # Convert single-string argument to a list
37		- if isinstance(command, str):
38		- command = [command]
	28	+ def _run(self, command, flags, *, cwd, env):
39	29
40	30	if not self._has_command(command[0], env):
41	31	raise SandboxError("Staged artifacts do not provide command "

buildstream/sandbox/_sandboxremote.py

---

...	...	@@ -19,12 +19,14 @@
19	19	# Jim MacArthur <jim macarthur codethink co uk>
20	20
21	21	import os
	22	+import shlex
22	23	from urllib.parse import urlparse
23	24	from functools import partial
24	25
25	26	import grpc
26	27
27		-from . import Sandbox
	28	+from . import Sandbox, SandboxCommandError
	29	+from .sandbox import _SandboxBatch
28	30	from ..storage._filebaseddirectory import FileBasedDirectory
29	31	from ..storage._casbaseddirectory import CasBasedDirectory
30	32	from .. import _signals
...	...	@@ -212,7 +214,7 @@ class SandboxRemote(Sandbox):
212	214	new_dir = CasBasedDirectory(self._get_context().artifactcache.cas, ref=dir_digest)
213	215	self._set_virtual_directory(new_dir)
214	216
215		- def run(self, command, flags, *, cwd=None, env=None):
	217	+ def _run(self, command, flags, *, cwd, env):
216	218	# Upload sources
217	219	upload_vdir = self.get_virtual_directory()
218	220
...	...	@@ -230,16 +232,6 @@ class SandboxRemote(Sandbox):
230	232	if not cascache.verify_digest_pushed(self._get_project(), upload_vdir.ref):
231	233	raise SandboxError("Failed to verify that source has been pushed to the remote artifact cache.")
232	234
233		- # Fallback to the sandbox default settings for
234		- # the cwd and env.
235		- #
236		- cwd = self._get_work_directory(cwd=cwd)
237		- env = self._get_environment(cwd=cwd, env=env)
238		-
239		- # We want command args as a list of strings
240		- if isinstance(command, str):
241		- command = [command]
242		-
243	235	# Now transmit the command to execute
244	236	operation = self.run_remote_command(command, upload_vdir.ref, cwd, env)
245	237
...	...	@@ -275,3 +267,69 @@ class SandboxRemote(Sandbox):
275	267	self.process_job_output(action_result.output_directories, action_result.output_files)
276	268
277	269	return 0
	270	+
	271	+ def _create_batch(self, main_group, flags, *, collect=None):
	272	+ return _SandboxRemoteBatch(self, main_group, flags, collect=collect)
	273	+
	274	+
	275	+# _SandboxRemoteBatch()
	276	+#
	277	+# Command batching by shell script generation.
	278	+#
	279	+class _SandboxRemoteBatch(_SandboxBatch):
	280	+
	281	+ def __init__(self, sandbox, main_group, flags, *, collect=None):
	282	+ super().__init__(sandbox, main_group, flags, collect=collect)
	283	+
	284	+ self.script = None
	285	+ self.first_command = None
	286	+ self.cwd = None
	287	+ self.env = None
	288	+
	289	+ def execute(self):
	290	+ self.script = ""
	291	+
	292	+ self.main_group.execute(self)
	293	+
	294	+ first = self.first_command
	295	+ if first and self.sandbox.run(['sh', '-c', '-e', self.script], self.flags, cwd=first.cwd, env=first.env) != 0:
	296	+ raise SandboxCommandError("Command execution failed", collect=self.collect)
	297	+
	298	+ def execute_group(self, group):
	299	+ group.execute_children(self)
	300	+
	301	+ def execute_command(self, command):
	302	+ if self.first_command is None:
	303	+ # First command in batch
	304	+ # Initial working directory and environment of script already matches
	305	+ # the command configuration.
	306	+ self.first_command = command
	307	+ else:
	308	+ # Change working directory for this command
	309	+ if command.cwd != self.cwd:
	310	+ self.script += "mkdir -p {}\n".format(command.cwd)
	311	+ self.script += "cd {}\n".format(command.cwd)
	312	+
	313	+ # Update environment for this command
	314	+ for key in self.env.keys():
	315	+ if key not in command.env:
	316	+ self.script += "unset {}\n".format(key)
	317	+ for key, value in command.env.items():
	318	+ if key not in self.env or self.env[key] != value:
	319	+ self.script += "export {}={}\n".format(key, shlex.quote(value))
	320	+
	321	+ # Keep track of current working directory and environment
	322	+ self.cwd = command.cwd
	323	+ self.env = command.env
	324	+
	325	+ # Actual command execution
	326	+ cmdline = ' '.join(shlex.quote(cmd) for cmd in command.command)
	327	+ self.script += "(set -ex; {})".format(cmdline)
	328	+
	329	+ # Error handling
	330	+ label = command.label or cmdline
	331	+ quoted_label = shlex.quote("'{}'".format(label))
	332	+ self.script += " || (echo Command {} failed with exitcode $? >&2 ; exit 1)\n".format(quoted_label)
	333	+
	334	+ def execute_call(self, call):
	335	+ raise SandboxError("SandboxRemote does not support callbacks in command batches")

buildstream/sandbox/sandbox.py

---

1	1	#
2	2	# Copyright (C) 2017 Codethink Limited
	3	+# Copyright (C) 2018 Bloomberg Finance LP
3	4	#
4	5	# This program is free software; you can redistribute it and/or
5	6	# modify it under the terms of the GNU Lesser General Public
...	...	@@ -29,7 +30,12 @@ See also: :ref:`sandboxing`.
29	30	"""
30	31
31	32	import os
32		-from .._exceptions import ImplError, BstError
	33	+import shlex
	34	+import contextlib
	35	+from contextlib import contextmanager
	36	+
	37	+from .._exceptions import ImplError, BstError, SandboxError
	38	+from .._message import Message, MessageType
33	39	from ..storage._filebaseddirectory import FileBasedDirectory
34	40	from ..storage._casbaseddirectory import CasBasedDirectory
35	41
...	...	@@ -71,6 +77,19 @@ class SandboxFlags():
71	77	"""
72	78
73	79
	80	+class SandboxCommandError(SandboxError):
	81	+ """Raised by :class:`.Sandbox` implementations when a command fails.
	82	+
	83	+ Args:
	84	+ message (str): The error message to report to the user
	85	+ collect (str): An optional directory containing partial install contents
	86	+ """
	87	+ def __init__(self, message, *, collect=None):
	88	+ super().__init__(message, reason='command-failed')
	89	+
	90	+ self.collect = collect
	91	+
	92	+
74	93	class Sandbox():
75	94	"""Sandbox()
76	95
...	...	@@ -94,6 +113,13 @@ class Sandbox():
94	113	self.__mount_sources = {}
95	114	self.__allow_real_directory = kwargs['allow_real_directory']
96	115
	116	+ # Plugin ID for logging
	117	+ plugin = kwargs.get('plugin', None)
	118	+ if plugin:
	119	+ self.__plugin_id = plugin._get_unique_id()
	120	+ else:
	121	+ self.__plugin_id = None
	122	+
97	123	# Configuration from kwargs common to all subclasses
98	124	self.__config = kwargs['config']
99	125	self.__stdout = kwargs['stdout']
...	...	@@ -121,6 +147,9 @@ class Sandbox():
121	147	# directory via get_directory.
122	148	self._never_cache_vdirs = False
123	149
	150	+ # Pending command batch
	151	+ self.__batch = None
	152	+
124	153	def get_directory(self):
125	154	"""Fetches the sandbox root directory
126	155
...	...	@@ -209,9 +238,16 @@ class Sandbox():
209	238	'artifact': artifact
210	239	})
211	240
212		- def run(self, command, flags, *, cwd=None, env=None):
	241	+ def run(self, command, flags, *, cwd=None, env=None, label=None):
213	242	"""Run a command in the sandbox.
214	243
	244	+ If this is called outside a batch context, the command is immediately
	245	+ executed.
	246	+
	247	+ If this is called in a batch context, the command is added to the batch
	248	+ for later execution. If the command fails, later commands will not be
	249	+ executed. Command flags must match batch flags.
	250	+
215	251	Args:
216	252	command (list): The command to run in the sandboxed environment, as a list
217	253	of strings starting with the binary to run.
...	...	@@ -219,9 +255,10 @@ class Sandbox():
219	255	cwd (str): The sandbox relative working directory in which to run the command.
220	256	env (dict): A dictionary of string key, value pairs to set as environment
221	257	variables inside the sandbox environment.
	258	+ label (str): An optional label for the command, used for logging. (*Since: 1.4*)
222	259
223	260	Returns:
224		- (int): The program exit code.
	261	+ (int|None): The program exit code, or None if running in batch context.
225	262
226	263	Raises:
227	264	(:class:`.ProgramNotFoundError`): If a host tool which the given sandbox
...	...	@@ -234,9 +271,115 @@ class Sandbox():
234	271	function must make sure the directory will be created if it does
235	272	not exist yet, even if a workspace is being used.
236	273	"""
237		- raise ImplError("Sandbox of type '{}' does not implement run()"
	274	+
	275	+ # Fallback to the sandbox default settings for
	276	+ # the cwd and env.
	277	+ #
	278	+ cwd = self._get_work_directory(cwd=cwd)
	279	+ env = self._get_environment(cwd=cwd, env=env)
	280	+
	281	+ # Convert single-string argument to a list
	282	+ if isinstance(command, str):
	283	+ command = [command]
	284	+
	285	+ if self.__batch:
	286	+ if flags != self.__batch.flags:
	287	+ raise SandboxError("Inconsistent sandbox flags in single command batch")
	288	+
	289	+ batch_command = _SandboxBatchCommand(command, cwd=cwd, env=env, label=label)
	290	+
	291	+ current_group = self.__batch.current_group
	292	+ current_group.append(batch_command)
	293	+ return None
	294	+ else:
	295	+ return self._run(command, flags, cwd=cwd, env=env)
	296	+
	297	+ @contextmanager
	298	+ def batch(self, flags, *, label=None, collect=None):
	299	+ """Context manager for command batching
	300	+
	301	+ This provides a batch context that defers execution of commands until
	302	+ the end of the context. If a command fails, the batch will be aborted
	303	+ and subsequent commands will not be executed.
	304	+
	305	+ Command batches may be nested. Execution will start only when the top
	306	+ level batch context ends.
	307	+
	308	+ Args:
	309	+ flags (:class:`.SandboxFlags`): The flags for this command batch.
	310	+ label (str): An optional label for the batch group, used for logging.
	311	+ collect (str): An optional directory containing partial install contents
	312	+ on command failure.
	313	+
	314	+ Raises:
	315	+ (:class:`.SandboxCommandError`): If a command fails.
	316	+
	317	+ *Since: 1.4*
	318	+ """
	319	+
	320	+ group = _SandboxBatchGroup(label=label)
	321	+
	322	+ if self.__batch:
	323	+ # Nested batch
	324	+ if flags != self.__batch.flags:
	325	+ raise SandboxError("Inconsistent sandbox flags in single command batch")
	326	+
	327	+ parent_group = self.__batch.current_group
	328	+ parent_group.append(group)
	329	+ self.__batch.current_group = group
	330	+ try:
	331	+ yield
	332	+ finally:
	333	+ self.__batch.current_group = parent_group
	334	+ else:
	335	+ # Top-level batch
	336	+ batch = self._create_batch(group, flags, collect=collect)
	337	+
	338	+ self.__batch = batch
	339	+ try:
	340	+ yield
	341	+ finally:
	342	+ self.__batch = None
	343	+
	344	+ batch.execute()
	345	+
	346	+ #####################################################
	347	+ # Abstract Methods for Sandbox implementations #
	348	+ #####################################################
	349	+
	350	+ # _run()
	351	+ #
	352	+ # Abstract method for running a single command
	353	+ #
	354	+ # Args:
	355	+ # command (list): The command to run in the sandboxed environment, as a list
	356	+ # of strings starting with the binary to run.
	357	+ # flags (:class:`.SandboxFlags`): The flags for running this command.
	358	+ # cwd (str): The sandbox relative working directory in which to run the command.
	359	+ # env (dict): A dictionary of string key, value pairs to set as environment
	360	+ # variables inside the sandbox environment.
	361	+ #
	362	+ # Returns:
	363	+ # (int): The program exit code.
	364	+ #
	365	+ def _run(self, command, flags, *, cwd, env):
	366	+ raise ImplError("Sandbox of type '{}' does not implement _run()"
238	367	.format(type(self).__name__))
239	368
	369	+ # _create_batch()
	370	+ #
	371	+ # Abstract method for creating a batch object. Subclasses can override
	372	+ # this method to instantiate a subclass of _SandboxBatch.
	373	+ #
	374	+ # Args:
	375	+ # main_group (:class:`_SandboxBatchGroup`): The top level batch group.
	376	+ # flags (:class:`.SandboxFlags`): The flags for commands in this batch.
	377	+ # collect (str): An optional directory containing partial install contents
	378	+ # on command failure.
	379	+ #
	380	+ def _create_batch(self, main_group, flags, *, collect=None):
	381	+ return _SandboxBatch(self, main_group, flags, collect=collect)
	382	+
240	383	################################################
241	384	# Private methods #
242	385	################################################
...	...	@@ -385,3 +528,138 @@ class Sandbox():
385	528	return True
386	529
387	530	return False
	531	+
	532	+ # _get_plugin_id()
	533	+ #
	534	+ # Get the plugin's unique identifier
	535	+ #
	536	+ def _get_plugin_id(self):
	537	+ return self.__plugin_id
	538	+
	539	+ # _callback()
	540	+ #
	541	+ # If this is called outside a batch context, the specified function is
	542	+ # invoked immediately.
	543	+ #
	544	+ # If this is called in a batch context, the function is added to the batch
	545	+ # for later invocation.
	546	+ #
	547	+ # Args:
	548	+ # callback (callable): The function to invoke
	549	+ #
	550	+ def _callback(self, callback):
	551	+ if self.__batch:
	552	+ batch_call = _SandboxBatchCall(callback)
	553	+
	554	+ current_group = self.__batch.current_group
	555	+ current_group.append(batch_call)
	556	+ else:
	557	+ callback()
	558	+
	559	+
	560	+# _SandboxBatch()
	561	+#
	562	+# A batch of sandbox commands.
	563	+#
	564	+class _SandboxBatch():
	565	+
	566	+ def __init__(self, sandbox, main_group, flags, *, collect=None):
	567	+ self.sandbox = sandbox
	568	+ self.main_group = main_group
	569	+ self.current_group = main_group
	570	+ self.flags = flags
	571	+ self.collect = collect
	572	+
	573	+ def execute(self):
	574	+ self.main_group.execute(self)
	575	+
	576	+ def execute_group(self, group):
	577	+ if group.label:
	578	+ context = self.sandbox._get_context()
	579	+ cm = context.timed_activity(group.label, unique_id=self.sandbox._get_plugin_id())
	580	+ else:
	581	+ cm = contextlib.suppress()
	582	+
	583	+ with cm:
	584	+ group.execute_children(self)
	585	+
	586	+ def execute_command(self, command):
	587	+ if command.label:
	588	+ context = self.sandbox._get_context()
	589	+ message = Message(self.sandbox._get_plugin_id(), MessageType.STATUS,
	590	+ 'Running {}'.format(command.label))
	591	+ context.message(message)
	592	+
	593	+ exitcode = self.sandbox._run(command.command, self.flags, cwd=command.cwd, env=command.env)
	594	+ if exitcode != 0:
	595	+ cmdline = ' '.join(shlex.quote(cmd) for cmd in command.command)
	596	+ label = command.label or cmdline
	597	+ raise SandboxCommandError("Command '{}' failed with exitcode {}".format(label, exitcode),
	598	+ collect=self.collect)
	599	+
	600	+ def execute_call(self, call):
	601	+ call.callback()
	602	+
	603	+
	604	+# _SandboxBatchItem()
	605	+#
	606	+# An item in a command batch.
	607	+#
	608	+class _SandboxBatchItem():
	609	+
	610	+ def __init__(self, *, label=None):
	611	+ self.label = label
	612	+
	613	+
	614	+# _SandboxBatchCommand()
	615	+#
	616	+# A command item in a command batch.
	617	+#
	618	+class _SandboxBatchCommand(_SandboxBatchItem):
	619	+
	620	+ def __init__(self, command, *, cwd, env, label=None):
	621	+ super().__init__(label=label)
	622	+
	623	+ self.command = command
	624	+ self.cwd = cwd
	625	+ self.env = env
	626	+
	627	+ def execute(self, batch):
	628	+ batch.execute_command(self)
	629	+
	630	+
	631	+# _SandboxBatchGroup()
	632	+#
	633	+# A group in a command batch.
	634	+#
	635	+class _SandboxBatchGroup(_SandboxBatchItem):
	636	+
	637	+ def __init__(self, *, label=None):
	638	+ super().__init__(label=label)
	639	+
	640	+ self.children = []
	641	+
	642	+ def append(self, item):
	643	+ self.children.append(item)
	644	+
	645	+ def execute(self, batch):
	646	+ batch.execute_group(self)
	647	+
	648	+ def execute_children(self, batch):
	649	+ for item in self.children:
	650	+ item.execute(batch)
	651	+
	652	+
	653	+# _SandboxBatchCall()
	654	+#
	655	+# A call item in a command batch.
	656	+#
	657	+class _SandboxBatchCall(_SandboxBatchItem):
	658	+
	659	+ def __init__(self, callback):
	660	+ super().__init__()
	661	+
	662	+ self.callback = callback
	663	+
	664	+ def execute(self, batch):
	665	+ batch.execute_call(self)

buildstream/scriptelement.py

---

...	...	@@ -226,10 +226,11 @@ class ScriptElement(Element):
226	226	.format(build_dep.name), silent_nested=True):
227	227	build_dep.stage_dependency_artifacts(sandbox, Scope.RUN, path="/")
228	228
229		- for build_dep in self.dependencies(Scope.BUILD, recurse=False):
230		- with self.timed_activity("Integrating {}".format(build_dep.name), silent_nested=True):
231		- for dep in build_dep.dependencies(Scope.RUN):
232		- dep.integrate(sandbox)
	229	+ with sandbox.batch(0):
	230	+ for build_dep in self.dependencies(Scope.BUILD, recurse=False):
	231	+ with self.timed_activity("Integrating {}".format(build_dep.name), silent_nested=True):
	232	+ for dep in build_dep.dependencies(Scope.RUN):
	233	+ dep.integrate(sandbox)
233	234	else:
234	235	# If layout, follow its rules.
235	236	for item in self.__layout:
...	...	@@ -251,37 +252,38 @@ class ScriptElement(Element):
251	252	virtual_dstdir.descend(item['destination'].lstrip(os.sep).split(os.sep), create=True)
252	253	element.stage_dependency_artifacts(sandbox, Scope.RUN, path=item['destination'])
253	254
254		- for item in self.__layout:
	255	+ with sandbox.batch(0):
	256	+ for item in self.__layout:
255	257
256		- # Skip layout members which dont stage an element
257		- if not item['element']:
258		- continue
	258	+ # Skip layout members which dont stage an element
	259	+ if not item['element']:
	260	+ continue
259	261
260		- element = self.search(Scope.BUILD, item['element'])
	262	+ element = self.search(Scope.BUILD, item['element'])
261	263
262		- # Integration commands can only be run for elements staged to /
263		- if item['destination'] == '/':
264		- with self.timed_activity("Integrating {}".format(element.name),
265		- silent_nested=True):
266		- for dep in element.dependencies(Scope.RUN):
267		- dep.integrate(sandbox)
	264	+ # Integration commands can only be run for elements staged to /
	265	+ if item['destination'] == '/':
	266	+ with self.timed_activity("Integrating {}".format(element.name),
	267	+ silent_nested=True):
	268	+ for dep in element.dependencies(Scope.RUN):
	269	+ dep.integrate(sandbox)
268	270
269	271	install_root_path_components = self.__install_root.lstrip(os.sep).split(os.sep)
270	272	sandbox.get_virtual_directory().descend(install_root_path_components, create=True)
271	273
272	274	def assemble(self, sandbox):
273	275
274		- for groupname, commands in self.__commands.items():
275		- with self.timed_activity("Running '{}'".format(groupname)):
276		- for cmd in commands:
277		- self.status("Running command", detail=cmd)
278		- # Note the -e switch to 'sh' means to exit with an error
279		- # if any untested command fails.
280		- exitcode = sandbox.run(['sh', '-c', '-e', cmd + '\n'],
281		- SandboxFlags.ROOT_READ_ONLY if self.__root_read_only else 0)
282		- if exitcode != 0:
283		- raise ElementError("Command '{}' failed with exitcode {}".format(cmd, exitcode),
284		- collect=self.__install_root)
	276	+ sandbox_flags = SandboxFlags.ROOT_READ_ONLY if self.__root_read_only else 0
	277	+
	278	+ with sandbox.batch(sandbox_flags, collect=self.__install_root):
	279	+ for groupname, commands in self.__commands.items():
	280	+ with sandbox.batch(sandbox_flags, label="Running '{}'".format(groupname)):
	281	+ for cmd in commands:
	282	+ # Note the -e switch to 'sh' means to exit with an error
	283	+ # if any untested command fails.
	284	+ sandbox.run(['sh', '-c', '-e', cmd + '\n'],
	285	+ sandbox_flags,
	286	+ label=cmd)
285	287
286	288	# Return where the result can be collected from
287	289	return self.__install_root

buildstream/utils.py

---

...	...	@@ -505,17 +505,19 @@ def get_bst_version():
505	505	.format(__version__))
506	506
507	507
508		-def move_atomic(source, destination, ensure_parents=True):
	508	+def move_atomic(source, destination, *, ensure_parents=True):
509	509	"""Move the source to the destination using atomic primitives.
510	510
511	511	This uses `os.rename` to move a file or directory to a new destination.
512	512	It wraps some `OSError` thrown errors to ensure their handling is correct.
513	513
514	514	The main reason for this to exist is that rename can throw different errors
515		- for the same symptom (https://www.unix.com/man-page/POSIX/3posix/rename/).
	515	+ for the same symptom (https://www.unix.com/man-page/POSIX/3posix/rename/)
	516	+ when we are moving a directory.
516	517
517	518	We are especially interested here in the case when the destination already
518		- exists. In this case, either EEXIST or ENOTEMPTY are thrown.
	519	+ exists, is a directory and is not empty. In this case, either EEXIST or
	520	+ ENOTEMPTY can be thrown.
519	521
520	522	In order to ensure consistent handling of these exceptions, this function
521	523	should be used instead of `os.rename`
...	...	@@ -525,6 +527,10 @@ def move_atomic(source, destination, ensure_parents=True):
525	527	destination (str or Path): destination to which to move the source
526	528	ensure_parents (bool): Whether or not to create the parent's directories
527	529	of the destination (default: True)
	530	+ Raises:
	531	+ DirectoryExistsError: if the destination directory already exists and is
	532	+ not empty
	533	+ OSError: if another filesystem level error occured
528	534	"""
529	535	if ensure_parents:
530	536	os.makedirs(os.path.dirname(str(destination)), exist_ok=True)

doc/sessions/developing.run

---

...	...	@@ -7,7 +7,7 @@ commands:
7	7	# Capture workspace open output
8	8	- directory: ../examples/developing/
9	9	output: ../source/sessions/developing-workspace-open.html
10		- command: workspace open hello.bst workspace_hello
	10	+ command: workspace open --directory workspace_hello hello.bst
11	11
12	12	# Catpure output from workspace list
13	13	- directory: ../examples/developing/
...	...	@@ -37,7 +37,7 @@ commands:
37	37	# Reopen workspace
38	38	- directory: ../examples/developing/
39	39	output: ../source/sessions/developing-reopen-workspace.html
40		- command: workspace open --no-checkout hello.bst workspace_hello
	40	+ command: workspace open --no-checkout --directory workspace_hello hello.bst
41	41
42	42	# Reset workspace
43	43	- directory: ../examples/developing/

doc/sessions/junctions.run

---

...	...	@@ -13,7 +13,7 @@ commands:
13	13	# Open a crossJunction workspace:
14	14	- directory: ../examples/junctions
15	15	output: ../source/sessions/junctions-workspace-open.html
16		- command: workspace open hello-junction.bst:hello.bst workspace_hello
	16	+ command: workspace open --directory workspace_hello hello-junction.bst:hello.bst
17	17
18	18	# Remove the workspace
19	19	- directory: ../examples/junctions

tests/examples/developing.py

---

...	...	@@ -59,7 +59,7 @@ def test_open_workspace(cli, tmpdir, datafiles):
59	59	project = os.path.join(datafiles.dirname, datafiles.basename)
60	60	workspace_dir = os.path.join(str(tmpdir), "workspace_hello")
61	61
62		- result = cli.run(project=project, args=['workspace', 'open', '-f', 'hello.bst', workspace_dir])
	62	+ result = cli.run(project=project, args=['workspace', 'open', '-f', '--directory', workspace_dir, 'hello.bst', ])
63	63	result.assert_success()
64	64
65	65	result = cli.run(project=project, args=['workspace', 'list'])
...	...	@@ -78,7 +78,7 @@ def test_make_change_in_workspace(cli, tmpdir, datafiles):
78	78	project = os.path.join(datafiles.dirname, datafiles.basename)
79	79	workspace_dir = os.path.join(str(tmpdir), "workspace_hello")
80	80
81		- result = cli.run(project=project, args=['workspace', 'open', '-f', 'hello.bst', workspace_dir])
	81	+ result = cli.run(project=project, args=['workspace', 'open', '-f', '--directory', workspace_dir, 'hello.bst'])
82	82	result.assert_success()
83	83
84	84	result = cli.run(project=project, args=['workspace', 'list'])

tests/examples/junctions.py

---

...	...	@@ -48,7 +48,7 @@ def test_open_cross_junction_workspace(cli, tmpdir, datafiles):
48	48	workspace_dir = os.path.join(str(tmpdir), "workspace_hello_junction")
49	49
50	50	result = cli.run(project=project,
51		- args=['workspace', 'open', 'hello-junction.bst:hello.bst', workspace_dir])
	51	+ args=['workspace', 'open', '--directory', workspace_dir, 'hello-junction.bst:hello.bst'])
52	52	result.assert_success()
53	53
54	54	result = cli.run(project=project,

tests/frontend/buildcheckout.py

---

...	...	@@ -509,7 +509,7 @@ def test_build_checkout_workspaced_junction(cli, tmpdir, datafiles):
509	509
510	510	# Now open a workspace on the junction
511	511	#
512		- result = cli.run(project=project, args=['workspace', 'open', 'junction.bst', workspace])
	512	+ result = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, 'junction.bst'])
513	513	result.assert_success()
514	514	filename = os.path.join(workspace, 'files', 'etc-files', 'etc', 'animal.conf')
515	515

tests/frontend/cross_junction_workspace.py

---

...	...	@@ -47,7 +47,7 @@ def open_cross_junction(cli, tmpdir):
47	47	workspace = tmpdir.join("workspace")
48	48
49	49	element = 'sub.bst:data.bst'
50		- args = ['workspace', 'open', element, str(workspace)]
	50	+ args = ['workspace', 'open', '--directory', str(workspace), element]
51	51	result = cli.run(project=project, args=args)
52	52	result.assert_success()
53	53

tests/frontend/workspace.py

---

...	...	@@ -21,9 +21,11 @@
21	21	# Phillip Smyth <phillip smyth codethink co uk>
22	22	# Jonathan Maw <jonathan maw codethink co uk>
23	23	# Richard Maw <richard maw codethink co uk>
	24	+# William Salmon <will salmon codethink co uk>
24	25	#
25	26
26	27	import os
	28	+import stat
27	29	import pytest
28	30	import shutil
29	31	import subprocess
...	...	@@ -43,65 +45,120 @@ DATA_DIR = os.path.join(
43	45	)
44	46
45	47
46		-def open_workspace(cli, tmpdir, datafiles, kind, track, suffix='', workspace_dir=None,
47		- project_path=None, element_attrs=None):
48		- if not workspace_dir:
49		- workspace_dir = os.path.join(str(tmpdir), 'workspace{}'.format(suffix))
50		- if not project_path:
51		- project_path = os.path.join(datafiles.dirname, datafiles.basename)
52		- else:
53		- shutil.copytree(os.path.join(datafiles.dirname, datafiles.basename), project_path)
54		- bin_files_path = os.path.join(project_path, 'files', 'bin-files')
55		- element_path = os.path.join(project_path, 'elements')
56		- element_name = 'workspace-test-{}{}.bst'.format(kind, suffix)
	48	+class WorkspaceCreater():
	49	+ def __init__(self, cli, tmpdir, datafiles, project_path=None):
	50	+ self.cli = cli
	51	+ self.tmpdir = tmpdir
	52	+ self.datafiles = datafiles
	53	+
	54	+ if not project_path:
	55	+ project_path = os.path.join(datafiles.dirname, datafiles.basename)
	56	+ else:
	57	+ shutil.copytree(os.path.join(datafiles.dirname, datafiles.basename), project_path)
	58	+
	59	+ self.project_path = project_path
	60	+ self.bin_files_path = os.path.join(project_path, 'files', 'bin-files')
	61	+
	62	+ self.workspace_cmd = os.path.join(self.project_path, 'workspace_cmd')
	63	+
	64	+ def create_workspace_element(self, kind, track, suffix='', workspace_dir=None,
	65	+ element_attrs=None):
	66	+ element_name = 'workspace-test-{}{}.bst'.format(kind, suffix)
	67	+ element_path = os.path.join(self.project_path, 'elements')
	68	+ if not workspace_dir:
	69	+ workspace_dir = os.path.join(self.workspace_cmd, element_name)
	70	+ if workspace_dir[-4:] == '.bst':
	71	+ workspace_dir = workspace_dir[:-4]
	72	+
	73	+ # Create our repo object of the given source type with
	74	+ # the bin files, and then collect the initial ref.
	75	+ repo = create_repo(kind, str(self.tmpdir))
	76	+ ref = repo.create(self.bin_files_path)
	77	+ if track:
	78	+ ref = None
	79	+
	80	+ # Write out our test target
	81	+ element = {
	82	+ 'kind': 'import',
	83	+ 'sources': [
	84	+ repo.source_config(ref=ref)
	85	+ ]
	86	+ }
	87	+ if element_attrs:
	88	+ element = {**element, **element_attrs}
	89	+ _yaml.dump(element,
	90	+ os.path.join(element_path,
	91	+ element_name))
	92	+ return element_name, element_path, workspace_dir
57	93
58		- # Create our repo object of the given source type with
59		- # the bin files, and then collect the initial ref.
60		- #
61		- repo = create_repo(kind, str(tmpdir))
62		- ref = repo.create(bin_files_path)
63		- if track:
64		- ref = None
	94	+ def create_workspace_elements(self, kinds, track, suffixs=None, workspace_dir_usr=None,
	95	+ element_attrs=None):
65	96
66		- # Write out our test target
67		- element = {
68		- 'kind': 'import',
69		- 'sources': [
70		- repo.source_config(ref=ref)
71		- ]
72		- }
73		- if element_attrs:
74		- element = {**element, **element_attrs}
75		- _yaml.dump(element,
76		- os.path.join(element_path,
77		- element_name))
	97	+ element_tuples = []
78	98
79		- # Assert that there is no reference, a track & fetch is needed
80		- state = cli.get_element_state(project_path, element_name)
81		- if track:
82		- assert state == 'no reference'
83		- else:
84		- assert state == 'fetch needed'
	99	+ if suffixs is None:
	100	+ suffixs = ['', ] * len(kinds)
	101	+ else:
	102	+ if len(suffixs) != len(kinds):
	103	+ raise "terable error"
85	104
86		- # Now open the workspace, this should have the effect of automatically
87		- # tracking & fetching the source from the repo.
88		- args = ['workspace', 'open']
89		- if track:
90		- args.append('--track')
91		- args.extend([element_name, workspace_dir])
92		- result = cli.run(project=project_path, args=args)
	105	+ for suffix, kind in zip(suffixs, kinds):
	106	+ element_name, element_path, workspace_dir = \
	107	+ self.create_workspace_element(kind, track, suffix, workspace_dir_usr,
	108	+ element_attrs)
93	109
94		- result.assert_success()
	110	+ # Assert that there is no reference, a track & fetch is needed
	111	+ state = self.cli.get_element_state(self.project_path, element_name)
	112	+ if track:
	113	+ assert state == 'no reference'
	114	+ else:
	115	+ assert state == 'fetch needed'
	116	+ element_tuples.append((element_name, workspace_dir))
95	117
96		- # Assert that we are now buildable because the source is
97		- # now cached.
98		- assert cli.get_element_state(project_path, element_name) == 'buildable'
	118	+ return element_tuples
99	119
100		- # Check that the executable hello file is found in the workspace
101		- filename = os.path.join(workspace_dir, 'usr', 'bin', 'hello')
102		- assert os.path.exists(filename)
	120	+ def open_workspaces(self, kinds, track, suffixs=None, workspace_dir=None,
	121	+ element_attrs=None):
	122	+
	123	+ element_tuples = self.create_workspace_elements(kinds, track, suffixs, workspace_dir,
	124	+ element_attrs)
	125	+ os.makedirs(self.workspace_cmd, exist_ok=True)
	126	+
	127	+ # Now open the workspace, this should have the effect of automatically
	128	+ # tracking & fetching the source from the repo.
	129	+ args = ['workspace', 'open']
	130	+ if track:
	131	+ args.append('--track')
	132	+ if workspace_dir is not None:
	133	+ assert len(element_tuples) == 1, "test logic error"
	134	+ _, workspace_dir = element_tuples[0]
	135	+ args.extend(['--directory', workspace_dir])
	136	+
	137	+ args.extend([element_name for element_name, workspace_dir_suffix in element_tuples])
	138	+ result = self.cli.run(cwd=self.workspace_cmd, project=self.project_path, args=args)
	139	+
	140	+ result.assert_success()
	141	+
	142	+ for element_name, workspace_dir in element_tuples:
	143	+ # Assert that we are now buildable because the source is
	144	+ # now cached.
	145	+ assert self.cli.get_element_state(self.project_path, element_name) == 'buildable'
	146	+
	147	+ # Check that the executable hello file is found in the workspace
	148	+ filename = os.path.join(workspace_dir, 'usr', 'bin', 'hello')
	149	+ assert os.path.exists(filename)
	150	+
	151	+ return element_tuples
103	152
104		- return (element_name, project_path, workspace_dir)
	153	+
	154	+def open_workspace(cli, tmpdir, datafiles, kind, track, suffix='', workspace_dir=None,
	155	+ project_path=None, element_attrs=None):
	156	+ workspace_object = WorkspaceCreater(cli, tmpdir, datafiles, project_path)
	157	+ workspaces = workspace_object.open_workspaces((kind, ), track, (suffix, ), workspace_dir,
	158	+ element_attrs)
	159	+ assert len(workspaces) == 1
	160	+ element_name, workspace = workspaces[0]
	161	+ return element_name, workspace_object.project_path, workspace
105	162
106	163
107	164	@pytest.mark.datafiles(DATA_DIR)
...	...	@@ -128,6 +185,128 @@ def test_open_bzr_customize(cli, tmpdir, datafiles):
128	185	assert(expected_output_str in str(output))
129	186
130	187
	188	+@pytest.mark.datafiles(DATA_DIR)
	189	+def test_open_multi(cli, tmpdir, datafiles):
	190	+
	191	+ workspace_object = WorkspaceCreater(cli, tmpdir, datafiles)
	192	+ workspaces = workspace_object.open_workspaces(repo_kinds, False)
	193	+
	194	+ for (elname, workspace), kind in zip(workspaces, repo_kinds):
	195	+ assert kind in elname
	196	+ workspace_lsdir = os.listdir(workspace)
	197	+ if kind == 'git':
	198	+ assert('.git' in workspace_lsdir)
	199	+ elif kind == 'bzr':
	200	+ assert('.bzr' in workspace_lsdir)
	201	+ else:
	202	+ assert not ('.git' in workspace_lsdir)
	203	+ assert not ('.bzr' in workspace_lsdir)
	204	+
	205	+
	206	+@pytest.mark.datafiles(DATA_DIR)
	207	+def test_open_multi_unwritable(cli, tmpdir, datafiles):
	208	+ workspace_object = WorkspaceCreater(cli, tmpdir, datafiles)
	209	+
	210	+ element_tuples = workspace_object.create_workspace_elements(repo_kinds, False, repo_kinds)
	211	+ os.makedirs(workspace_object.workspace_cmd, exist_ok=True)
	212	+
	213	+ # Now open the workspace, this should have the effect of automatically
	214	+ # tracking & fetching the source from the repo.
	215	+ args = ['workspace', 'open']
	216	+ args.extend([element_name for element_name, workspace_dir_suffix in element_tuples])
	217	+ cli.configure({'workspacedir': workspace_object.workspace_cmd})
	218	+
	219	+ cwdstat = os.stat(workspace_object.workspace_cmd)
	220	+ try:
	221	+ os.chmod(workspace_object.workspace_cmd, cwdstat.st_mode - stat.S_IWRITE)
	222	+ result = workspace_object.cli.run(project=workspace_object.project_path, args=args)
	223	+ finally:
	224	+ # Using this finally to make sure we always put thing back how they should be.
	225	+ os.chmod(workspace_object.workspace_cmd, cwdstat.st_mode)
	226	+
	227	+ result.assert_main_error(ErrorDomain.STREAM, None)
	228	+ # Normally we avoid checking stderr in favour of using the mechine readable result.assert_main_error
	229	+ # But Tristan was very keen that the names of the elements left needing workspaces were present in the out put
	230	+ assert (" ".join([element_name for element_name, workspace_dir_suffix in element_tuples[1:]]) in result.stderr)
	231	+
	232	+
	233	+@pytest.mark.datafiles(DATA_DIR)
	234	+def test_open_multi_with_directory(cli, tmpdir, datafiles):
	235	+ workspace_object = WorkspaceCreater(cli, tmpdir, datafiles)
	236	+
	237	+ element_tuples = workspace_object.create_workspace_elements(repo_kinds, False, repo_kinds)
	238	+ os.makedirs(workspace_object.workspace_cmd, exist_ok=True)
	239	+
	240	+ # Now open the workspace, this should have the effect of automatically
	241	+ # tracking & fetching the source from the repo.
	242	+ args = ['workspace', 'open']
	243	+ args.extend(['--directory', 'any/dir/should/fail'])
	244	+
	245	+ args.extend([element_name for element_name, workspace_dir_suffix in element_tuples])
	246	+ result = workspace_object.cli.run(cwd=workspace_object.workspace_cmd, project=workspace_object.project_path,
	247	+ args=args)
	248	+
	249	+ result.assert_main_error(ErrorDomain.STREAM, 'directory-with-multiple-elements')
	250	+
	251	+
	252	+@pytest.mark.datafiles(DATA_DIR)
	253	+def test_open_defaultlocation(cli, tmpdir, datafiles):
	254	+ workspace_object = WorkspaceCreater(cli, tmpdir, datafiles)
	255	+
	256	+ ((element_name, workspace_dir), ) = workspace_object.create_workspace_elements(['git'], False, ['git'])
	257	+ os.makedirs(workspace_object.workspace_cmd, exist_ok=True)
	258	+
	259	+ # Now open the workspace, this should have the effect of automatically
	260	+ # tracking & fetching the source from the repo.
	261	+ args = ['workspace', 'open']
	262	+ args.append(element_name)
	263	+
	264	+ # In the other tests we set the cmd to workspace_object.workspace_cmd with the optional
	265	+ # argument, cwd for the workspace_object.cli.run function. But hear we set the default
	266	+ # workspace location to workspace_object.workspace_cmd and run the cli.run function with
	267	+ # no cwd option so that it runs in the project directory.
	268	+ cli.configure({'workspacedir': workspace_object.workspace_cmd})
	269	+ result = workspace_object.cli.run(project=workspace_object.project_path,
	270	+ args=args)
	271	+
	272	+ result.assert_success()
	273	+
	274	+ assert cli.get_element_state(workspace_object.project_path, element_name) == 'buildable'
	275	+
	276	+ # Check that the executable hello file is found in the workspace
	277	+ # even though the cli.run function was not run with cwd = workspace_object.workspace_cmd
	278	+ # the workspace should be created in there as we used the 'workspacedir' configuration
	279	+ # option.
	280	+ filename = os.path.join(workspace_dir, 'usr', 'bin', 'hello')
	281	+ assert os.path.exists(filename)
	282	+
	283	+
	284	+@pytest.mark.datafiles(DATA_DIR)
	285	+def test_open_defaultlocation_exists(cli, tmpdir, datafiles):
	286	+ workspace_object = WorkspaceCreater(cli, tmpdir, datafiles)
	287	+
	288	+ ((element_name, workspace_dir), ) = workspace_object.create_workspace_elements(['git'], False, ['git'])
	289	+ os.makedirs(workspace_object.workspace_cmd, exist_ok=True)
	290	+
	291	+ with open(workspace_dir, 'w') as fl:
	292	+ fl.write('foo')
	293	+
	294	+ # Now open the workspace, this should have the effect of automatically
	295	+ # tracking & fetching the source from the repo.
	296	+ args = ['workspace', 'open']
	297	+ args.append(element_name)
	298	+
	299	+ # In the other tests we set the cmd to workspace_object.workspace_cmd with the optional
	300	+ # argument, cwd for the workspace_object.cli.run function. But hear we set the default
	301	+ # workspace location to workspace_object.workspace_cmd and run the cli.run function with
	302	+ # no cwd option so that it runs in the project directory.
	303	+ cli.configure({'workspacedir': workspace_object.workspace_cmd})
	304	+ result = workspace_object.cli.run(project=workspace_object.project_path,
	305	+ args=args)
	306	+
	307	+ result.assert_main_error(ErrorDomain.STREAM, 'bad-directory')
	308	+
	309	+
131	310	@pytest.mark.datafiles(DATA_DIR)
132	311	@pytest.mark.parametrize("kind", repo_kinds)
133	312	def test_open_track(cli, tmpdir, datafiles, kind):
...	...	@@ -150,7 +329,7 @@ def test_open_force(cli, tmpdir, datafiles, kind):
150	329
151	330	# Now open the workspace again with --force, this should happily succeed
152	331	result = cli.run(project=project, args=[
153		- 'workspace', 'open', '--force', element_name, workspace
	332	+ 'workspace', 'open', '--force', '--directory', workspace, element_name
154	333	])
155	334	result.assert_success()
156	335
...	...	@@ -165,7 +344,7 @@ def test_open_force_open(cli, tmpdir, datafiles, kind):
165	344
166	345	# Now open the workspace again with --force, this should happily succeed
167	346	result = cli.run(project=project, args=[
168		- 'workspace', 'open', '--force', element_name, workspace
	347	+ 'workspace', 'open', '--force', '--directory', workspace, element_name
169	348	])
170	349	result.assert_success()
171	350
...	...	@@ -196,7 +375,7 @@ def test_open_force_different_workspace(cli, tmpdir, datafiles, kind):
196	375
197	376	# Now open the workspace again with --force, this should happily succeed
198	377	result = cli.run(project=project, args=[
199		- 'workspace', 'open', '--force', element_name2, workspace
	378	+ 'workspace', 'open', '--force', '--directory', workspace, element_name2
200	379	])
201	380
202	381	# Assert that the file in workspace 1 has been replaced
...	...	@@ -504,7 +683,7 @@ def test_buildable_no_ref(cli, tmpdir, datafiles):
504	683	# Now open the workspace. We don't need to checkout the source though.
505	684	workspace = os.path.join(str(tmpdir), 'workspace-no-ref')
506	685	os.makedirs(workspace)
507		- args = ['workspace', 'open', '--no-checkout', element_name, workspace]
	686	+ args = ['workspace', 'open', '--no-checkout', '--directory', workspace, element_name]
508	687	result = cli.run(project=project, args=args)
509	688	result.assert_success()
510	689
...	...	@@ -766,7 +945,7 @@ def test_list_supported_workspace(cli, tmpdir, datafiles, workspace_cfg, expecte
766	945	element_name))
767	946
768	947	# Make a change to the workspaces file
769		- result = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
	948	+ result = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
770	949	result.assert_success()
771	950	result = cli.run(project=project, args=['workspace', 'close', '--remove-dir', element_name])
772	951	result.assert_success()

tests/integration/manual.py

---

...	...	@@ -128,3 +128,28 @@ def test_manual_element_noparallel(cli, tmpdir, datafiles):
128	128	assert text == """-j1 -Wall
129	129	2
130	130	"""
	131	+
	132	+
	133	+@pytest.mark.datafiles(DATA_DIR)
	134	+@pytest.mark.skipif(IS_LINUX and not HAVE_BWRAP, reason='Only available with bubblewrap on Linux')
	135	+def test_manual_element_logging(cli, tmpdir, datafiles):
	136	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	137	+ checkout = os.path.join(cli.directory, 'checkout')
	138	+ element_path = os.path.join(project, 'elements')
	139	+ element_name = 'import/import.bst'
	140	+
	141	+ create_manual_element(element_name, element_path, {
	142	+ 'configure-commands': ["echo configure"],
	143	+ 'build-commands': ["echo build"],
	144	+ 'install-commands': ["echo install"],
	145	+ 'strip-commands': ["echo strip"]
	146	+ }, {}, {})
	147	+
	148	+ res = cli.run(project=project, args=['build', element_name])
	149	+ assert res.exit_code == 0
	150	+
	151	+ # Verify that individual commands are logged
	152	+ assert "echo configure" in res.stderr
	153	+ assert "echo build" in res.stderr
	154	+ assert "echo install" in res.stderr
	155	+ assert "echo strip" in res.stderr

tests/integration/sandbox-bwrap.py

---

...	...	@@ -58,5 +58,5 @@ def test_sandbox_bwrap_return_subprocess(cli, tmpdir, datafiles):
58	58	})
59	59
60	60	result = cli.run(project=project, args=['build', element_name])
61		- result.assert_task_error(error_domain=ErrorDomain.ELEMENT, error_reason=None)
	61	+ result.assert_task_error(error_domain=ErrorDomain.SANDBOX, error_reason="command-failed")
62	62	assert "sandbox-bwrap/command-exit-42.bst|Command 'exit 42' failed with exitcode 42" in result.stderr

tests/integration/shell.py

---

...	...	@@ -290,7 +290,7 @@ def test_workspace_visible(cli, tmpdir, datafiles):
290	290
291	291	# Open a workspace on our build failing element
292	292	#
293		- res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
	293	+ res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
294	294	assert res.exit_code == 0
295	295
296	296	# Ensure the dependencies of our build failing element are built

tests/integration/workspace.py

---

...	...	@@ -24,7 +24,7 @@ def test_workspace_mount(cli, tmpdir, datafiles):
24	24	workspace = os.path.join(cli.directory, 'workspace')
25	25	element_name = 'workspace/workspace-mount.bst'
26	26
27		- res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
	27	+ res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
28	28	assert res.exit_code == 0
29	29
30	30	res = cli.run(project=project, args=['build', element_name])
...	...	@@ -41,7 +41,7 @@ def test_workspace_commanddir(cli, tmpdir, datafiles):
41	41	workspace = os.path.join(cli.directory, 'workspace')
42	42	element_name = 'workspace/workspace-commanddir.bst'
43	43
44		- res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
	44	+ res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
45	45	assert res.exit_code == 0
46	46
47	47	res = cli.run(project=project, args=['build', element_name])
...	...	@@ -78,7 +78,7 @@ def test_workspace_updated_dependency(cli, tmpdir, datafiles):
78	78	_yaml.dump(dependency, os.path.join(element_path, dep_name))
79	79
80	80	# First open the workspace
81		- res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
	81	+ res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
82	82	assert res.exit_code == 0
83	83
84	84	# We build the workspaced element, so that we have an artifact
...	...	@@ -134,7 +134,7 @@ def test_workspace_update_dependency_failed(cli, tmpdir, datafiles):
134	134	_yaml.dump(dependency, os.path.join(element_path, dep_name))
135	135
136	136	# First open the workspace
137		- res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
	137	+ res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
138	138	assert res.exit_code == 0
139	139
140	140	# We build the workspaced element, so that we have an artifact
...	...	@@ -210,7 +210,7 @@ def test_updated_dependency_nested(cli, tmpdir, datafiles):
210	210	_yaml.dump(dependency, os.path.join(element_path, dep_name))
211	211
212	212	# First open the workspace
213		- res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
	213	+ res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
214	214	assert res.exit_code == 0
215	215
216	216	# We build the workspaced element, so that we have an artifact
...	...	@@ -264,7 +264,7 @@ def test_incremental_configure_commands_run_only_once(cli, tmpdir, datafiles):
264	264	_yaml.dump(element, os.path.join(element_path, element_name))
265	265
266	266	# We open a workspace on the above element
267		- res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
	267	+ res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
268	268	res.assert_success()
269	269
270	270	# Then we build, and check whether the configure step succeeded

tests/plugins/filter.py

---

...	...	@@ -108,19 +108,28 @@ def test_filter_forbid_also_rdep(datafiles, cli):
108	108	def test_filter_workspace_open(datafiles, cli, tmpdir):
109	109	project = os.path.join(datafiles.dirname, datafiles.basename)
110	110	workspace_dir = os.path.join(tmpdir.dirname, tmpdir.basename, "workspace")
111		- result = cli.run(project=project, args=['workspace', 'open', 'deps-permitted.bst', workspace_dir])
	111	+ result = cli.run(project=project, args=['workspace', 'open', '--directory', workspace_dir, 'deps-permitted.bst'])
112	112	result.assert_success()
113	113	assert os.path.exists(os.path.join(workspace_dir, "foo"))
114	114	assert os.path.exists(os.path.join(workspace_dir, "bar"))
115	115	assert os.path.exists(os.path.join(workspace_dir, "baz"))
116	116
117	117
	118	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'basic'))
	119	+def test_filter_workspace_open_multi(datafiles, cli, tmpdir):
	120	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	121	+ result = cli.run(cwd=project, project=project, args=['workspace', 'open', 'deps-permitted.bst',
	122	+ 'output-orphans.bst'])
	123	+ result.assert_success()
	124	+ assert os.path.exists(os.path.join(project, "input"))
	125	+
	126	+
118	127	@pytest.mark.datafiles(os.path.join(DATA_DIR, 'basic'))
119	128	def test_filter_workspace_build(datafiles, cli, tmpdir):
120	129	project = os.path.join(datafiles.dirname, datafiles.basename)
121	130	tempdir = os.path.join(tmpdir.dirname, tmpdir.basename)
122	131	workspace_dir = os.path.join(tempdir, "workspace")
123		- result = cli.run(project=project, args=['workspace', 'open', 'output-orphans.bst', workspace_dir])
	132	+ result = cli.run(project=project, args=['workspace', 'open', '--directory', workspace_dir, 'output-orphans.bst'])
124	133	result.assert_success()
125	134	src = os.path.join(workspace_dir, "foo")
126	135	dst = os.path.join(workspace_dir, "quux")
...	...	@@ -138,7 +147,7 @@ def test_filter_workspace_close(datafiles, cli, tmpdir):
138	147	project = os.path.join(datafiles.dirname, datafiles.basename)
139	148	tempdir = os.path.join(tmpdir.dirname, tmpdir.basename)
140	149	workspace_dir = os.path.join(tempdir, "workspace")
141		- result = cli.run(project=project, args=['workspace', 'open', 'output-orphans.bst', workspace_dir])
	150	+ result = cli.run(project=project, args=['workspace', 'open', '--directory', workspace_dir, 'output-orphans.bst'])
142	151	result.assert_success()
143	152	src = os.path.join(workspace_dir, "foo")
144	153	dst = os.path.join(workspace_dir, "quux")
...	...	@@ -158,7 +167,7 @@ def test_filter_workspace_reset(datafiles, cli, tmpdir):
158	167	project = os.path.join(datafiles.dirname, datafiles.basename)
159	168	tempdir = os.path.join(tmpdir.dirname, tmpdir.basename)
160	169	workspace_dir = os.path.join(tempdir, "workspace")
161		- result = cli.run(project=project, args=['workspace', 'open', 'output-orphans.bst', workspace_dir])
	170	+ result = cli.run(project=project, args=['workspace', 'open', '--directory', workspace_dir, 'output-orphans.bst'])
162	171	result.assert_success()
163	172	src = os.path.join(workspace_dir, "foo")
164	173	dst = os.path.join(workspace_dir, "quux")

tests/plugins/pipeline.py

---

...	...	@@ -14,7 +14,7 @@ DATA_DIR = os.path.join(
14	14
15	15	def create_pipeline(tmpdir, basedir, target):
16	16	context = Context()
17		- context.load()
	17	+ context.load(config=os.devnull)
18	18	context.deploydir = os.path.join(str(tmpdir), 'deploy')
19	19	context.artifactdir = os.path.join(str(tmpdir), 'artifact')
20	20	project = Project(basedir, context)

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.