[Notes] [Git][BuildStream/buildstream][tpollard/workspacebuildtree] 6 commits: _scheduler/queues: Don't call update state outside of error handling harness

[Tom Pollard <gitlab mg gitlab com>]

Title: GitLab

Tom Pollard pushed to branch tpollard/workspacebuildtree at BuildStream / buildstream

Commits:

• 98c15463
  by Tristan Van Berkom at 2018-12-05T07:04:56Z

  _scheduler/queues: Don't call update state outside of error handling harness
  
  Commit 3fa79d8da, part of an initiative for caching of the failed builds,
  introduced a call to Element._update_state() after a job completes and before
  entering the error handling harness intended for handling plugin raised errors.
  
  Element._update_state() can result in triggering plugin code to
  run, so this is incorrect, and causes raised errors to crash BuildStream
  if they happen here.
  
  After analyzing the code, it appears that this additional call to
  Element._update_state() is unneeded, and was only added because the
  state needs to be updated for a failure as well as a success.
  
  Instead, we now have the BuildQueue call Element._assemble_done()
  unconditionally, regardless of whether the build was successful
  or not, which has the same effect and also reads better.
  
  In addition, added a FIXME comment that we are still conditionally
  updating the artifact cache size from BuildQueue.done() only
  if the build is successful, which is incorrect because failed builds
  also increase the local artifact cache size - to fix this we need
  to communicate the added artifact size through Element._assemble()
  regardless of whether the build succeeded or failed.
  

• 5d77b871
  by Tristan Van Berkom at 2018-12-05T07:37:54Z

  Merge branch 'tristan/refactor-queues-update-state' into 'master'
  
  _scheduler/queues/queue.py: Don't call update state outside of error handling harness
  
  See merge request BuildStream/buildstream!988

• 82513109
  by Valentin David at 2018-12-05T09:47:52Z

  git source plugin: Track git tags and save them to reproduce a minimum shallow repository
  
  Instead of tag information being fetched which can change with time,
  they are tracked and saved in the projects.refs/.bst. Then we re-tag
  automatically the closest tag so that `git describe` works and is
  reproducible.
  
  This new feature is opt-in with the new `track-tags` configuration,
  and must be used to fix modules which are broken by our new policy
  of omitting the `.git/` repository when staging git sources.
  
  This fixes issue #487
  

• ec7b62c3
  by Valentin David at 2018-12-05T09:50:15Z

  NEWS: Add news about broken 'git describe' by default.
  

• 761e7570
  by Tristan Van Berkom at 2018-12-05T10:23:49Z

  Merge branch 'valentindavid/git_describe_tracking' into 'master'
  
  Track of git tags and save them to reproduce minimum shallow repository
  
  Closes #487
  
  See merge request BuildStream/buildstream!906

• b1ad9d28
  by Tom Pollard at 2018-12-05T10:41:31Z

  WIP: Opening a workspace with a cached build
  

22 changed files:

• NEWS
• buildstream/_artifactcache/artifactcache.py
• buildstream/_artifactcache/cascache.py
• buildstream/_context.py
• buildstream/_frontend/cli.py
• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/queues/queue.py
• buildstream/_stream.py
• buildstream/_versions.py
• buildstream/_workspaces.py
• buildstream/element.py
• buildstream/plugins/sources/git.py
• buildstream/source.py
• doc/source/developing/workspaces.rst
• + tests/cachekey/project/sources/git3.bst
• + tests/cachekey/project/sources/git3.expected
• tests/cachekey/project/target.bst
• tests/cachekey/project/target.expected
• tests/frontend/workspace.py
• tests/integration/workspace.py
• tests/sources/git.py
• tests/testutils/repo/git.py

Changes:

NEWS

---

...	...	@@ -74,6 +74,10 @@ buildstream 1.3.1
74	74	o Add sandbox API for command batching and use it for build, script, and
75	75	compose elements.
76	76
	77	+ o BREAKING CHANGE: The `git` plugin does not create a local `.git`
	78	+ repository by default. If `git describe` is required to work, the
	79	+ plugin has now a tag tracking feature instead. This can be enabled
	80	+ by setting 'track-tags'.
77	81
78	82	=================
79	83	buildstream 1.1.5

buildstream/_artifactcache/artifactcache.py

---

...	...	@@ -766,6 +766,20 @@ class ArtifactCache():
766	766
767	767	self.cas.link_ref(oldref, newref)
768	768
	769	+ # checkout_artifact_subdir()
	770	+ #
	771	+ # Checkout given artifact subdir into provided directory
	772	+ #
	773	+ # Args:
	774	+ # element (Element): The Element
	775	+ # key (str): The cache key to use
	776	+ # subdir (str): The subdir to checkout
	777	+ # tmpdir (str): The dir to place the subdir content
	778	+ #
	779	+ def checkout_artifact_subdir(self, element, key, subdir, tmpdir):
	780	+ ref = self.get_artifact_fullname(element, key)
	781	+ return self.cas.checkout_artifact_subdir(ref, subdir, tmpdir)
	782	+
769	783	################################################
770	784	# Local Private Methods #
771	785	################################################

buildstream/_artifactcache/cascache.py

---

...	...	@@ -460,6 +460,21 @@ class CASCache():
460	460
461	461	return True
462	462
	463	+ # checkout_artifact_subdir():
	464	+ #
	465	+ # Checkout given artifact subdir into provided directory
	466	+ #
	467	+ # Args:
	468	+ # ref (str): The ref to check
	469	+ # subdir (str): The subdir to checkout
	470	+ # tmpdir (str): The dir to place the subdir content
	471	+ #
	472	+ def checkout_artifact_subdir(self, ref, subdir, tmpdir):
	473	+ tree = self.resolve_ref(ref)
	474	+ # This assumes that the subdir digest is present in the element tree
	475	+ subdirdigest = self._get_subdir(tree, subdir)
	476	+ self._checkout(tmpdir, subdirdigest)
	477	+
463	478	# objpath():
464	479	#
465	480	# Return the path of an object based on its digest.

buildstream/_context.py

---

...	...	@@ -126,6 +126,9 @@ class Context():
126	126	# a hard reset of a workspace, potentially losing changes.
127	127	self.prompt_workspace_reset_hard = None
128	128
	129	+ # Whether to not include artifact buildtrees in workspaces if available
	130	+ self.workspace_buildtrees = True
	131	+
129	132	# Whether elements must be rebuilt when their dependencies have changed
130	133	self._strict_build_plan = None
131	134
...	...	@@ -182,7 +185,7 @@ class Context():
182	185	_yaml.node_validate(defaults, [
183	186	'sourcedir', 'builddir', 'artifactdir', 'logdir',
184	187	'scheduler', 'artifacts', 'logging', 'projects',
185		- 'cache', 'prompt', 'workspacedir',
	188	+ 'cache', 'prompt', 'workspacedir', 'workspace-buildtrees'
186	189	])
187	190
188	191	for directory in ['sourcedir', 'builddir', 'artifactdir', 'logdir', 'workspacedir']:
...	...	@@ -210,6 +213,9 @@ class Context():
210	213	# Load pull build trees configuration
211	214	self.pull_buildtrees = _yaml.node_get(cache, bool, 'pull-buildtrees')
212	215
	216	+ # Load workspace buildtrees configuration
	217	+ self.workspace_buildtrees = _yaml.node_get(defaults, bool, 'workspace-buildtrees', default_value='True')
	218	+
213	219	# Load logging config
214	220	logging = _yaml.node_get(defaults, Mapping, 'logging')
215	221	_yaml.node_validate(logging, [

buildstream/_frontend/cli.py

---

...	...	@@ -721,7 +721,7 @@ def workspace():
721	721	##################################################################
722	722	@workspace.command(name='open', short_help="Open a new workspace")
723	723	@click.option('--no-checkout', default=False, is_flag=True,
724		- help="Do not checkout the source, only link to the given directory")
	724	+ help="Do not checkout the source or cached buildtree, only link to the given directory")
725	725	@click.option('--force', '-f', default=False, is_flag=True,
726	726	help="The workspace will be created even if the directory in which it will be created is not empty " +
727	727	"or if a workspace for that element already exists")
...	...	@@ -730,16 +730,25 @@ def workspace():
730	730	@click.option('--directory', type=click.Path(file_okay=False), default=None,
731	731	help="Only for use when a single Element is given: Set the directory to use to create the workspace")
732	732	@click.argument('elements', nargs=-1, type=click.Path(readable=False), required=True)
	733	+@click.option('--no-cache', default=False, is_flag=True,
	734	+ help="Do not checkout the cached buildtree")
733	735	@click.pass_obj
734		-def workspace_open(app, no_checkout, force, track_, directory, elements):
735		- """Open a workspace for manual source modification"""
	736	+def workspace_open(app, no_checkout, force, track_, directory, elements, no_cache):
	737	+
	738	+ """Open a workspace for manual source modification, the elements buildtree
	739	+ will be provided if available in the local artifact cache.
	740	+ """
	741	+
	742	+ if not no_cache and not no_checkout:
	743	+ click.echo("WARNING: Workspace will be opened without the cached buildtree if not cached locally")
736	744
737	745	with app.initialized():
738	746	app.stream.workspace_open(elements,
739	747	no_checkout=no_checkout,
740	748	track_first=track_,
741	749	force=force,
742		- custom_dir=directory)
	750	+ custom_dir=directory,
	751	+ no_cache=no_cache)
743	752
744	753
745	754	##################################################################

buildstream/_scheduler/queues/buildqueue.py

---

...	...	@@ -106,10 +106,16 @@ class BuildQueue(Queue):
106	106
107	107	def done(self, job, element, result, success):
108	108
109		- if success:
110		- # Inform element in main process that assembly is done
111		- element._assemble_done()
	109	+ # Inform element in main process that assembly is done
	110	+ element._assemble_done()
112	111
113		- # This has to be done after _assemble_done, such that the
114		- # element may register its cache key as required
	112	+ # This has to be done after _assemble_done, such that the
	113	+ # element may register its cache key as required
	114	+ #
	115	+ # FIXME: Element._assemble() does not report both the failure state and the
	116	+ # size of the newly cached failed artifact, so we can only adjust the
	117	+ # artifact cache size for a successful build even though we know a
	118	+ # failed build also grows the artifact cache size.
	119	+ #
	120	+ if success:
115	121	self._check_cache_size(job, element, result)

buildstream/_scheduler/queues/queue.py

---

...	...	@@ -292,7 +292,6 @@ class Queue():
292	292	# See the Job object for an explanation of the call signature
293	293	#
294	294	def _job_done(self, job, element, success, result):
295		- element._update_state()
296	295
297	296	# Update values that need to be synchronized in the main task
298	297	# before calling any queue implementation

buildstream/_stream.py

---

...	...	@@ -482,14 +482,21 @@ class Stream():
482	482	# track_first (bool): Whether to track and fetch first
483	483	# force (bool): Whether to ignore contents in an existing directory
484	484	# custom_dir (str): Custom location to create a workspace or false to use default location.
	485	+ # no_cache (bool): Whether to not include the cached buildtree
485	486	#
486	487	def workspace_open(self, targets, *,
487	488	no_checkout,
488	489	track_first,
489	490	force,
490		- custom_dir):
	491	+ custom_dir,
	492	+ no_cache):
	493	+
491	494	# This function is a little funny but it is trying to be as atomic as possible.
492	495
	496	+ # Set no_cache if the global user conf workspacebuildtrees is false
	497	+ if not self._context.workspace_buildtrees:
	498	+ no_cache = True
	499	+
493	500	if track_first:
494	501	track_targets = targets
495	502	else:
...	...	@@ -546,7 +553,7 @@ class Stream():
546	553	reason='directory-with-multiple-elements')
547	554	expanded_directories = [custom_dir, ]
548	555	else:
549		- # If this fails it is a bug in what ever calls this, usually cli.py and so can not be tested for via the
	556	+ # If this fails it is a bug in whatever calls this, usually cli.py and so can not be tested for via the
550	557	# run bst test mechanism.
551	558	assert len(elements) == len(expanded_directories)
552	559
...	...	@@ -561,12 +568,26 @@ class Stream():
561	568	.format(target.name, directory), reason='bad-directory')
562	569
563	570	# So far this function has tried to catch as many issues as possible with out making any changes
564		- # Now it dose the bits that can not be made atomic.
	571	+ # Now it does the bits that can not be made atomic.
565	572	targetGenerator = zip(elements, expanded_directories)
566	573	for target, directory in targetGenerator:
567	574	self._message(MessageType.INFO, "Creating workspace for element {}"
568	575	.format(target.name))
569	576
	577	+ # Check if given target has a buildtree artifact cached locally
	578	+ buildtree = None
	579	+ if target._cached():
	580	+ buildtree = target._cached_buildtree()
	581	+
	582	+ # If we're running in the default state, make the user aware of buildtree usage
	583	+ if not no_cache and not no_checkout:
	584	+ if buildtree:
	585	+ self._message(MessageType.INFO, "{} buildtree artifact is available,"
	586	+ " workspace will be opened with it".format(target.name))
	587	+ else:
	588	+ self._message(MessageType.WARN, "{} buildtree artifact not available,"
	589	+ " workspace will be opened with source checkout".format(target.name))
	590	+
570	591	workspace = workspaces.get_workspace(target._get_full_name())
571	592	if workspace:
572	593	workspaces.delete_workspace(target._get_full_name())
...	...	@@ -581,11 +602,16 @@ class Stream():
581	602	todo_elements = "\nDid not try to create workspaces for " + todo_elements
582	603	raise StreamError("Failed to create workspace directory: {}".format(e) + todo_elements) from e
583	604
584		- workspaces.create_workspace(target._get_full_name(), directory)
585		-
586		- if not no_checkout:
587		- with target.timed_activity("Staging sources to {}".format(directory)):
588		- target._open_workspace()
	605	+ # Handle opening workspace with buildtree included
	606	+ if (buildtree and not no_cache) and not no_checkout:
	607	+ workspaces.create_workspace(target._get_full_name(), directory, cached_build=buildtree)
	608	+ with target.timed_activity("Staging buildtree to {}".format(directory)):
	609	+ target._open_workspace(buildtree=buildtree)
	610	+ else:
	611	+ workspaces.create_workspace(target._get_full_name(), directory)
	612	+ if (not buildtree or no_cache) and not no_checkout:
	613	+ with target.timed_activity("Staging sources to {}".format(directory)):
	614	+ target._open_workspace()
589	615
590	616	# Saving the workspace once it is set up means that if the next workspace fails to be created before
591	617	# the configuration gets saved. The successfully created workspace still gets saved.
...	...	@@ -672,10 +698,24 @@ class Stream():
672	698	.format(workspace_path, e)) from e
673	699
674	700	workspaces.delete_workspace(element._get_full_name())
675		- workspaces.create_workspace(element._get_full_name(), workspace_path)
676	701
677		- with element.timed_activity("Staging sources to {}".format(workspace_path)):
678		- element._open_workspace()
	702	+ # Create the workspace, ensuring the original optional cached build state is preserved if
	703	+ # possible.
	704	+ buildtree = False
	705	+ if workspace.cached_build and element._cached():
	706	+ if self._artifacts.contains_subdir_artifact(element, element._get_cache_key(), 'buildtree'):
	707	+ buildtree = True
	708	+
	709	+ # Warn the user if the workspace cannot be opened with the original cached build state
	710	+ if workspace.cached_build and not buildtree:
	711	+ self._message(MessageType.WARN, "{} original buildtree artifact not available,"
	712	+ " workspace will be opened with source checkout".format(element.name))
	713	+
	714	+ workspaces.create_workspace(element._get_full_name(), workspace_path,
	715	+ cached_build=buildtree)
	716	+
	717	+ with element.timed_activity("Staging to {}".format(workspace_path)):
	718	+ element._open_workspace(buildtree=buildtree)
679	719
680	720	self._message(MessageType.INFO,
681	721	"Reset workspace for {} at: {}".format(element.name,

buildstream/_versions.py

---

...	...	@@ -23,7 +23,7 @@
23	23	# This version is bumped whenever enhancements are made
24	24	# to the `project.conf` format or the core element format.
25	25	#
26		-BST_FORMAT_VERSION = 18
	26	+BST_FORMAT_VERSION = 19
27	27
28	28
29	29	# The base BuildStream artifact version

buildstream/_workspaces.py

---

...	...	@@ -24,7 +24,7 @@ from . import _yaml
24	24	from ._exceptions import LoadError, LoadErrorReason
25	25
26	26
27		-BST_WORKSPACE_FORMAT_VERSION = 3
	27	+BST_WORKSPACE_FORMAT_VERSION = 4
28	28
29	29
30	30	# Workspace()
...	...	@@ -43,9 +43,11 @@ BST_WORKSPACE_FORMAT_VERSION = 3
43	43	# running_files (dict): A dict mapping dependency elements to files
44	44	# changed between failed builds. Should be
45	45	# made obsolete with failed build artifacts.
	46	+# cached_build (bool): If the workspace is staging the cached build artifact
46	47	#
47	48	class Workspace():
48		- def __init__(self, toplevel_project, *, last_successful=None, path=None, prepared=False, running_files=None):
	49	+ def __init__(self, toplevel_project, *, last_successful=None, path=None, prepared=False,
	50	+ running_files=None, cached_build=False):
49	51	self.prepared = prepared
50	52	self.last_successful = last_successful
51	53	self._path = path
...	...	@@ -53,6 +55,7 @@ class Workspace():
53	55
54	56	self._toplevel_project = toplevel_project
55	57	self._key = None
	58	+ self.cached_build = cached_build
56	59
57	60	# to_dict()
58	61	#
...	...	@@ -65,7 +68,8 @@ class Workspace():
65	68	ret = {
66	69	'prepared': self.prepared,
67	70	'path': self._path,
68		- 'running_files': self.running_files
	71	+ 'running_files': self.running_files,
	72	+ 'cached_build': self.cached_build
69	73	}
70	74	if self.last_successful is not None:
71	75	ret["last_successful"] = self.last_successful
...	...	@@ -224,12 +228,13 @@ class Workspaces():
224	228	# Args:
225	229	# element_name (str) - The element name to create a workspace for
226	230	# path (str) - The path in which the workspace should be kept
	231	+ # cached_build (bool) - If the workspace is staging the cached build artifact
227	232	#
228		- def create_workspace(self, element_name, path):
	233	+ def create_workspace(self, element_name, path, cached_build=False):
229	234	if path.startswith(self._toplevel_project.directory):
230	235	path = os.path.relpath(path, self._toplevel_project.directory)
231	236
232		- self._workspaces[element_name] = Workspace(self._toplevel_project, path=path)
	237	+ self._workspaces[element_name] = Workspace(self._toplevel_project, path=path, cached_build=cached_build)
233	238
234	239	return self._workspaces[element_name]
235	240
...	...	@@ -396,6 +401,7 @@ class Workspaces():
396	401	'path': _yaml.node_get(node, str, 'path'),
397	402	'last_successful': _yaml.node_get(node, str, 'last_successful', default_value=None),
398	403	'running_files': _yaml.node_get(node, dict, 'running_files', default_value=None),
	404	+ 'cached_build': _yaml.node_get(node, bool, 'cached_build', default_value=False)
399	405	}
400	406	return Workspace.from_dict(self._toplevel_project, dictionary)
401	407

buildstream/element.py

---

...	...	@@ -1911,7 +1911,10 @@ class Element(Plugin):
1911	1911	# This requires that a workspace already be created in
1912	1912	# the workspaces metadata first.
1913	1913	#
1914		- def _open_workspace(self):
	1914	+ # Args:
	1915	+ # buildtree (bool): Whether to open workspace with artifact buildtree
	1916	+ #
	1917	+ def _open_workspace(self, buildtree=False):
1915	1918	context = self._get_context()
1916	1919	workspace = self._get_workspace()
1917	1920	assert workspace is not None
...	...	@@ -1924,11 +1927,19 @@ class Element(Plugin):
1924	1927	# files in the target directory actually works without any
1925	1928	# additional support from Source implementations.
1926	1929	#
	1930	+
1927	1931	os.makedirs(context.builddir, exist_ok=True)
1928	1932	with utils._tempdir(dir=context.builddir, prefix='workspace-{}'
1929	1933	.format(self.normal_name)) as temp:
1930		- for source in self.sources():
1931		- source._init_workspace(temp)
	1934	+
	1935	+ # Checkout cached buildtree, augment with source plugin if applicable
	1936	+ if buildtree:
	1937	+ self.__artifacts.checkout_artifact_subdir(self, self._get_cache_key(), 'buildtree', temp)
	1938	+ for source in self.sources():
	1939	+ source._init_cached_build_workspace(temp)
	1940	+ else:
	1941	+ for source in self.sources():
	1942	+ source._init_workspace(temp)
1932	1943
1933	1944	# Now hardlink the files into the workspace target.
1934	1945	utils.link_files(temp, workspace.get_absolute_path())

buildstream/plugins/sources/git.py

---

...	...	@@ -76,6 +76,56 @@ git - stage files from a git repository
76	76	url: upstream:baz.git
77	77	checkout: False
78	78
	79	+ # Enable tag tracking.
	80	+ #
	81	+ # This causes the `tags` metadata to be populated automatically
	82	+ # as a result of tracking the git source.
	83	+ #
	84	+ # By default this is 'False'.
	85	+ #
	86	+ track-tags: True
	87	+
	88	+ # If the list of tags below is set, then a lightweight dummy
	89	+ # git repository will be staged along with the content at
	90	+ # build time.
	91	+ #
	92	+ # This is useful for a growing number of modules which use
	93	+ # `git describe` at build time in order to determine the version
	94	+ # which will be encoded into the built software.
	95	+ #
	96	+ # The 'tags' below is considered as a part of the git source
	97	+ # reference and will be stored in the 'project.refs' file if
	98	+ # that has been selected as your project's ref-storage.
	99	+ #
	100	+ # Migration notes:
	101	+ #
	102	+ # If you are upgrading from BuildStream 1.2, which used to
	103	+ # stage the entire repository by default, you will notice that
	104	+ # some modules which use `git describe` are broken, and will
	105	+ # need to enable this feature in order to fix them.
	106	+ #
	107	+ # If you need to enable this feature without changing the
	108	+ # the specific commit that you are building, then we recommend
	109	+ # the following migration steps for any git sources where
	110	+ # `git describe` is required:
	111	+ #
	112	+ # o Enable `track-tags` feature
	113	+ # o Set the `track` parameter to the desired commit sha which
	114	+ # the current `ref` points to
	115	+ # o Run `bst track` for these elements, this will result in
	116	+ # populating the `tags` portion of the refs without changing
	117	+ # the refs
	118	+ # o Restore the `track` parameter to the branches which you have
	119	+ # previously been tracking afterwards.
	120	+ #
	121	+ tags:
	122	+ - tag: lightweight-example
	123	+ commit: 04ad0dc656cb7cc6feb781aa13bdbf1d67d0af78
	124	+ annotated: false
	125	+ - tag: annotated-example
	126	+ commit: 10abe77fe8d77385d86f225b503d9185f4ef7f3a
	127	+ annotated: true
	128	+
79	129	See :ref:`built-in functionality doumentation <core_source_builtins>` for
80	130	details on common configuration options for sources.
81	131
...	...	@@ -95,6 +145,7 @@ import re
95	145	import shutil
96	146	from collections.abc import Mapping
97	147	from io import StringIO
	148	+from tempfile import TemporaryFile
98	149
99	150	from configparser import RawConfigParser
100	151
...	...	@@ -115,13 +166,14 @@ INCONSISTENT_SUBMODULE = "inconsistent-submodules"
115	166	#
116	167	class GitMirror(SourceFetcher):
117	168
118		- def __init__(self, source, path, url, ref, *, primary=False):
	169	+ def __init__(self, source, path, url, ref, *, primary=False, tags=[]):
119	170
120	171	super().__init__()
121	172	self.source = source
122	173	self.path = path
123	174	self.url = url
124	175	self.ref = ref
	176	+ self.tags = tags
125	177	self.primary = primary
126	178	self.mirror = os.path.join(source.get_mirror_directory(), utils.url_directory_name(url))
127	179	self.mark_download_url(url)
...	...	@@ -214,7 +266,7 @@ class GitMirror(SourceFetcher):
214	266	raise SourceError("{}: expected ref '{}' was not found in git repository: '{}'"
215	267	.format(self.source, self.ref, self.url))
216	268
217		- def latest_commit(self, tracking):
	269	+ def latest_commit_with_tags(self, tracking, track_tags=False):
218	270	_, output = self.source.check_output(
219	271	[self.source.host_git, 'rev-parse', tracking],
220	272	fail="Unable to find commit for specified branch name '{}'".format(tracking),
...	...	@@ -230,7 +282,28 @@ class GitMirror(SourceFetcher):
230	282	if exit_code == 0:
231	283	ref = output.rstrip('\n')
232	284
233		- return ref
	285	+ if not track_tags:
	286	+ return ref, []
	287	+
	288	+ tags = set()
	289	+ for options in [[], ['--first-parent'], ['--tags'], ['--tags', '--first-parent']]:
	290	+ exit_code, output = self.source.check_output(
	291	+ [self.source.host_git, 'describe', '--abbrev=0', ref] + options,
	292	+ cwd=self.mirror)
	293	+ if exit_code == 0:
	294	+ tag = output.strip()
	295	+ _, commit_ref = self.source.check_output(
	296	+ [self.source.host_git, 'rev-parse', tag + '^{commit}'],
	297	+ fail="Unable to resolve tag '{}'".format(tag),
	298	+ cwd=self.mirror)
	299	+ exit_code = self.source.call(
	300	+ [self.source.host_git, 'cat-file', 'tag', tag],
	301	+ cwd=self.mirror)
	302	+ annotated = (exit_code == 0)
	303	+
	304	+ tags.add((tag, commit_ref.strip(), annotated))
	305	+
	306	+ return ref, list(tags)
234	307
235	308	def stage(self, directory, track=None):
236	309	fullpath = os.path.join(directory, self.path)
...	...	@@ -246,13 +319,15 @@ class GitMirror(SourceFetcher):
246	319	fail="Failed to checkout git ref {}".format(self.ref),
247	320	cwd=fullpath)
248	321
	322	+ # Remove .git dir
	323	+ shutil.rmtree(os.path.join(fullpath, ".git"))
	324	+
	325	+ self._rebuild_git(fullpath)
	326	+
249	327	# Check that the user specified ref exists in the track if provided & not already tracked
250	328	if track:
251	329	self.assert_ref_in_track(fullpath, track)
252	330
253		- # Remove .git dir
254		- shutil.rmtree(os.path.join(fullpath, ".git"))
255		-
256	331	def init_workspace(self, directory, track=None):
257	332	fullpath = os.path.join(directory, self.path)
258	333	url = self.source.translate_url(self.url)
...	...	@@ -273,6 +348,35 @@ class GitMirror(SourceFetcher):
273	348	if track:
274	349	self.assert_ref_in_track(fullpath, track)
275	350
	351	+ def init_cached_build_workspace(self, directory, track=None):
	352	+ fullpath = os.path.join(directory, self.path)
	353	+ url = self.source.translate_url(self.url)
	354	+
	355	+ self.source.call([self.source.host_git, 'init', fullpath],
	356	+ fail="Failed to init git in directory: {}".format(fullpath),
	357	+ fail_temporarily=True,
	358	+ cwd=fullpath)
	359	+
	360	+ self.source.call([self.source.host_git, 'fetch', self.mirror],
	361	+ fail='Failed to fetch from local mirror "{}"'.format(self.mirror),
	362	+ cwd=fullpath)
	363	+
	364	+ self.source.call([self.source.host_git, 'remote', 'add', 'origin', url],
	365	+ fail='Failed to add remote origin "{}"'.format(url),
	366	+ cwd=fullpath)
	367	+
	368	+ self.source.call([self.source.host_git, 'update-ref', '--no-deref', 'HEAD', self.ref],
	369	+ fail='Failed update HEAD to ref "{}"'.format(self.ref),
	370	+ cwd=fullpath)
	371	+
	372	+ self.source.call([self.source.host_git, 'read-tree', 'HEAD'],
	373	+ fail='Failed to read HEAD into index',
	374	+ cwd=fullpath)
	375	+
	376	+ # Check that the user specified ref exists in the track if provided & not already tracked
	377	+ if track:
	378	+ self.assert_ref_in_track(fullpath, track)
	379	+
276	380	# List the submodules (path/url tuples) present at the given ref of this repo
277	381	def submodule_list(self):
278	382	modules = "{}:{}".format(self.ref, GIT_MODULES)
...	...	@@ -359,6 +463,78 @@ class GitMirror(SourceFetcher):
359	463	.format(self.source, self.ref, track, self.url),
360	464	detail=detail, warning_token=CoreWarnings.REF_NOT_IN_TRACK)
361	465
	466	+ def _rebuild_git(self, fullpath):
	467	+ if not self.tags:
	468	+ return
	469	+
	470	+ with self.source.tempdir() as tmpdir:
	471	+ included = set()
	472	+ shallow = set()
	473	+ for _, commit_ref, _ in self.tags:
	474	+
	475	+ _, out = self.source.check_output([self.source.host_git, 'rev-list',
	476	+ '--boundary', '{}..{}'.format(commit_ref, self.ref)],
	477	+ fail="Failed to get git history {}..{} in directory: {}"
	478	+ .format(commit_ref, self.ref, fullpath),
	479	+ fail_temporarily=True,
	480	+ cwd=self.mirror)
	481	+ for line in out.splitlines():
	482	+ rev = line.lstrip('-')
	483	+ if line[0] == '-':
	484	+ shallow.add(rev)
	485	+ else:
	486	+ included.add(rev)
	487	+
	488	+ shallow -= included
	489	+ included |= shallow
	490	+
	491	+ self.source.call([self.source.host_git, 'init'],
	492	+ fail="Cannot initialize git repository: {}".format(fullpath),
	493	+ cwd=fullpath)
	494	+
	495	+ for rev in included:
	496	+ with TemporaryFile(dir=tmpdir) as commit_file:
	497	+ self.source.call([self.source.host_git, 'cat-file', 'commit', rev],
	498	+ stdout=commit_file,
	499	+ fail="Failed to get commit {}".format(rev),
	500	+ cwd=self.mirror)
	501	+ commit_file.seek(0, 0)
	502	+ self.source.call([self.source.host_git, 'hash-object', '-w', '-t', 'commit', '--stdin'],
	503	+ stdin=commit_file,
	504	+ fail="Failed to add commit object {}".format(rev),
	505	+ cwd=fullpath)
	506	+
	507	+ with open(os.path.join(fullpath, '.git', 'shallow'), 'w') as shallow_file:
	508	+ for rev in shallow:
	509	+ shallow_file.write('{}\n'.format(rev))
	510	+
	511	+ for tag, commit_ref, annotated in self.tags:
	512	+ if annotated:
	513	+ with TemporaryFile(dir=tmpdir) as tag_file:
	514	+ tag_data = 'object {}\ntype commit\ntag {}\n'.format(commit_ref, tag)
	515	+ tag_file.write(tag_data.encode('ascii'))
	516	+ tag_file.seek(0, 0)
	517	+ _, tag_ref = self.source.check_output(
	518	+ [self.source.host_git, 'hash-object', '-w', '-t',
	519	+ 'tag', '--stdin'],
	520	+ stdin=tag_file,
	521	+ fail="Failed to add tag object {}".format(tag),
	522	+ cwd=fullpath)
	523	+
	524	+ self.source.call([self.source.host_git, 'tag', tag, tag_ref.strip()],
	525	+ fail="Failed to tag: {}".format(tag),
	526	+ cwd=fullpath)
	527	+ else:
	528	+ self.source.call([self.source.host_git, 'tag', tag, commit_ref],
	529	+ fail="Failed to tag: {}".format(tag),
	530	+ cwd=fullpath)
	531	+
	532	+ with open(os.path.join(fullpath, '.git', 'HEAD'), 'w') as head:
	533	+ self.source.call([self.source.host_git, 'rev-parse', self.ref],
	534	+ stdout=head,
	535	+ fail="Failed to parse commit {}".format(self.ref),
	536	+ cwd=self.mirror)
	537	+
362	538
363	539	class GitSource(Source):
364	540	# pylint: disable=attribute-defined-outside-init
...	...	@@ -366,11 +542,20 @@ class GitSource(Source):
366	542	def configure(self, node):
367	543	ref = self.node_get_member(node, str, 'ref', None)
368	544
369		- config_keys = ['url', 'track', 'ref', 'submodules', 'checkout-submodules', 'ref-format']
	545	+ config_keys = ['url', 'track', 'ref', 'submodules',
	546	+ 'checkout-submodules', 'ref-format',
	547	+ 'track-tags', 'tags']
370	548	self.node_validate(node, config_keys + Source.COMMON_CONFIG_KEYS)
371	549
	550	+ tags_node = self.node_get_member(node, list, 'tags', [])
	551	+ for tag_node in tags_node:
	552	+ self.node_validate(tag_node, ['tag', 'commit', 'annotated'])
	553	+
	554	+ tags = self._load_tags(node)
	555	+ self.track_tags = self.node_get_member(node, bool, 'track-tags', False)
	556	+
372	557	self.original_url = self.node_get_member(node, str, 'url')
373		- self.mirror = GitMirror(self, '', self.original_url, ref, primary=True)
	558	+ self.mirror = GitMirror(self, '', self.original_url, ref, tags=tags, primary=True)
374	559	self.tracking = self.node_get_member(node, str, 'track', None)
375	560
376	561	self.ref_format = self.node_get_member(node, str, 'ref-format', 'sha1')
...	...	@@ -417,6 +602,9 @@ class GitSource(Source):
417	602	# the ref, if the user changes the alias to fetch the same sources
418	603	# from another location, it should not affect the cache key.
419	604	key = [self.original_url, self.mirror.ref]
	605	+ if self.mirror.tags:
	606	+ tags = {tag: (commit, annotated) for tag, commit, annotated in self.mirror.tags}
	607	+ key.append({'tags': tags})
420	608
421	609	# Only modify the cache key with checkout_submodules if it's something
422	610	# other than the default behaviour.
...	...	@@ -442,12 +630,33 @@ class GitSource(Source):
442	630
443	631	def load_ref(self, node):
444	632	self.mirror.ref = self.node_get_member(node, str, 'ref', None)
	633	+ self.mirror.tags = self._load_tags(node)
445	634
446	635	def get_ref(self):
447		- return self.mirror.ref
448		-
449		- def set_ref(self, ref, node):
450		- node['ref'] = self.mirror.ref = ref
	636	+ return self.mirror.ref, self.mirror.tags
	637	+
	638	+ def set_ref(self, ref_data, node):
	639	+ if not ref_data:
	640	+ self.mirror.ref = None
	641	+ if 'ref' in node:
	642	+ del node['ref']
	643	+ self.mirror.tags = []
	644	+ if 'tags' in node:
	645	+ del node['tags']
	646	+ else:
	647	+ ref, tags = ref_data
	648	+ node['ref'] = self.mirror.ref = ref
	649	+ self.mirror.tags = tags
	650	+ if tags:
	651	+ node['tags'] = []
	652	+ for tag, commit_ref, annotated in tags:
	653	+ data = {'tag': tag,
	654	+ 'commit': commit_ref,
	655	+ 'annotated': annotated}
	656	+ node['tags'].append(data)
	657	+ else:
	658	+ if 'tags' in node:
	659	+ del node['tags']
451	660
452	661	def track(self):
453	662
...	...	@@ -470,7 +679,7 @@ class GitSource(Source):
470	679	self.mirror._fetch()
471	680
472	681	# Update self.mirror.ref and node.ref from the self.tracking branch
473		- ret = self.mirror.latest_commit(self.tracking)
	682	+ ret = self.mirror.latest_commit_with_tags(self.tracking, self.track_tags)
474	683
475	684	# Set tracked attribute, parameter for if self.mirror.assert_ref_in_track is needed
476	685	self.tracked = True
...	...	@@ -485,6 +694,14 @@ class GitSource(Source):
485	694	for mirror in self.submodules:
486	695	mirror.init_workspace(directory)
487	696
	697	+ def init_cached_build_workspace(self, directory):
	698	+ self.refresh_submodules()
	699	+
	700	+ with self.timed_activity('Setting up workspace "{}"'.format(directory), silent_nested=True):
	701	+ self.mirror.init_cached_build_workspace(directory, track=(self.tracking if not self.tracked else None))
	702	+ for mirror in self.submodules:
	703	+ mirror.init_cached_build_workspace(directory)
	704	+
488	705	def stage(self, directory):
489	706
490	707	# Need to refresh submodule list here again, because
...	...	@@ -556,6 +773,16 @@ class GitSource(Source):
556	773
557	774	self.submodules = submodules
558	775
	776	+ def _load_tags(self, node):
	777	+ tags = []
	778	+ tags_node = self.node_get_member(node, list, 'tags', [])
	779	+ for tag_node in tags_node:
	780	+ tag = self.node_get_member(tag_node, str, 'tag')
	781	+ commit_ref = self.node_get_member(tag_node, str, 'commit')
	782	+ annotated = self.node_get_member(tag_node, bool, 'annotated')
	783	+ tags.append((tag, commit_ref, annotated))
	784	+ return tags
	785	+
559	786
560	787	# Plugin entry point
561	788	def setup():

buildstream/source.py

---

...	...	@@ -459,6 +459,24 @@ class Source(Plugin):
459	459	"""
460	460	self.stage(directory)
461	461
	462	+ def init_cached_build_workspace(self, directory):
	463	+ """Initialises a new cached build workspace
	464	+
	465	+ Args:
	466	+ directory (str): Path of the workspace to init
	467	+
	468	+ Raises:
	469	+ :class:`.SourceError`
	470	+
	471	+ Implementors overriding this method should assume that *directory*
	472	+ already exists.
	473	+
	474	+ Implementors should raise :class:`.SourceError` when encountering
	475	+ some system error.
	476	+ """
	477	+ # Allow a non implementation
	478	+ return None
	479	+
462	480	def get_source_fetchers(self):
463	481	"""Get the objects that are used for fetching
464	482
...	...	@@ -694,6 +712,12 @@ class Source(Plugin):
694	712
695	713	self.init_workspace(directory)
696	714
	715	+ # Wrapper for init_cached_build_workspace()
	716	+ def _init_cached_build_workspace(self, directory):
	717	+ directory = self.__ensure_directory(directory)
	718	+
	719	+ self.init_cached_build_workspace(directory)
	720	+
697	721	# _get_unique_key():
698	722	#
699	723	# Wrapper for get_unique_key() api

doc/source/developing/workspaces.rst

---

...	...	@@ -24,9 +24,32 @@ Suppose we now want to alter the functionality of the *hello* command. We can
24	24	make changes to the source code of Buildstream elements by making use of
25	25	BuildStream's workspace command.
26	26
	27	+Utilising cached buildtrees
	28	+---------------------------
	29	+ When a BuildStream build element artifact is created and cached, a snapshot of
	30	+ the build directory after the build commands have completed is included in the
	31	+ artifact. This `build tree` can be considered an intermediary state of element,
	32	+ where the source is present along with any output created during the build
	33	+ execution.
	34	+
	35	+ By default when opening a workspace, bst will attempt to stage the build tree
	36	+ into the workspace if it's available in the local cache. If the respective
	37	+ build tree is not present in the cache (element not cached, partially cached or
	38	+ is a non build element) then the source will be staged as is. The default
	39	+ behaviour to attempt to use the build tree can be overriden with specific bst
	40	+ workspace open option of `--no-cache`, or via setting user configuration option
	41	+ `workspacebuildtrees: False`
	42	+
27	43
28	44	Opening a workspace
29	45	-------------------
	46	+.. note::
	47	+
	48	+ This example presumes you built the hello.bst during
	49	+ :ref:`running commands <tutorial_running_commands>`
	50	+ if not, please start by building it.
	51	+
	52	+
30	53	First we need to open a workspace, we can do this by running
31	54
32	55	.. raw:: html
...	...	@@ -88,6 +111,15 @@ Alternatively, if we wish to discard the changes we can use
88	111
89	112	This resets the workspace to its original state.
90	113
	114	+.. note::
	115	+
	116	+ bst reset will attempt to open the workspace in
	117	+ the condition in which it was originally staged,
	118	+ i.e with or without consuming the element build tree.
	119	+ If it was originally staged with a cached build tree
	120	+ and there's no longer one available, the source will
	121	+ be staged as is.
	122	+
91	123	To discard the workspace completely we can do:
92	124
93	125	.. raw:: html

tests/cachekey/project/sources/git3.bst

---

	1	+kind: import
	2	+sources:
	3	+- kind: git
	4	+ url: https://example.com/git/repo.git
	5	+ ref: 6ac68af3e80b7b17c23a3c65233043550a7fa685
	6	+ tags:
	7	+ - tag: lightweight
	8	+ commit: 0a3917d57477ee9afe7be49a0e8a76f56d176df1
	9	+ annotated: false
	10	+ - tag: annotated
	11	+ commit: 68c7f0bd386684742c41ec2a54ce2325e3922f6c
	12	+ annotated: true

tests/cachekey/project/sources/git3.expected

---

	1	+6a25f539bd8629a36399c58efd2f5c9c117feb845076a37dc321b55d456932b6
		\ No newline at end of file

tests/cachekey/project/target.bst

---

...	...	@@ -7,6 +7,7 @@ depends:
7	7	- sources/bzr1.bst
8	8	- sources/git1.bst
9	9	- sources/git2.bst
	10	+- sources/git3.bst
10	11	- sources/local1.bst
11	12	- sources/local2.bst
12	13	- sources/ostree1.bst

tests/cachekey/project/target.expected

---

1		-125d9e7dcf4f49e5f80d85b7f144b43ed43186064afc2e596e57f26cce679cf5
		\ No newline at end of file
	1	+bc99c288f855ac2619787f0067223f7812d2e10a9d2c7f2bf47de7113c0fd25c
		\ No newline at end of file

tests/frontend/workspace.py

---

...	...	@@ -91,18 +91,18 @@ class WorkspaceCreater():
91	91	element_name))
92	92	return element_name, element_path, workspace_dir
93	93
94		- def create_workspace_elements(self, kinds, track, suffixs=None, workspace_dir_usr=None,
	94	+ def create_workspace_elements(self, kinds, track, suffixes=None, workspace_dir_usr=None,
95	95	element_attrs=None):
96	96
97	97	element_tuples = []
98	98
99		- if suffixs is None:
100		- suffixs = ['', ] * len(kinds)
	99	+ if suffixes is None:
	100	+ suffixes = ['', ] * len(kinds)
101	101	else:
102		- if len(suffixs) != len(kinds):
	102	+ if len(suffixes) != len(kinds):
103	103	raise "terable error"
104	104
105		- for suffix, kind in zip(suffixs, kinds):
	105	+ for suffix, kind in zip(suffixes, kinds):
106	106	element_name, element_path, workspace_dir = \
107	107	self.create_workspace_element(kind, track, suffix, workspace_dir_usr,
108	108	element_attrs)
...	...	@@ -117,10 +117,10 @@ class WorkspaceCreater():
117	117
118	118	return element_tuples
119	119
120		- def open_workspaces(self, kinds, track, suffixs=None, workspace_dir=None,
121		- element_attrs=None):
	120	+ def open_workspaces(self, kinds, track, suffixes=None, workspace_dir=None,
	121	+ element_attrs=None, no_cache=False):
122	122
123		- element_tuples = self.create_workspace_elements(kinds, track, suffixs, workspace_dir,
	123	+ element_tuples = self.create_workspace_elements(kinds, track, suffixes, workspace_dir,
124	124	element_attrs)
125	125	os.makedirs(self.workspace_cmd, exist_ok=True)
126	126
...	...	@@ -129,12 +129,15 @@ class WorkspaceCreater():
129	129	args = ['workspace', 'open']
130	130	if track:
131	131	args.append('--track')
	132	+ if no_cache:
	133	+ args.append('--no-cache')
132	134	if workspace_dir is not None:
133	135	assert len(element_tuples) == 1, "test logic error"
134	136	_, workspace_dir = element_tuples[0]
135	137	args.extend(['--directory', workspace_dir])
136		-
	138	+ print("element_tuples", element_tuples)
137	139	args.extend([element_name for element_name, workspace_dir_suffix in element_tuples])
	140	+ print("args", args)
138	141	result = self.cli.run(cwd=self.workspace_cmd, project=self.project_path, args=args)
139	142
140	143	result.assert_success()
...	...	@@ -148,14 +151,14 @@ class WorkspaceCreater():
148	151	filename = os.path.join(workspace_dir, 'usr', 'bin', 'hello')
149	152	assert os.path.exists(filename)
150	153
151		- return element_tuples
	154	+ return element_tuples, result
152	155
153	156
154	157	def open_workspace(cli, tmpdir, datafiles, kind, track, suffix='', workspace_dir=None,
155		- project_path=None, element_attrs=None):
	158	+ project_path=None, element_attrs=None, no_cache=False):
156	159	workspace_object = WorkspaceCreater(cli, tmpdir, datafiles, project_path)
157		- workspaces = workspace_object.open_workspaces((kind, ), track, (suffix, ), workspace_dir,
158		- element_attrs)
	160	+ workspaces, _ = workspace_object.open_workspaces((kind, ), track, (suffix, ), workspace_dir,
	161	+ element_attrs, no_cache)
159	162	assert len(workspaces) == 1
160	163	element_name, workspace = workspaces[0]
161	164	return element_name, workspace_object.project_path, workspace
...	...	@@ -189,7 +192,7 @@ def test_open_bzr_customize(cli, tmpdir, datafiles):
189	192	def test_open_multi(cli, tmpdir, datafiles):
190	193
191	194	workspace_object = WorkspaceCreater(cli, tmpdir, datafiles)
192		- workspaces = workspace_object.open_workspaces(repo_kinds, False)
	195	+ workspaces, _ = workspace_object.open_workspaces(repo_kinds, False)
193	196
194	197	for (elname, workspace), kind in zip(workspaces, repo_kinds):
195	198	assert kind in elname
...	...	@@ -816,7 +819,9 @@ def test_list_unsupported_workspace(cli, tmpdir, datafiles, workspace_cfg):
816	819	"alpha.bst": {
817	820	"prepared": False,
818	821	"path": "/workspaces/bravo",
819		- "running_files": {}
	822	+ "running_files": {},
	823	+ "cached_build": False
	824	+
820	825	}
821	826	}
822	827	}),
...	...	@@ -831,7 +836,8 @@ def test_list_unsupported_workspace(cli, tmpdir, datafiles, workspace_cfg):
831	836	"alpha.bst": {
832	837	"prepared": False,
833	838	"path": "/workspaces/bravo",
834		- "running_files": {}
	839	+ "running_files": {},
	840	+ "cached_build": False
835	841	}
836	842	}
837	843	}),
...	...	@@ -849,7 +855,8 @@ def test_list_unsupported_workspace(cli, tmpdir, datafiles, workspace_cfg):
849	855	"alpha.bst": {
850	856	"prepared": False,
851	857	"path": "/workspaces/bravo",
852		- "running_files": {}
	858	+ "running_files": {},
	859	+ "cached_build": False
853	860	}
854	861	}
855	862	}),
...	...	@@ -874,7 +881,8 @@ def test_list_unsupported_workspace(cli, tmpdir, datafiles, workspace_cfg):
874	881	"last_successful": "some_key",
875	882	"running_files": {
876	883	"beta.bst": ["some_file"]
877		- }
	884	+ },
	885	+ "cached_build": False
878	886	}
879	887	}
880	888	}),
...	...	@@ -894,7 +902,30 @@ def test_list_unsupported_workspace(cli, tmpdir, datafiles, workspace_cfg):
894	902	"alpha.bst": {
895	903	"prepared": True,
896	904	"path": "/workspaces/bravo",
897		- "running_files": {}
	905	+ "running_files": {},
	906	+ "cached_build": False
	907	+ }
	908	+ }
	909	+ }),
	910	+ # Test loading version 4
	911	+ ({
	912	+ "format-version": 4,
	913	+ "workspaces": {
	914	+ "alpha.bst": {
	915	+ "prepared": False,
	916	+ "path": "/workspaces/bravo",
	917	+ "running_files": {},
	918	+ "cached_build": True
	919	+ }
	920	+ }
	921	+ }, {
	922	+ "format-version": BST_WORKSPACE_FORMAT_VERSION,
	923	+ "workspaces": {
	924	+ "alpha.bst": {
	925	+ "prepared": False,
	926	+ "path": "/workspaces/bravo",
	927	+ "running_files": {},
	928	+ "cached_build": True
898	929	}
899	930	}
900	931	})
...	...	@@ -1055,3 +1086,80 @@ def test_multiple_failed_builds(cli, tmpdir, datafiles):
1055	1086	result = cli.run(project=project, args=["build", element_name])
1056	1087	assert "BUG" not in result.stderr
1057	1088	assert cli.get_element_state(project, element_name) != "cached"
	1089	+
	1090	+
	1091	+@pytest.mark.datafiles(DATA_DIR)
	1092	+def test_nocache_open_messages(cli, tmpdir, datafiles):
	1093	+
	1094	+ workspace_object = WorkspaceCreater(cli, tmpdir, datafiles)
	1095	+ _, result = workspace_object.open_workspaces(('git', ), False)
	1096	+
	1097	+ # cli default WARN for source dropback possibility when no-cache flag is not passed
	1098	+ assert "WARNING: Workspace will be opened without the cached buildtree if not cached locally" in result.output
	1099	+
	1100	+ # cli WARN for source dropback happening when no-cache flag not given, but buildtree not available
	1101	+ assert "workspace will be opened with source checkout" in result.stderr
	1102	+
	1103	+ # cli default WARN for source dropback possibilty not given when no-cache flag is passed
	1104	+ tmpdir = os.path.join(str(tmpdir), "2")
	1105	+ workspace_object = WorkspaceCreater(cli, tmpdir, datafiles)
	1106	+ _, result = workspace_object.open_workspaces(('git', ), False, suffixes='1', no_cache=True)
	1107	+
	1108	+ assert "WARNING: Workspace will be opened without the cached buildtree if not cached locally" not in result.output
	1109	+
	1110	+
	1111	+@pytest.mark.datafiles(DATA_DIR)
	1112	+def test_nocache_reset_messages(cli, tmpdir, datafiles):
	1113	+
	1114	+ workspace_object = WorkspaceCreater(cli, tmpdir, datafiles)
	1115	+ workspaces, result = workspace_object.open_workspaces(('git', ), False)
	1116	+ element_name, workspace = workspaces[0]
	1117	+ project = workspace_object.project_path
	1118	+
	1119	+ # Modify workspace, without building so the artifact is not cached
	1120	+ shutil.rmtree(os.path.join(workspace, 'usr', 'bin'))
	1121	+ os.makedirs(os.path.join(workspace, 'etc'))
	1122	+ with open(os.path.join(workspace, 'etc', 'pony.conf'), 'w') as f:
	1123	+ f.write("PONY='pink'")
	1124	+
	1125	+ # Now reset the open workspace, this should have the
	1126	+ # effect of reverting our changes to the original source, as it
	1127	+ # was not originally opened with a cached buildtree and as such
	1128	+ # should not notify the user
	1129	+ result = cli.run(cwd=workspace_object.workspace_cmd, project=project, args=[
	1130	+ 'workspace', 'reset', element_name
	1131	+ ])
	1132	+ result.assert_success()
	1133	+ assert "original buildtree artifact not available" not in result.output
	1134	+ assert os.path.exists(os.path.join(workspace, 'usr', 'bin', 'hello'))
	1135	+ assert not os.path.exists(os.path.join(workspace, 'etc', 'pony.conf'))
	1136	+
	1137	+ # Close the workspace
	1138	+ result = cli.run(cwd=workspace_object.workspace_cmd, project=project, args=[
	1139	+ 'workspace', 'close', '--remove-dir', element_name
	1140	+ ])
	1141	+ result.assert_success()
	1142	+
	1143	+ # Build the workspace so we have a cached buildtree artifact for the element
	1144	+ assert cli.get_element_state(project, element_name) == 'buildable'
	1145	+ result = cli.run(project=project, args=['build', element_name])
	1146	+ result.assert_success()
	1147	+
	1148	+ # Opening the workspace after a build should lead to the cached buildtree being
	1149	+ # staged by default
	1150	+ result = cli.run(cwd=workspace_object.workspace_cmd, project=project, args=[
	1151	+ 'workspace', 'open', element_name
	1152	+ ])
	1153	+ result.assert_success()
	1154	+
	1155	+ result = cli.run(cwd=workspace_object.workspace_cmd, project=project, args=[
	1156	+ 'workspace', 'list'
	1157	+ ])
	1158	+ result.assert_success()
	1159	+ # Now reset the workspace and ensure that a warning is not given about the artifact
	1160	+ # buildtree not being available
	1161	+ result = cli.run(cwd=workspace_object.workspace_cmd, project=project, args=[
	1162	+ 'workspace', 'reset', element_name
	1163	+ ])
	1164	+ result.assert_success()
	1165	+ assert "original buildtree artifact not available" not in result.output

tests/integration/workspace.py

---

...	...	@@ -278,3 +278,39 @@ def test_incremental_configure_commands_run_only_once(cli, tmpdir, datafiles):
278	278	res = cli.run(project=project, args=['build', element_name])
279	279	res.assert_success()
280	280	assert not os.path.exists(os.path.join(workspace, 'prepared-again'))
	281	+
	282	+
	283	+@pytest.mark.integration
	284	+@pytest.mark.datafiles(DATA_DIR)
	285	+@pytest.mark.skipif(IS_LINUX and not HAVE_BWRAP, reason='Only available with bubblewrap on Linux')
	286	+def test_workspace_contains_buildtree(cli, tmpdir, datafiles):
	287	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	288	+ workspace = os.path.join(cli.directory, 'workspace')
	289	+ element_name = 'autotools/amhello.bst'
	290	+
	291	+ # Ensure we're not using the shared artifact cache
	292	+ cli.configure({
	293	+ 'artifactdir': os.path.join(str(tmpdir), 'artifacts')
	294	+ })
	295	+
	296	+ # First open the workspace
	297	+ res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
	298	+ res.assert_success()
	299	+
	300	+ # Check that by default the buildtree wasn't staged as not yet available in the cache
	301	+ assert not os.path.exists(os.path.join(workspace, 'src', 'hello'))
	302	+
	303	+ # Close the workspace, removing the dir
	304	+ res = cli.run(project=project, args=['workspace', 'close', '--remove-dir', element_name])
	305	+ res.assert_success()
	306	+
	307	+ # Build the element, so we have it cached along with the buildtreee
	308	+ res = cli.run(project=project, args=['build', element_name])
	309	+ res.assert_success()
	310	+
	311	+ # Open up the workspace, as the buildtree is cached by default it should open with the buildtree
	312	+ res = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, element_name])
	313	+ res.assert_success()
	314	+
	315	+ # Check that the buildtree was staged, by asserting output of the build exists in the dir
	316	+ assert os.path.exists(os.path.join(workspace, 'src', 'hello'))

tests/sources/git.py

---

...	...	@@ -22,6 +22,7 @@
22	22
23	23	import os
24	24	import pytest
	25	+import subprocess
25	26
26	27	from buildstream._exceptions import ErrorDomain
27	28	from buildstream import _yaml
...	...	@@ -523,3 +524,155 @@ def test_track_fetch(cli, tmpdir, datafiles, ref_format, tag, extra_commit):
523	524	# Fetch it
524	525	result = cli.run(project=project, args=['fetch', 'target.bst'])
525	526	result.assert_success()
	527	+
	528	+
	529	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	530	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	531	+@pytest.mark.parametrize("ref_storage", [('inline'), ('project.refs')])
	532	+@pytest.mark.parametrize("tag_type", [('annotated'), ('lightweight')])
	533	+def test_git_describe(cli, tmpdir, datafiles, ref_storage, tag_type):
	534	+ project = str(datafiles)
	535	+
	536	+ project_config = _yaml.load(os.path.join(project, 'project.conf'))
	537	+ project_config['ref-storage'] = ref_storage
	538	+ _yaml.dump(_yaml.node_sanitize(project_config), os.path.join(project, 'project.conf'))
	539	+
	540	+ repofiles = os.path.join(str(tmpdir), 'repofiles')
	541	+ os.makedirs(repofiles, exist_ok=True)
	542	+ file0 = os.path.join(repofiles, 'file0')
	543	+ with open(file0, 'w') as f:
	544	+ f.write('test\n')
	545	+
	546	+ repo = create_repo('git', str(tmpdir))
	547	+
	548	+ def tag(name):
	549	+ if tag_type == 'annotated':
	550	+ repo.add_annotated_tag(name, name)
	551	+ else:
	552	+ repo.add_tag(name)
	553	+
	554	+ ref = repo.create(repofiles)
	555	+ tag('uselesstag')
	556	+
	557	+ file1 = os.path.join(str(tmpdir), 'file1')
	558	+ with open(file1, 'w') as f:
	559	+ f.write('test\n')
	560	+ repo.add_file(file1)
	561	+ tag('tag1')
	562	+
	563	+ file2 = os.path.join(str(tmpdir), 'file2')
	564	+ with open(file2, 'w') as f:
	565	+ f.write('test\n')
	566	+ repo.branch('branch2')
	567	+ repo.add_file(file2)
	568	+ tag('tag2')
	569	+
	570	+ repo.checkout('master')
	571	+ file3 = os.path.join(str(tmpdir), 'file3')
	572	+ with open(file3, 'w') as f:
	573	+ f.write('test\n')
	574	+ repo.add_file(file3)
	575	+
	576	+ repo.merge('branch2')
	577	+
	578	+ config = repo.source_config()
	579	+ config['track'] = repo.latest_commit()
	580	+ config['track-tags'] = True
	581	+
	582	+ # Write out our test target
	583	+ element = {
	584	+ 'kind': 'import',
	585	+ 'sources': [
	586	+ config
	587	+ ],
	588	+ }
	589	+ element_path = os.path.join(project, 'target.bst')
	590	+ _yaml.dump(element, element_path)
	591	+
	592	+ if ref_storage == 'inline':
	593	+ result = cli.run(project=project, args=['track', 'target.bst'])
	594	+ result.assert_success()
	595	+ else:
	596	+ result = cli.run(project=project, args=['track', 'target.bst', '--deps', 'all'])
	597	+ result.assert_success()
	598	+
	599	+ if ref_storage == 'inline':
	600	+ element = _yaml.load(element_path)
	601	+ tags = _yaml.node_sanitize(element['sources'][0]['tags'])
	602	+ assert len(tags) == 2
	603	+ for tag in tags:
	604	+ assert 'tag' in tag
	605	+ assert 'commit' in tag
	606	+ assert 'annotated' in tag
	607	+ assert tag['annotated'] == (tag_type == 'annotated')
	608	+
	609	+ assert set([(tag['tag'], tag['commit']) for tag in tags]) == set([('tag1', repo.rev_parse('tag1^{commit}')),
	610	+ ('tag2', repo.rev_parse('tag2^{commit}'))])
	611	+
	612	+ checkout = os.path.join(str(tmpdir), 'checkout')
	613	+
	614	+ result = cli.run(project=project, args=['build', 'target.bst'])
	615	+ result.assert_success()
	616	+ result = cli.run(project=project, args=['checkout', 'target.bst', checkout])
	617	+ result.assert_success()
	618	+
	619	+ if tag_type == 'annotated':
	620	+ options = []
	621	+ else:
	622	+ options = ['--tags']
	623	+ describe = subprocess.check_output(['git', 'describe'] + options,
	624	+ cwd=checkout).decode('ascii')
	625	+ assert describe.startswith('tag2-2-')
	626	+
	627	+ describe_fp = subprocess.check_output(['git', 'describe', '--first-parent'] + options,
	628	+ cwd=checkout).decode('ascii')
	629	+ assert describe_fp.startswith('tag1-2-')
	630	+
	631	+ tags = subprocess.check_output(['git', 'tag'],
	632	+ cwd=checkout).decode('ascii')
	633	+ tags = set(tags.splitlines())
	634	+ assert tags == set(['tag1', 'tag2'])
	635	+
	636	+ p = subprocess.run(['git', 'log', repo.rev_parse('uselesstag')],
	637	+ cwd=checkout)
	638	+ assert p.returncode != 0
	639	+
	640	+
	641	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	642	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	643	+def test_default_do_not_track_tags(cli, tmpdir, datafiles):
	644	+ project = str(datafiles)
	645	+
	646	+ project_config = _yaml.load(os.path.join(project, 'project.conf'))
	647	+ project_config['ref-storage'] = 'inline'
	648	+ _yaml.dump(_yaml.node_sanitize(project_config), os.path.join(project, 'project.conf'))
	649	+
	650	+ repofiles = os.path.join(str(tmpdir), 'repofiles')
	651	+ os.makedirs(repofiles, exist_ok=True)
	652	+ file0 = os.path.join(repofiles, 'file0')
	653	+ with open(file0, 'w') as f:
	654	+ f.write('test\n')
	655	+
	656	+ repo = create_repo('git', str(tmpdir))
	657	+
	658	+ ref = repo.create(repofiles)
	659	+ repo.add_tag('tag')
	660	+
	661	+ config = repo.source_config()
	662	+ config['track'] = repo.latest_commit()
	663	+
	664	+ # Write out our test target
	665	+ element = {
	666	+ 'kind': 'import',
	667	+ 'sources': [
	668	+ config
	669	+ ],
	670	+ }
	671	+ element_path = os.path.join(project, 'target.bst')
	672	+ _yaml.dump(element, element_path)
	673	+
	674	+ result = cli.run(project=project, args=['track', 'target.bst'])
	675	+ result.assert_success()
	676	+
	677	+ element = _yaml.load(element_path)
	678	+ assert 'tags' not in element['sources'][0]

tests/testutils/repo/git.py

---

...	...	@@ -45,6 +45,9 @@ class Git(Repo):
45	45	def add_tag(self, tag):
46	46	self._run_git('tag', tag)
47	47
	48	+ def add_annotated_tag(self, tag, message):
	49	+ self._run_git('tag', '-a', tag, '-m', message)
	50	+
48	51	def add_commit(self):
49	52	self._run_git('commit', '--allow-empty', '-m', 'Additional commit')
50	53	return self.latest_commit()
...	...	@@ -95,3 +98,14 @@ class Git(Repo):
95	98
96	99	def branch(self, branch_name):
97	100	self._run_git('checkout', '-b', branch_name)
	101	+
	102	+ def checkout(self, commit):
	103	+ self._run_git('checkout', commit)
	104	+
	105	+ def merge(self, commit):
	106	+ self._run_git('merge', '-m', 'Merge', commit)
	107	+ return self.latest_commit()
	108	+
	109	+ def rev_parse(self, rev):
	110	+ output = self._run_git('rev-parse', rev, stdout=subprocess.PIPE).stdout
	111	+ return output.decode('UTF-8').strip()

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.