[Notes] [Git][BuildStream/buildstream][valentindavid/remove-strip-binaries] 22 commits: tests/sources/git.py: Refactor ref-not-in-track test to use parameterization

[Valentin David <gitlab mg gitlab com>]

Title: GitLab

Valentin David pushed to branch valentindavid/remove-strip-binaries at BuildStream / buildstream

Commits:

• 4688b2c9
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  tests/sources/git.py: Refactor ref-not-in-track test to use parameterization
  
  Instead of duplicating the whole test body, run it twice while
  checking for a warning or an error depending on the parameter.
  

• 03028c7f
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  types.py: Moving CoreWarnings to the types.py module
  
  This also ensures it is exposed via the main buildstream __init__.py
  file, fixes the imports of CoreWarnings, and adjusts the documentation
  links to point to the right place.
  

• 3b7f8df8
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  Source: Fix typo in documentation, and make it a link to `bst track`
  
  Now the commands have documentation linkability so we can link to
  them when referring to them anywhere in the docs. This reference
  was previously referring to a bogus non-existent `build-stream track`,
  maybe this dates way back to the days before we named the frontend `bst`.
  

• e5c0f067
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  git source plugin: Fixing documentation linking and typo
  
  Fixes the configurable warning part of this to properly link
  to the project.conf documentation describing what configurable
  warnings actually are.
  
  Also fix the plugin raise the actual configurable warning which
  it advertizes that it raises; which is 'inconsistent-submodule',
  not 'inconsistent-submodules'.
  

• 205c27d8
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  git source plugin: Rename symbol for consistency and readability
  
  Change 'INCONSISTENT_SUBMODULE' for 'WARN_INCONSISTENT_SUBMODULE' to
  make things a bit more readable and comprehensive.
  

• bea4d4f5
  by Tristan Van Berkom at 2018-12-06T09:37:50Z

  Merge branch 'tristan/misc-cleanup' into 'master'
  
  Misc cleanups
  
  See merge request BuildStream/buildstream!993

• b1d2f001
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  source.py: Add new delegate method validate_cache()
  
  This is guaranteed to be called only once for a given session once the
  sources are known to be Consistency.CACHED, if source tracking is enabled
  in the session for this source, then this will only be called if the
  sources become cached after tracking completes.
  

• ee7fc47f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  git source plugin: Implementing submodule warnings
  
   o Unlisted submodule warning
  
     Now the git plugin will issue a configurable warning if a submodule
     exists and is used (checking out the submodule is not disabled),
     but is not specified in the source configuration.
  
   o Invalid submodule warning
  
     Now the git source plugin will issue a warning if the configuration
     specified a submodule which does not exist in the underlying git repository.
  
  As a side effect, this patch also changes the flow control of the git
  plugin such that submodules which are explicitly set to not be
  checked out, are also not fetched but instead ignored completely.
  

• 5fc9a1da
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  git source plugin: Emmit the ref-not-in-track warning from validate_cache()
  
  Now that we have Source.validate_cache(), this is a better place to emmit
  the ref-not-in-track warning, since it will be emmitted at the earliest
  opportunity and not only at Source.stage() or Source.init_workspace().
  
  This also allows us to remove the `self.tracked` local state, and cleanup
  some convoluted calling paths, removing some unnecessary parameters from
  the usual codepaths and making the plugin overall more readable.
  

• 50cb2706
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/testutils/repo/git.py: Adding remove_path() helper
  
  A function for removing paths from a git repo, can also
  be used to remove submodules.
  

• f139ca8f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Testing the git:unlisted-submodule warning
  
   o Test that it is not triggered in show before fetch, because we
     don't know about the unlisted submodules yet
  
   o Test that it is triggered by a fetch command
  
   o Test that it is triggered by `show` after having completed a
     fetch command, since now we have the repository and know about
     the unlisted submodule
  
   o Test all of this under warning or error conditions (parameterized
     for fatal-warnings)
  

• afe1aa9f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Testing the git:invalid-submodule warning
  
     o Test that it is not triggered in show before fetch, because we
       don't know the submodules yet so we cannot know if they are
       valid or not.
  
     o Test that it is triggered by a fetch command
  
     o Test that it is triggered by `show` after having completed a
       fetch command, since now we have the repository and know which
       specified submodules are invalid
  
     o Test all of this under warning or error conditions (parameterized
       for fatal-warnings)
  

• 57300201
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Test unlisted submodules warning appearing after track
  

• a0297625
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Test invalid submodules warning appearing after track
  

• 60e96781
  by Tristan Van Berkom at 2018-12-06T15:14:37Z

  Merge branch 'tristan/submodule-warnings' into 'master'
  
  Implement submodule warnings
  
  See merge request BuildStream/buildstream!996

• 7fe55143
  by Jonathan Maw at 2018-12-06T15:48:12Z

  Fix bst source-checkout not working with open workspaces
  

• 2a6879a5
  by Jonathan Maw at 2018-12-06T17:46:33Z

  Merge branch 'jonathan/source-checkout-workspace' into 'master'
  
  Fix bst source-checkout not working with open workspaces
  
  Closes #807
  
  See merge request BuildStream/buildstream!997

• e61e6e50
  by Tristan Van Berkom at 2018-12-07T07:57:13Z

  _yaml.py: Added `allow_none` parameter to _yaml.node_get()
  
  This allows specifying whether or not the code believes a None
  value is acceptable if explicitly expressed to be None in the YAML.
  
  A previous patch 3ba544b80f9f268be8ffe62fc8589b30212ec4a2 changed
  the behavior to accept None for all values across the board, along
  with allowing explicitly setting the `default_value` to `None` which
  caused the code to be much more readable (this was the main motivation
  of the patch, but it had the side effect of allowing None for everything).
  
  In the majority of cases we load YAML however, it either has a value or
  it is not specified in the YAML, and None is hardly ever acceptable
  to be explicitly specified (it may be in the case you want to override
  something with nothing in the YAML, but it is the edge case).
  
  This had the side effect of causing unexpected crashes where the
  YAML specifies an empty dictionary for instance.
  
  Instead of forcing the caller to handle a possible None value, give
  the choice to the caller if they really want to accept a None value,
  and raise the LoadError otherwise.
  
  This fixes issue #803, which is a crash due to receival of an
  unexpected None value through _yaml.node_get()
  

• 504ecb3e
  by Tristan Van Berkom at 2018-12-07T07:57:56Z

  plugin.py: Added new `allow_none` parameter to Plugin.node_get_member(), defaulting to False.
  
  Allow plugins to conveniently leverage the new `allow_none` parameter,
  and have Plugin.node_get_member() raise a LoadError in the usual
  case where None is not an acceptable value to be explicitly set in
  the user provided YAML.
  

• 642ae4e8
  by Tristan Van Berkom at 2018-12-07T07:57:56Z

  tests/format/project.py: Added new regression test for empty dependency dictionaries
  
  This adds a regression test for issue #803
  

• 2a0676c3
  by Tristan Van Berkom at 2018-12-07T08:33:14Z

  Merge branch 'tristan/yaml-optionally-allow-none' into 'master'
  
  Only optionally allow None values in user provided YAML
  
  Closes #803
  
  See merge request BuildStream/buildstream!999

• 29ff75c1
  by Valentin David at 2018-12-07T11:23:46Z

  Replace strip-binaries by a NOOP script
  

28 changed files:

• buildstream/__init__.py
• buildstream/_loader/loader.py
• buildstream/_project.py
• buildstream/_stream.py
• buildstream/_versions.py
• buildstream/_yaml.py
• buildstream/data/projectconfig.yaml
• buildstream/element.py
• buildstream/plugin.py
• buildstream/plugins/sources/git.py
• buildstream/source.py
• buildstream/types.py
• doc/source/format_project.rst
• tests/cachekey/project/elements/build1.expected
• tests/cachekey/project/elements/build2.expected
• tests/cachekey/project/target.expected
• tests/examples/autotools.py
• tests/examples/developing.py
• tests/examples/flatpak-autotools.py
• tests/format/project.py
• + tests/format/project/empty-depends/manual.bst
• + tests/format/project/empty-depends/project.conf
• tests/frontend/source_checkout.py
• tests/integration/autotools.py
• tests/integration/cmake.py
• tests/integration/compose.py
• tests/sources/git.py
• tests/testutils/repo/git.py

Changes:

buildstream/__init__.py

---

...	...	@@ -28,7 +28,7 @@ if "_BST_COMPLETION" not in os.environ:
28	28
29	29	from .utils import UtilError, ProgramNotFoundError
30	30	from .sandbox import Sandbox, SandboxFlags, SandboxCommandError
31		- from .types import Scope, Consistency
	31	+ from .types import Scope, Consistency, CoreWarnings
32	32	from .plugin import Plugin
33	33	from .source import Source, SourceError, SourceFetcher
34	34	from .element import Element, ElementError

buildstream/_loader/loader.py

---

...	...	@@ -36,7 +36,7 @@ from .types import Symbol, Dependency
36	36	from .loadelement import LoadElement
37	37	from . import MetaElement
38	38	from . import MetaSource
39		-from ..plugin import CoreWarnings
	39	+from ..types import CoreWarnings
40	40	from .._message import Message, MessageType
41	41
42	42

buildstream/_project.py

---

...	...	@@ -33,7 +33,7 @@ from ._artifactcache import ArtifactCache
33	33	from .sandbox import SandboxRemote
34	34	from ._elementfactory import ElementFactory
35	35	from ._sourcefactory import SourceFactory
36		-from .plugin import CoreWarnings
	36	+from .types import CoreWarnings
37	37	from ._projectrefs import ProjectRefs, ProjectRefStorage
38	38	from ._versions import BST_FORMAT_VERSION
39	39	from ._loader import Loader

buildstream/_stream.py

---

...	...	@@ -1199,7 +1199,7 @@ class Stream():
1199	1199	element_source_dir = self._get_element_dirname(directory, element)
1200	1200	if list(element.sources()):
1201	1201	os.makedirs(element_source_dir)
1202		- element._stage_sources_at(element_source_dir)
	1202	+ element._stage_sources_at(element_source_dir, mount_workspaces=False)
1203	1203
1204	1204	# Write a master build script to the sandbox
1205	1205	def _write_build_script(self, directory, elements):

buildstream/_versions.py

---

...	...	@@ -23,7 +23,7 @@
23	23	# This version is bumped whenever enhancements are made
24	24	# to the `project.conf` format or the core element format.
25	25	#
26		-BST_FORMAT_VERSION = 19
	26	+BST_FORMAT_VERSION = 20
27	27
28	28
29	29	# The base BuildStream artifact version

buildstream/_yaml.py

---

...	...	@@ -352,6 +352,7 @@ _sentinel = object()
352	352	# key (str): The key to get a value for in node
353	353	# indices (list of ints): Optionally decend into lists of lists
354	354	# default_value: Optionally return this value if the key is not found
	355	+# allow_none: (bool): Allow None to be a valid value
355	356	#
356	357	# Returns:
357	358	# The value if found in node, otherwise default_value is returned
...	...	@@ -362,7 +363,7 @@ _sentinel = object()
362	363	# Note:
363	364	# Returned strings are stripped of leading and trailing whitespace
364	365	#
365		-def node_get(node, expected_type, key, indices=None, default_value=_sentinel):
	366	+def node_get(node, expected_type, key, indices=None, *, default_value=_sentinel, allow_none=False):
366	367	value = node.get(key, default_value)
367	368	provenance = node_get_provenance(node)
368	369	if value is _sentinel:
...	...	@@ -377,8 +378,8 @@ def node_get(node, expected_type, key, indices=None, default_value=_sentinel):
377	378	value = value[index]
378	379	path += '[{:d}]'.format(index)
379	380
380		- # We want to allow None as a valid value for any type
381		- if value is None:
	381	+ # Optionally allow None as a valid value for any type
	382	+ if value is None and (allow_none or default_value is None):
382	383	return None
383	384
384	385	if not isinstance(value, expected_type):

buildstream/data/projectconfig.yaml

---

...	...	@@ -54,28 +54,11 @@ variables:
54	54
55	55	--remove-section=.comment --remove-section=.note --strip-unneeded
56	56
57		- # Generic implementation for stripping debugging symbols
	57	+ # Noop implementation of strip-binaries. Because it is not possible
	58	+ # to have portable script, it is the dutie of projects to define it.
58	59	strip-binaries: |
59	60
60		- cd "%{install-root}" && find -type f \
61		- '(' -perm -111 -o -name '*.so*' \
62		- -o -name '*.cmxs' -o -name '*.node' ')' \
63		- -exec sh -ec \
64		- 'read -n4 hdr <"$1" # check for elf header
65		- case "$1" in
66		- %{install-root}%{debugdir}/*)
67		- exit 0
68		- ;;
69		- esac
70		- if [ "$hdr" != "$(printf \\x7fELF)" ]; then
71		- exit 0
72		- fi
73		- debugfile="%{install-root}%{debugdir}/$1"
74		- mkdir -p "$(dirname "$debugfile")"
75		- objcopy %{objcopy-extract-args} "$1" "$debugfile"
76		- chmod 644 "$debugfile"
77		- strip %{strip-args} "$1"
78		- objcopy %{objcopy-link-args} "$debugfile" "$1"' - {} ';'
	61	+ true
79	62
80	63	# Generic implementation for reproducible python builds
81	64	fix-pyc-timestamps: |

buildstream/element.py

---

...	...	@@ -96,10 +96,9 @@ from . import _cachekey
96	96	from . import _signals
97	97	from . import _site
98	98	from ._platform import Platform
99		-from .plugin import CoreWarnings
100	99	from .sandbox._config import SandboxConfig
101	100	from .sandbox._sandboxremote import SandboxRemote
102		-from .types import _KeyStrength
	101	+from .types import _KeyStrength, CoreWarnings
103	102
104	103	from .storage.directory import Directory
105	104	from .storage._filebaseddirectory import FileBasedDirectory

buildstream/plugin.py

---

...	...	@@ -119,6 +119,7 @@ from . import _yaml
119	119	from . import utils
120	120	from ._exceptions import PluginError, ImplError
121	121	from ._message import Message, MessageType
	122	+from .types import CoreWarnings
122	123
123	124
124	125	class Plugin():
...	...	@@ -322,7 +323,7 @@ class Plugin():
322	323	provenance = _yaml.node_get_provenance(node, key=member_name)
323	324	return str(provenance)
324	325
325		- def node_get_member(self, node, expected_type, member_name, default=_yaml._sentinel):
	326	+ def node_get_member(self, node, expected_type, member_name, default=_yaml._sentinel, *, allow_none=False):
326	327	"""Fetch the value of a node member, raising an error if the value is
327	328	missing or incorrectly typed.
328	329
...	...	@@ -331,6 +332,7 @@ class Plugin():
331	332	expected_type (type): The expected type of the node member
332	333	member_name (str): The name of the member to fetch
333	334	default (expected_type): A value to return when *member_name* is not specified in *node*
	335	+ allow_none (bool): Allow explicitly set None values in the YAML (*Since: 1.4*)
334	336
335	337	Returns:
336	338	The value of *member_name* in *node*, otherwise *default*
...	...	@@ -351,7 +353,7 @@ class Plugin():
351	353	# Fetch an optional integer
352	354	level = self.node_get_member(node, int, 'level', -1)
353	355	"""
354		- return _yaml.node_get(node, expected_type, member_name, default_value=default)
	356	+ return _yaml.node_get(node, expected_type, member_name, default_value=default, allow_none=allow_none)
355	357
356	358	def node_get_project_path(self, node, key, *,
357	359	check_is_file=False, check_is_dir=False):
...	...	@@ -766,38 +768,6 @@ class Plugin():
766	768	return self.name
767	769
768	770
769		-class CoreWarnings():
770		- """CoreWarnings()
771		-
772		- Some common warnings which are raised by core functionalities within BuildStream are found in this class.
773		- """
774		-
775		- OVERLAPS = "overlaps"
776		- """
777		- This warning will be produced when buildstream detects an overlap on an element
778		- which is not whitelisted. See :ref:`Overlap Whitelist <public_overlap_whitelist>`
779		- """
780		-
781		- REF_NOT_IN_TRACK = "ref-not-in-track"
782		- """
783		- This warning will be produced when a source is configured with a reference
784		- which is found to be invalid based on the configured track
785		- """
786		-
787		- BAD_ELEMENT_SUFFIX = "bad-element-suffix"
788		- """
789		- This warning will be produced when an element whose name does not end in .bst
790		- is referenced either on the command line or by another element
791		- """
792		-
793		-
794		-__CORE_WARNINGS = [
795		- value
796		- for name, value in CoreWarnings.__dict__.items()
797		- if not name.startswith("__")
798		-]
799		-
800		-
801	771	# Hold on to a lookup table by counter of all instantiated plugins.
802	772	# We use this to send the id back from child processes so we can lookup
803	773	# corresponding element/source in the master process.
...	...	@@ -828,6 +798,24 @@ def _plugin_lookup(unique_id):
828	798	return __PLUGINS_TABLE[unique_id]
829	799
830	800
	801	+# No need for unregister, WeakValueDictionary() will remove entries
	802	+# in itself when the referenced plugins are garbage collected.
	803	+def _plugin_register(plugin):
	804	+ global __PLUGINS_UNIQUE_ID # pylint: disable=global-statement
	805	+ __PLUGINS_UNIQUE_ID += 1
	806	+ __PLUGINS_TABLE[__PLUGINS_UNIQUE_ID] = plugin
	807	+ return __PLUGINS_UNIQUE_ID
	808	+
	809	+
	810	+# A local table for _prefix_warning()
	811	+#
	812	+__CORE_WARNINGS = [
	813	+ value
	814	+ for name, value in CoreWarnings.__dict__.items()
	815	+ if not name.startswith("__")
	816	+]
	817	+
	818	+
831	819	# _prefix_warning():
832	820	#
833	821	# Prefix a warning with the plugin kind. CoreWarnings are not prefixed.
...	...	@@ -843,12 +831,3 @@ def _prefix_warning(plugin, warning):
843	831	if any((warning is core_warning for core_warning in __CORE_WARNINGS)):
844	832	return warning
845	833	return "{}:{}".format(plugin.get_kind(), warning)
846		-
847		-
848		-# No need for unregister, WeakValueDictionary() will remove entries
849		-# in itself when the referenced plugins are garbage collected.
850		-def _plugin_register(plugin):
851		- global __PLUGINS_UNIQUE_ID # pylint: disable=global-statement
852		- __PLUGINS_UNIQUE_ID += 1
853		- __PLUGINS_TABLE[__PLUGINS_UNIQUE_ID] = plugin
854		- return __PLUGINS_UNIQUE_ID

buildstream/plugins/sources/git.py

---

...	...	@@ -131,13 +131,29 @@ details on common configuration options for sources.
131	131
132	132	**Configurable Warnings:**
133	133
134		-This plugin provides the following configurable warnings:
	134	+This plugin provides the following :ref:`configurable warnings <configurable_warnings>`:
135	135
136		-- 'git:inconsistent-submodule' - A submodule was found to be missing from the underlying git repository.
	136	+- ``git:inconsistent-submodule`` - A submodule present in the git repository's .gitmodules was never
	137	+ added with `git submodule add`.
137	138
138		-This plugin also utilises the following configurable core plugin warnings:
	139	+- ``git:unlisted-submodule`` - A submodule is present in the git repository but was not specified in
	140	+ the source configuration and was not disabled for checkout.
139	141
140		-- 'ref-not-in-track' - The provided ref was not found in the provided track in the element's git repository.
	142	+ .. note::
	143	+
	144	+ The ``git:unlisted-submodule`` warning is available since :ref:`format version 20 <project_format_version>`
	145	+
	146	+- ``git:invalid-submodule`` - A submodule is specified in the source configuration but does not exist
	147	+ in the repository.
	148	+
	149	+ .. note::
	150	+
	151	+ The ``git:invalid-submodule`` warning is available since :ref:`format version 20 <project_format_version>`
	152	+
	153	+This plugin also utilises the following configurable :class:`core warnings <buildstream.types.CoreWarnings>`:
	154	+
	155	+- :attr:`ref-not-in-track <buildstream.types.CoreWarnings.REF_NOT_IN_TRACK>` - The provided ref was not
	156	+ found in the provided track in the element's git repository.
141	157	"""
142	158
143	159	import os
...	...	@@ -149,15 +165,16 @@ from tempfile import TemporaryFile
149	165
150	166	from configparser import RawConfigParser
151	167
152		-from buildstream import Source, SourceError, Consistency, SourceFetcher
	168	+from buildstream import Source, SourceError, Consistency, SourceFetcher, CoreWarnings
153	169	from buildstream import utils
154		-from buildstream.plugin import CoreWarnings
155	170	from buildstream.utils import move_atomic, DirectoryExistsError
156	171
157	172	GIT_MODULES = '.gitmodules'
158	173
159	174	# Warnings
160		-INCONSISTENT_SUBMODULE = "inconsistent-submodules"
	175	+WARN_INCONSISTENT_SUBMODULE = "inconsistent-submodule"
	176	+WARN_UNLISTED_SUBMODULE = "unlisted-submodule"
	177	+WARN_INVALID_SUBMODULE = "invalid-submodule"
161	178
162	179
163	180	# Because of handling of submodules, we maintain a GitMirror
...	...	@@ -305,7 +322,7 @@ class GitMirror(SourceFetcher):
305	322
306	323	return ref, list(tags)
307	324
308		- def stage(self, directory, track=None):
	325	+ def stage(self, directory):
309	326	fullpath = os.path.join(directory, self.path)
310	327
311	328	# Using --shared here avoids copying the objects into the checkout, in any
...	...	@@ -324,11 +341,7 @@ class GitMirror(SourceFetcher):
324	341
325	342	self._rebuild_git(fullpath)
326	343
327		- # Check that the user specified ref exists in the track if provided & not already tracked
328		- if track:
329		- self.assert_ref_in_track(fullpath, track)
330		-
331		- def init_workspace(self, directory, track=None):
	344	+ def init_workspace(self, directory):
332	345	fullpath = os.path.join(directory, self.path)
333	346	url = self.source.translate_url(self.url)
334	347
...	...	@@ -344,10 +357,6 @@ class GitMirror(SourceFetcher):
344	357	fail="Failed to checkout git ref {}".format(self.ref),
345	358	cwd=fullpath)
346	359
347		- # Check that the user specified ref exists in the track if provided & not already tracked
348		- if track:
349		- self.assert_ref_in_track(fullpath, track)
350		-
351	360	# List the submodules (path/url tuples) present at the given ref of this repo
352	361	def submodule_list(self):
353	362	modules = "{}:{}".format(self.ref, GIT_MODULES)
...	...	@@ -408,32 +417,11 @@ class GitMirror(SourceFetcher):
408	417	"underlying git repository with `git submodule add`."
409	418
410	419	self.source.warn("{}: Ignoring inconsistent submodule '{}'"
411		- .format(self.source, submodule), detail=detail, warning_token=INCONSISTENT_SUBMODULE)
	420	+ .format(self.source, submodule), detail=detail,
	421	+ warning_token=WARN_INCONSISTENT_SUBMODULE)
412	422
413	423	return None
414	424
415		- # Assert that ref exists in track, if track has been specified.
416		- def assert_ref_in_track(self, fullpath, track):
417		- _, branch = self.source.check_output([self.source.host_git, 'branch', '--list', track,
418		- '--contains', self.ref],
419		- cwd=fullpath,)
420		- if branch:
421		- return
422		- else:
423		- _, tag = self.source.check_output([self.source.host_git, 'tag', '--list', track,
424		- '--contains', self.ref],
425		- cwd=fullpath,)
426		- if tag:
427		- return
428		-
429		- detail = "The ref provided for the element does not exist locally in the provided track branch / tag " + \
430		- "'{}'.\nYou may wish to track the element to update the ref from '{}' ".format(track, track) + \
431		- "with `bst track`,\nor examine the upstream at '{}' for the specific ref.".format(self.url)
432		-
433		- self.source.warn("{}: expected ref '{}' was not found in given track '{}' for staged repository: '{}'\n"
434		- .format(self.source, self.ref, track, self.url),
435		- detail=detail, warning_token=CoreWarnings.REF_NOT_IN_TRACK)
436		-
437	425	def _rebuild_git(self, fullpath):
438	426	if not self.tags:
439	427	return
...	...	@@ -562,7 +550,6 @@ class GitSource(Source):
562	550	self.submodule_checkout_overrides[path] = checkout
563	551
564	552	self.mark_download_url(self.original_url)
565		- self.tracked = False
566	553
567	554	def preflight(self):
568	555	# Check if git is installed, get the binary at the same time
...	...	@@ -652,8 +639,6 @@ class GitSource(Source):
652	639	# Update self.mirror.ref and node.ref from the self.tracking branch
653	640	ret = self.mirror.latest_commit_with_tags(self.tracking, self.track_tags)
654	641
655		- # Set tracked attribute, parameter for if self.mirror.assert_ref_in_track is needed
656		- self.tracked = True
657	642	return ret
658	643
659	644	def init_workspace(self, directory):
...	...	@@ -661,7 +646,7 @@ class GitSource(Source):
661	646	self.refresh_submodules()
662	647
663	648	with self.timed_activity('Setting up workspace "{}"'.format(directory), silent_nested=True):
664		- self.mirror.init_workspace(directory, track=(self.tracking if not self.tracked else None))
	649	+ self.mirror.init_workspace(directory)
665	650	for mirror in self.submodules:
666	651	mirror.init_workspace(directory)
667	652
...	...	@@ -677,15 +662,9 @@ class GitSource(Source):
677	662	# Stage the main repo in the specified directory
678	663	#
679	664	with self.timed_activity("Staging {}".format(self.mirror.url), silent_nested=True):
680		- self.mirror.stage(directory, track=(self.tracking if not self.tracked else None))
	665	+ self.mirror.stage(directory)
681	666	for mirror in self.submodules:
682		- if mirror.path in self.submodule_checkout_overrides:
683		- checkout = self.submodule_checkout_overrides[mirror.path]
684		- else:
685		- checkout = self.checkout_submodules
686		-
687		- if checkout:
688		- mirror.stage(directory)
	667	+ mirror.stage(directory)
689	668
690	669	def get_source_fetchers(self):
691	670	yield self.mirror
...	...	@@ -693,6 +672,74 @@ class GitSource(Source):
693	672	for submodule in self.submodules:
694	673	yield submodule
695	674
	675	+ def validate_cache(self):
	676	+ discovered_submodules = {}
	677	+ unlisted_submodules = []
	678	+ invalid_submodules = []
	679	+
	680	+ for path, url in self.mirror.submodule_list():
	681	+ discovered_submodules[path] = url
	682	+ if self.ignore_submodule(path):
	683	+ continue
	684	+
	685	+ override_url = self.submodule_overrides.get(path)
	686	+ if not override_url:
	687	+ unlisted_submodules.append((path, url))
	688	+
	689	+ # Warn about submodules which are explicitly configured but do not exist
	690	+ for path, url in self.submodule_overrides.items():
	691	+ if path not in discovered_submodules:
	692	+ invalid_submodules.append((path, url))
	693	+
	694	+ if invalid_submodules:
	695	+ detail = []
	696	+ for path, url in invalid_submodules:
	697	+ detail.append(" Submodule URL '{}' at path '{}'".format(url, path))
	698	+
	699	+ self.warn("{}: Invalid submodules specified".format(self),
	700	+ warning_token=WARN_INVALID_SUBMODULE,
	701	+ detail="The following submodules are specified in the source "
	702	+ "description but do not exist according to the repository\n\n" +
	703	+ "\n".join(detail))
	704	+
	705	+ # Warn about submodules which exist but have not been explicitly configured
	706	+ if unlisted_submodules:
	707	+ detail = []
	708	+ for path, url in unlisted_submodules:
	709	+ detail.append(" Submodule URL '{}' at path '{}'".format(url, path))
	710	+
	711	+ self.warn("{}: Unlisted submodules exist".format(self),
	712	+ warning_token=WARN_UNLISTED_SUBMODULE,
	713	+ detail="The following submodules exist but are not specified " +
	714	+ "in the source description\n\n" +
	715	+ "\n".join(detail))
	716	+
	717	+ # Assert that the ref exists in the track tag/branch, if track has been specified.
	718	+ ref_in_track = False
	719	+ if self.tracking:
	720	+ _, branch = self.check_output([self.host_git, 'branch', '--list', self.tracking,
	721	+ '--contains', self.mirror.ref],
	722	+ cwd=self.mirror.mirror)
	723	+ if branch:
	724	+ ref_in_track = True
	725	+ else:
	726	+ _, tag = self.check_output([self.host_git, 'tag', '--list', self.tracking,
	727	+ '--contains', self.mirror.ref],
	728	+ cwd=self.mirror.mirror)
	729	+ if tag:
	730	+ ref_in_track = True
	731	+
	732	+ if not ref_in_track:
	733	+ detail = "The ref provided for the element does not exist locally " + \
	734	+ "in the provided track branch / tag '{}'.\n".format(self.tracking) + \
	735	+ "You may wish to track the element to update the ref from '{}' ".format(self.tracking) + \
	736	+ "with `bst track`,\n" + \
	737	+ "or examine the upstream at '{}' for the specific ref.".format(self.mirror.url)
	738	+
	739	+ self.warn("{}: expected ref '{}' was not found in given track '{}' for staged repository: '{}'\n"
	740	+ .format(self, self.mirror.ref, self.tracking, self.mirror.url),
	741	+ detail=detail, warning_token=CoreWarnings.REF_NOT_IN_TRACK)
	742	+
696	743	###########################################################
697	744	# Local Functions #
698	745	###########################################################
...	...	@@ -717,12 +764,12 @@ class GitSource(Source):
717	764	self.mirror.ensure()
718	765	submodules = []
719	766
720		- # XXX Here we should issue a warning if either:
721		- # A.) A submodule exists but is not defined in the element configuration
722		- # B.) The element configuration configures submodules which dont exist at the current ref
723		- #
724	767	for path, url in self.mirror.submodule_list():
725	768
	769	+ # Completely ignore submodules which are disabled for checkout
	770	+ if self.ignore_submodule(path):
	771	+ continue
	772	+
726	773	# Allow configuration to override the upstream
727	774	# location of the submodules.
728	775	override_url = self.submodule_overrides.get(path)
...	...	@@ -746,6 +793,16 @@ class GitSource(Source):
746	793	tags.append((tag, commit_ref, annotated))
747	794	return tags
748	795
	796	+ # Checks whether the plugin configuration has explicitly
	797	+ # configured this submodule to be ignored
	798	+ def ignore_submodule(self, path):
	799	+ try:
	800	+ checkout = self.submodule_checkout_overrides[path]
	801	+ except KeyError:
	802	+ checkout = self.checkout_submodules
	803	+
	804	+ return not checkout
	805	+
749	806
750	807	# Plugin entry point
751	808	def setup():

buildstream/source.py

---

...	...	@@ -102,6 +102,11 @@ these methods are mandatory to implement.
102	102	submodules). For details on how to define a SourceFetcher, see
103	103	:ref:`SourceFetcher <core_source_fetcher>`.
104	104
	105	+* :func:`Source.validate_cache() <buildstream.source.Source.validate_cache>`
	106	+
	107	+ Perform any validations which require the sources to be cached.
	108	+
	109	+ **Optional**: This is completely optional and will do nothing if left unimplemented.
105	110
106	111	Accessing previous sources
107	112	--------------------------
...	...	@@ -391,7 +396,8 @@ class Source(Plugin):
391	396
392	397	If the backend in question supports resolving references from
393	398	a symbolic tracking branch or tag, then this should be implemented
394		- to perform this task on behalf of ``build-stream track`` commands.
	399	+ to perform this task on behalf of :ref:`bst track <invoking_track>`
	400	+ commands.
395	401
396	402	This usually requires fetching new content from a remote origin
397	403	to see if a new ref has appeared for your branch or tag. If the
...	...	@@ -479,9 +485,22 @@ class Source(Plugin):
479	485
480	486	*Since: 1.2*
481	487	"""
482		-
483	488	return []
484	489
	490	+ def validate_cache(self):
	491	+ """Implement any validations once we know the sources are cached
	492	+
	493	+ This is guaranteed to be called only once for a given session
	494	+ once the sources are known to be
	495	+ :attr:`Consistency.CACHED <buildstream.types.Consistency.CACHED>`,
	496	+ if source tracking is enabled in the session for this source,
	497	+ then this will only be called if the sources become cached after
	498	+ tracking completes.
	499	+
	500	+ *Since: 1.4*
	501	+ """
	502	+ pass
	503	+
485	504	#############################################################
486	505	# Public Methods #
487	506	#############################################################
...	...	@@ -658,6 +677,11 @@ class Source(Plugin):
658	677	with context.silence():
659	678	self.__consistency = self.get_consistency() # pylint: disable=assignment-from-no-return
660	679
	680	+ # Give the Source an opportunity to validate the cached
	681	+ # sources as soon as the Source becomes Consistency.CACHED.
	682	+ if self.__consistency == Consistency.CACHED:
	683	+ self.validate_cache()
	684	+
661	685	# Return cached consistency
662	686	#
663	687	def _get_consistency(self):

buildstream/types.py

---

...	...	@@ -81,6 +81,31 @@ class Consistency():
81	81	"""
82	82
83	83
	84	+class CoreWarnings():
	85	+ """CoreWarnings()
	86	+
	87	+ Some common warnings which are raised by core functionalities within BuildStream are found in this class.
	88	+ """
	89	+
	90	+ OVERLAPS = "overlaps"
	91	+ """
	92	+ This warning will be produced when buildstream detects an overlap on an element
	93	+ which is not whitelisted. See :ref:`Overlap Whitelist <public_overlap_whitelist>`
	94	+ """
	95	+
	96	+ REF_NOT_IN_TRACK = "ref-not-in-track"
	97	+ """
	98	+ This warning will be produced when a source is configured with a reference
	99	+ which is found to be invalid based on the configured track
	100	+ """
	101	+
	102	+ BAD_ELEMENT_SUFFIX = "bad-element-suffix"
	103	+ """
	104	+ This warning will be produced when an element whose name does not end in .bst
	105	+ is referenced either on the command line or by another element
	106	+ """
	107	+
	108	+
84	109	# _KeyStrength():
85	110	#
86	111	# Strength of cache key

doc/source/format_project.rst

---

...	...	@@ -143,7 +143,7 @@ Individual warnings can be configured as fatal by setting ``fatal-warnings`` to
143	143	- ref-not-in-track
144	144	- <plugin>:<warning>
145	145
146		-BuildStream provides a collection of :class:`Core Warnings <buildstream.plugin.CoreWarnings>` which may be raised
	146	+BuildStream provides a collection of :class:`Core Warnings <buildstream.types.CoreWarnings>` which may be raised
147	147	by a variety of plugins. Other configurable warnings are plugin specific and should be noted within their individual documentation.
148	148
149	149	.. note::

tests/cachekey/project/elements/build1.expected

---

1		-a0d000abc1dea8714cd27f348d0b798b35e7246c44e330c4b3f7912fabacc6db
		\ No newline at end of file
	1	+883246ee07fd0a742bff5178f3385c61829cb386a90ec8b5699c64f8c4953d7c
		\ No newline at end of file

tests/cachekey/project/elements/build2.expected

---

1		-79f546a78748d943a6958c99ab4ad03305f96fefd0b424b6b246b0c9816e00c6
		\ No newline at end of file
	1	+cc470b6341200c663faece2b526abe9c8ce229e7af25bb94608f09068332313a
		\ No newline at end of file

tests/cachekey/project/target.expected

---

1		-d6d283ed1fb0467fcfa5bf69f8596d0f0ac6638281bc9d8e52e1212e2ec0bcab
		\ No newline at end of file
	1	+3a4ca16f9250abb0a23277e41f23802116b504b13ccbcdf172cc426fa979a6db
		\ No newline at end of file

tests/examples/autotools.py

---

...	...	@@ -29,10 +29,7 @@ def test_autotools_build(cli, tmpdir, datafiles):
29	29	result.assert_success()
30	30
31	31	assert_contains(checkout, ['/usr', '/usr/lib', '/usr/bin',
32		- '/usr/share', '/usr/lib/debug',
33		- '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
34		- '/usr/lib/debug/usr/bin/hello',
35		- '/usr/bin/hello',
	32	+ '/usr/share', '/usr/bin/hello',
36	33	'/usr/share/doc', '/usr/share/doc/amhello',
37	34	'/usr/share/doc/amhello/README'])
38	35

tests/examples/developing.py

---

...	...	@@ -30,10 +30,7 @@ def test_autotools_build(cli, tmpdir, datafiles):
30	30	result.assert_success()
31	31
32	32	assert_contains(checkout, ['/usr', '/usr/lib', '/usr/bin',
33		- '/usr/share', '/usr/lib/debug',
34		- '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
35		- '/usr/lib/debug/usr/bin/hello',
36		- '/usr/bin/hello'])
	33	+ '/usr/share', '/usr/bin/hello'])
37	34
38	35
39	36	# Test the unmodified hello command works as expected.

tests/examples/flatpak-autotools.py

---

...	...	@@ -48,10 +48,7 @@ def test_autotools_build(cli, tmpdir, datafiles):
48	48	assert result.exit_code == 0
49	49
50	50	assert_contains(checkout, ['/usr', '/usr/lib', '/usr/bin',
51		- '/usr/share', '/usr/lib/debug',
52		- '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
53		- '/usr/lib/debug/usr/bin/hello',
54		- '/usr/bin/hello', '/usr/share/doc',
	51	+ '/usr/share', '/usr/bin/hello', '/usr/share/doc',
55	52	'/usr/share/doc/amhello',
56	53	'/usr/share/doc/amhello/README'])
57	54

tests/format/project.py

---

...	...	@@ -200,3 +200,10 @@ def test_element_path_project_path_contains_symlinks(cli, datafiles, tmpdir):
200	200	f.write("kind: manual\n")
201	201	result = cli.run(project=linked_project, args=['show', 'element.bst'])
202	202	result.assert_success()
	203	+
	204	+
	205	+@pytest.mark.datafiles(os.path.join(DATA_DIR))
	206	+def test_empty_depends(cli, datafiles):
	207	+ project = os.path.join(datafiles.dirname, datafiles.basename, "empty-depends")
	208	+ result = cli.run(project=project, args=['show', 'manual.bst'])
	209	+ result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA)

tests/format/project/empty-depends/manual.bst

---

	1	+kind: manual
	2	+
	3	+depends:

tests/format/project/empty-depends/project.conf

---

1

+name: test

tests/frontend/source_checkout.py

---

...	...	@@ -28,10 +28,17 @@ def generate_remote_import_element(input_path, output_path):
28	28
29	29
30	30	@pytest.mark.datafiles(DATA_DIR)
31		-def test_source_checkout(datafiles, cli):
	31	+@pytest.mark.parametrize('with_workspace', [('workspace'), ('no-workspace')])
	32	+def test_source_checkout(datafiles, tmpdir_factory, cli, with_workspace):
	33	+ tmpdir = tmpdir_factory.mktemp("")
32	34	project = os.path.join(datafiles.dirname, datafiles.basename)
33	35	checkout = os.path.join(cli.directory, 'source-checkout')
34	36	target = 'checkout-deps.bst'
	37	+ workspace = os.path.join(str(tmpdir), 'workspace')
	38	+
	39	+ if with_workspace == "workspace":
	40	+ result = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, target])
	41	+ result.assert_success()
35	42
36	43	result = cli.run(project=project, args=['source-checkout', target, '--deps', 'none', checkout])
37	44	result.assert_success()

tests/integration/autotools.py

---

...	...	@@ -32,9 +32,7 @@ def test_autotools_build(cli, tmpdir, datafiles):
32	32	assert result.exit_code == 0
33	33
34	34	assert_contains(checkout, ['/usr', '/usr/lib', '/usr/bin',
35		- '/usr/share', '/usr/lib/debug',
36		- '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
37		- '/usr/lib/debug/usr/bin/hello',
	35	+ '/usr/share',
38	36	'/usr/bin/hello', '/usr/share/doc',
39	37	'/usr/share/doc/amhello',
40	38	'/usr/share/doc/amhello/README'])
...	...	@@ -57,9 +55,7 @@ def test_autotools_confroot_build(cli, tmpdir, datafiles):
57	55	assert result.exit_code == 0
58	56
59	57	assert_contains(checkout, ['/usr', '/usr/lib', '/usr/bin',
60		- '/usr/share', '/usr/lib/debug',
61		- '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
62		- '/usr/lib/debug/usr/bin/hello',
	58	+ '/usr/share',
63	59	'/usr/bin/hello', '/usr/share/doc',
64	60	'/usr/share/doc/amhello',
65	61	'/usr/share/doc/amhello/README'])

tests/integration/cmake.py

---

...	...	@@ -28,10 +28,7 @@ def test_cmake_build(cli, tmpdir, datafiles):
28	28	result = cli.run(project=project, args=['checkout', element_name, checkout])
29	29	assert result.exit_code == 0
30	30
31		- assert_contains(checkout, ['/usr', '/usr/bin', '/usr/bin/hello',
32		- '/usr/lib/debug', '/usr/lib/debug/usr',
33		- '/usr/lib/debug/usr/bin',
34		- '/usr/lib/debug/usr/bin/hello'])
	31	+ assert_contains(checkout, ['/usr', '/usr/bin', '/usr/bin/hello'])
35	32
36	33
37	34	@pytest.mark.datafiles(DATA_DIR)
...	...	@@ -47,10 +44,7 @@ def test_cmake_confroot_build(cli, tmpdir, datafiles):
47	44	result = cli.run(project=project, args=['checkout', element_name, checkout])
48	45	assert result.exit_code == 0
49	46
50		- assert_contains(checkout, ['/usr', '/usr/bin', '/usr/bin/hello',
51		- '/usr/lib/debug', '/usr/lib/debug/usr',
52		- '/usr/lib/debug/usr/bin',
53		- '/usr/lib/debug/usr/bin/hello'])
	47	+ assert_contains(checkout, ['/usr', '/usr/bin', '/usr/bin/hello'])
54	48
55	49
56	50	@pytest.mark.datafiles(DATA_DIR)

tests/integration/compose.py

---

...	...	@@ -38,48 +38,38 @@ def create_compose_element(name, path, config={}):
38	38	@pytest.mark.datafiles(DATA_DIR)
39	39	@pytest.mark.parametrize("include_domains,exclude_domains,expected", [
40	40	# Test flat inclusion
41		- ([], [], ['/usr', '/usr/lib', '/usr/bin',
42		- '/usr/share', '/usr/lib/debug',
43		- '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
44		- '/usr/lib/debug/usr/bin/hello', '/usr/bin/hello',
	41	+ ([], [], ['/usr', '/usr/bin',
	42	+ '/usr/share', '/usr/bin/hello',
45	43	'/usr/share/doc', '/usr/share/doc/amhello',
46	44	'/usr/share/doc/amhello/README',
47	45	'/tests', '/tests/test']),
48	46	# Test only runtime
49		- (['runtime'], [], ['/usr', '/usr/lib', '/usr/share',
	47	+ (['runtime'], [], ['/usr', '/usr/share',
50	48	'/usr/bin', '/usr/bin/hello']),
51	49	# Test with runtime and doc
52		- (['runtime', 'doc'], [], ['/usr', '/usr/lib', '/usr/share',
	50	+ (['runtime', 'doc'], [], ['/usr', '/usr/share',
53	51	'/usr/bin', '/usr/bin/hello',
54	52	'/usr/share/doc', '/usr/share/doc/amhello',
55	53	'/usr/share/doc/amhello/README']),
56	54	# Test with only runtime excluded
57		- ([], ['runtime'], ['/usr', '/usr/lib', '/usr/share',
58		- '/usr/lib/debug', '/usr/lib/debug/usr',
59		- '/usr/lib/debug/usr/bin',
60		- '/usr/lib/debug/usr/bin/hello',
	55	+ ([], ['runtime'], ['/usr', '/usr/share',
61	56	'/usr/share/doc', '/usr/share/doc/amhello',
62	57	'/usr/share/doc/amhello/README',
63	58	'/tests', '/tests/test']),
64	59	# Test with runtime and doc excluded
65		- ([], ['runtime', 'doc'], ['/usr', '/usr/lib', '/usr/share',
66		- '/usr/lib/debug', '/usr/lib/debug/usr',
67		- '/usr/lib/debug/usr/bin',
68		- '/usr/lib/debug/usr/bin/hello',
	60	+ ([], ['runtime', 'doc'], ['/usr', '/usr/share',
69	61	'/tests', '/tests/test']),
70	62	# Test with runtime simultaneously in- and excluded
71		- (['runtime'], ['runtime'], ['/usr', '/usr/lib', '/usr/share']),
	63	+ (['runtime'], ['runtime'], ['/usr', '/usr/share']),
72	64	# Test with runtime included and doc excluded
73		- (['runtime'], ['doc'], ['/usr', '/usr/lib', '/usr/share',
	65	+ (['runtime'], ['doc'], ['/usr', '/usr/share',
74	66	'/usr/bin', '/usr/bin/hello']),
75	67	# Test including a custom 'test' domain
76		- (['test'], [], ['/usr', '/usr/lib', '/usr/share',
	68	+ (['test'], [], ['/usr', '/usr/share',
77	69	'/tests', '/tests/test']),
78	70	# Test excluding a custom 'test' domain
79		- ([], ['test'], ['/usr', '/usr/lib', '/usr/bin',
80		- '/usr/share', '/usr/lib/debug',
81		- '/usr/lib/debug/usr', '/usr/lib/debug/usr/bin',
82		- '/usr/lib/debug/usr/bin/hello', '/usr/bin/hello',
	71	+ ([], ['test'], ['/usr', '/usr/bin',
	72	+ '/usr/share', '/usr/bin/hello',
83	73	'/usr/share/doc', '/usr/share/doc/amhello',
84	74	'/usr/share/doc/amhello/README'])
85	75	])

tests/sources/git.py

---

...	...	@@ -414,9 +414,18 @@ def test_submodule_track_no_ref_or_track(cli, tmpdir, datafiles):
414	414
415	415	@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
416	416	@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
417		-def test_ref_not_in_track_warn(cli, tmpdir, datafiles):
	417	+@pytest.mark.parametrize("fail", ['warn', 'error'])
	418	+def test_ref_not_in_track(cli, tmpdir, datafiles, fail):
418	419	project = os.path.join(datafiles.dirname, datafiles.basename)
419	420
	421	+ # Make the warning an error if we're testing errors
	422	+ if fail == 'error':
	423	+ project_template = {
	424	+ "name": "foo",
	425	+ "fatal-warnings": [CoreWarnings.REF_NOT_IN_TRACK]
	426	+ }
	427	+ _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	428	+
420	429	# Create the repo from 'repofiles', create a branch without latest commit
421	430	repo = create_repo('git', str(tmpdir))
422	431	ref = repo.create(os.path.join(project, 'repofiles'))
...	...	@@ -435,33 +444,184 @@ def test_ref_not_in_track_warn(cli, tmpdir, datafiles):
435	444	}
436	445	_yaml.dump(element, os.path.join(project, 'target.bst'))
437	446
438		- # Assert the warning is raised as ref is not in branch foo.
439		- # Assert warning not error to the user, when not set as fatal.
440	447	result = cli.run(project=project, args=['build', 'target.bst'])
441		- assert "The ref provided for the element does not exist locally" in result.stderr
	448	+
	449	+ # Assert a warning or an error depending on what we're checking
	450	+ if fail == 'error':
	451	+ result.assert_main_error(ErrorDomain.STREAM, None)
	452	+ result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.REF_NOT_IN_TRACK)
	453	+ else:
	454	+ result.assert_success()
	455	+ assert "ref-not-in-track" in result.stderr
442	456
443	457
444	458	@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
445	459	@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
446		-def test_ref_not_in_track_warn_error(cli, tmpdir, datafiles):
	460	+@pytest.mark.parametrize("fail", ['warn', 'error'])
	461	+def test_unlisted_submodule(cli, tmpdir, datafiles, fail):
447	462	project = os.path.join(datafiles.dirname, datafiles.basename)
448	463
449		- # Add fatal-warnings ref-not-in-track to project.conf
450		- project_template = {
451		- "name": "foo",
452		- "fatal-warnings": [CoreWarnings.REF_NOT_IN_TRACK]
	464	+ # Make the warning an error if we're testing errors
	465	+ if fail == 'error':
	466	+ project_template = {
	467	+ "name": "foo",
	468	+ "fatal-warnings": ['git:unlisted-submodule']
	469	+ }
	470	+ _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	471	+
	472	+ # Create the submodule first from the 'subrepofiles' subdir
	473	+ subrepo = create_repo('git', str(tmpdir), 'subrepo')
	474	+ subrepo.create(os.path.join(project, 'subrepofiles'))
	475	+
	476	+ # Create the repo from 'repofiles' subdir
	477	+ repo = create_repo('git', str(tmpdir))
	478	+ ref = repo.create(os.path.join(project, 'repofiles'))
	479	+
	480	+ # Add a submodule pointing to the one we created
	481	+ ref = repo.add_submodule('subdir', 'file://' + subrepo.repo)
	482	+
	483	+ # Create the source, and delete the explicit configuration
	484	+ # of the submodules.
	485	+ #
	486	+ # We expect this to cause an unlisted submodule warning
	487	+ # after the source has been fetched.
	488	+ #
	489	+ gitsource = repo.source_config(ref=ref)
	490	+ del gitsource['submodules']
	491	+
	492	+ # Write out our test target
	493	+ element = {
	494	+ 'kind': 'import',
	495	+ 'sources': [
	496	+ gitsource
	497	+ ]
453	498	}
	499	+ _yaml.dump(element, os.path.join(project, 'target.bst'))
454	500
455		- _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	501	+ # We will not see the warning or error before the first fetch, because
	502	+ # we don't have the repository yet and so we have no knowledge of
	503	+ # the unlisted submodule.
	504	+ result = cli.run(project=project, args=['show', 'target.bst'])
	505	+ result.assert_success()
	506	+ assert "git:unlisted-submodule" not in result.stderr
456	507
457		- # Create the repo from 'repofiles', create a branch without latest commit
	508	+ # We will notice this directly in fetch, as it will try to fetch
	509	+ # the submodules it discovers as a result of fetching the primary repo.
	510	+ result = cli.run(project=project, args=['fetch', 'target.bst'])
	511	+
	512	+ # Assert a warning or an error depending on what we're checking
	513	+ if fail == 'error':
	514	+ result.assert_main_error(ErrorDomain.STREAM, None)
	515	+ result.assert_task_error(ErrorDomain.PLUGIN, 'git:unlisted-submodule')
	516	+ else:
	517	+ result.assert_success()
	518	+ assert "git:unlisted-submodule" in result.stderr
	519	+
	520	+ # Now that we've fetched it, `bst show` will discover the unlisted submodule too
	521	+ result = cli.run(project=project, args=['show', 'target.bst'])
	522	+
	523	+ # Assert a warning or an error depending on what we're checking
	524	+ if fail == 'error':
	525	+ result.assert_main_error(ErrorDomain.PLUGIN, 'git:unlisted-submodule')
	526	+ else:
	527	+ result.assert_success()
	528	+ assert "git:unlisted-submodule" in result.stderr
	529	+
	530	+
	531	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	532	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	533	+@pytest.mark.parametrize("fail", ['warn', 'error'])
	534	+def test_track_unlisted_submodule(cli, tmpdir, datafiles, fail):
	535	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	536	+
	537	+ # Make the warning an error if we're testing errors
	538	+ if fail == 'error':
	539	+ project_template = {
	540	+ "name": "foo",
	541	+ "fatal-warnings": ['git:unlisted-submodule']
	542	+ }
	543	+ _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	544	+
	545	+ # Create the submodule first from the 'subrepofiles' subdir
	546	+ subrepo = create_repo('git', str(tmpdir), 'subrepo')
	547	+ subrepo.create(os.path.join(project, 'subrepofiles'))
	548	+
	549	+ # Create the repo from 'repofiles' subdir
458	550	repo = create_repo('git', str(tmpdir))
459	551	ref = repo.create(os.path.join(project, 'repofiles'))
460	552
	553	+ # Add a submodule pointing to the one we created, but use
	554	+ # the original ref, let the submodules appear after tracking
	555	+ repo.add_submodule('subdir', 'file://' + subrepo.repo)
	556	+
	557	+ # Create the source, and delete the explicit configuration
	558	+ # of the submodules.
461	559	gitsource = repo.source_config(ref=ref)
	560	+ del gitsource['submodules']
462	561
463		- # Overwrite the track value to the added branch
464		- gitsource['track'] = 'foo'
	562	+ # Write out our test target
	563	+ element = {
	564	+ 'kind': 'import',
	565	+ 'sources': [
	566	+ gitsource
	567	+ ]
	568	+ }
	569	+ _yaml.dump(element, os.path.join(project, 'target.bst'))
	570	+
	571	+ # Fetch the repo, we will not see the warning because we
	572	+ # are still pointing to a ref which predates the submodules
	573	+ result = cli.run(project=project, args=['fetch', 'target.bst'])
	574	+ result.assert_success()
	575	+ assert "git:unlisted-submodule" not in result.stderr
	576	+
	577	+ # We won't get a warning/error when tracking either, the source
	578	+ # has not become Consistency.CACHED so the opportunity to check
	579	+ # for the warning has not yet arisen.
	580	+ result = cli.run(project=project, args=['track', 'target.bst'])
	581	+ result.assert_success()
	582	+ assert "git:unlisted-submodule" not in result.stderr
	583	+
	584	+ # Fetching the repo at the new ref will finally reveal the warning
	585	+ result = cli.run(project=project, args=['fetch', 'target.bst'])
	586	+ if fail == 'error':
	587	+ result.assert_main_error(ErrorDomain.STREAM, None)
	588	+ result.assert_task_error(ErrorDomain.PLUGIN, 'git:unlisted-submodule')
	589	+ else:
	590	+ result.assert_success()
	591	+ assert "git:unlisted-submodule" in result.stderr
	592	+
	593	+
	594	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	595	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	596	+@pytest.mark.parametrize("fail", ['warn', 'error'])
	597	+def test_invalid_submodule(cli, tmpdir, datafiles, fail):
	598	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	599	+
	600	+ # Make the warning an error if we're testing errors
	601	+ if fail == 'error':
	602	+ project_template = {
	603	+ "name": "foo",
	604	+ "fatal-warnings": ['git:invalid-submodule']
	605	+ }
	606	+ _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	607	+
	608	+ # Create the repo from 'repofiles' subdir
	609	+ repo = create_repo('git', str(tmpdir))
	610	+ ref = repo.create(os.path.join(project, 'repofiles'))
	611	+
	612	+ # Create the source without any submodules, and add
	613	+ # an invalid submodule configuration to it.
	614	+ #
	615	+ # We expect this to cause an invalid submodule warning
	616	+ # after the source has been fetched and we know what
	617	+ # the real submodules actually are.
	618	+ #
	619	+ gitsource = repo.source_config(ref=ref)
	620	+ gitsource['submodules'] = {
	621	+ 'subdir': {
	622	+ 'url': 'https://pony.org/repo.git'
	623	+ }
	624	+ }
465	625
466	626	# Write out our test target
467	627	element = {
...	...	@@ -472,11 +632,95 @@ def test_ref_not_in_track_warn_error(cli, tmpdir, datafiles):
472	632	}
473	633	_yaml.dump(element, os.path.join(project, 'target.bst'))
474	634
475		- # Assert that build raises a warning here that is captured
476		- # as plugin error, due to the fatal warning being set
477		- result = cli.run(project=project, args=['build', 'target.bst'])
478		- result.assert_main_error(ErrorDomain.STREAM, None)
479		- result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.REF_NOT_IN_TRACK)
	635	+ # We will not see the warning or error before the first fetch, because
	636	+ # we don't have the repository yet and so we have no knowledge of
	637	+ # the unlisted submodule.
	638	+ result = cli.run(project=project, args=['show', 'target.bst'])
	639	+ result.assert_success()
	640	+ assert "git:invalid-submodule" not in result.stderr
	641	+
	642	+ # We will notice this directly in fetch, as it will try to fetch
	643	+ # the submodules it discovers as a result of fetching the primary repo.
	644	+ result = cli.run(project=project, args=['fetch', 'target.bst'])
	645	+
	646	+ # Assert a warning or an error depending on what we're checking
	647	+ if fail == 'error':
	648	+ result.assert_main_error(ErrorDomain.STREAM, None)
	649	+ result.assert_task_error(ErrorDomain.PLUGIN, 'git:invalid-submodule')
	650	+ else:
	651	+ result.assert_success()
	652	+ assert "git:invalid-submodule" in result.stderr
	653	+
	654	+ # Now that we've fetched it, `bst show` will discover the unlisted submodule too
	655	+ result = cli.run(project=project, args=['show', 'target.bst'])
	656	+
	657	+ # Assert a warning or an error depending on what we're checking
	658	+ if fail == 'error':
	659	+ result.assert_main_error(ErrorDomain.PLUGIN, 'git:invalid-submodule')
	660	+ else:
	661	+ result.assert_success()
	662	+ assert "git:invalid-submodule" in result.stderr
	663	+
	664	+
	665	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	666	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	667	+@pytest.mark.parametrize("fail", ['warn', 'error'])
	668	+def test_track_invalid_submodule(cli, tmpdir, datafiles, fail):
	669	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	670	+
	671	+ # Make the warning an error if we're testing errors
	672	+ if fail == 'error':
	673	+ project_template = {
	674	+ "name": "foo",
	675	+ "fatal-warnings": ['git:invalid-submodule']
	676	+ }
	677	+ _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	678	+
	679	+ # Create the submodule first from the 'subrepofiles' subdir
	680	+ subrepo = create_repo('git', str(tmpdir), 'subrepo')
	681	+ subrepo.create(os.path.join(project, 'subrepofiles'))
	682	+
	683	+ # Create the repo from 'repofiles' subdir
	684	+ repo = create_repo('git', str(tmpdir))
	685	+ ref = repo.create(os.path.join(project, 'repofiles'))
	686	+
	687	+ # Add a submodule pointing to the one we created
	688	+ ref = repo.add_submodule('subdir', 'file://' + subrepo.repo)
	689	+
	690	+ # Add a commit beyond the ref which *removes* the submodule we've added
	691	+ repo.remove_path('subdir')
	692	+
	693	+ # Create the source, this will keep the submodules so initially
	694	+ # the configuration is valid for the ref we're using
	695	+ gitsource = repo.source_config(ref=ref)
	696	+
	697	+ # Write out our test target
	698	+ element = {
	699	+ 'kind': 'import',
	700	+ 'sources': [
	701	+ gitsource
	702	+ ]
	703	+ }
	704	+ _yaml.dump(element, os.path.join(project, 'target.bst'))
	705	+
	706	+ # Fetch the repo, we will not see the warning because we
	707	+ # are still pointing to a ref which predates the submodules
	708	+ result = cli.run(project=project, args=['fetch', 'target.bst'])
	709	+ result.assert_success()
	710	+ assert "git:invalid-submodule" not in result.stderr
	711	+
	712	+ # In this case, we will get the error directly after tracking,
	713	+ # since the new HEAD does not require any submodules which are
	714	+ # not locally cached, the Source will be CACHED directly after
	715	+ # tracking and the validations will occur as a result.
	716	+ #
	717	+ result = cli.run(project=project, args=['track', 'target.bst'])
	718	+ if fail == 'error':
	719	+ result.assert_main_error(ErrorDomain.STREAM, None)
	720	+ result.assert_task_error(ErrorDomain.PLUGIN, 'git:invalid-submodule')
	721	+ else:
	722	+ result.assert_success()
	723	+ assert "git:invalid-submodule" in result.stderr
480	724
481	725
482	726	@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")

tests/testutils/repo/git.py

---

...	...	@@ -76,6 +76,12 @@ class Git(Repo):
76	76	self._run_git('commit', '-m', 'Added the submodule')
77	77	return self.latest_commit()
78	78
	79	+ # This can also be used to a file or a submodule
	80	+ def remove_path(self, path):
	81	+ self._run_git('rm', path)
	82	+ self._run_git('commit', '-m', 'Removing {}'.format(path))
	83	+ return self.latest_commit()
	84	+
79	85	def source_config(self, ref=None, checkout_submodules=None):
80	86	config = {
81	87	'kind': 'git',

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.