[Notes] [Git][BuildStream/buildstream][edbaunton/doc-typo] 26 commits: cascache.py: Raise ArtifactError on grpc error

[Tristan Van Berkom <gitlab mg gitlab com>]

Title: GitLab

Tristan Van Berkom pushed to branch edbaunton/doc-typo at BuildStream / buildstream

Commits:

• fb222ba1
  by Josh Smith at 2018-07-27T08:50:45Z

  cascache.py: Raise ArtifactError on grpc error
  
  This aims to fix issue #502.
  

• 48916b8a
  by Tristan Maat at 2018-07-27T10:22:41Z

  Merge branch 'Qinusty/502' into 'master'
  
  cascache.py: Raise ArtifactError on grpc error
  
  Closes #502
  
  See merge request BuildStream/buildstream!576

• 23e080b9
  by James Ennis at 2018-07-27T11:13:50Z

  README.md: Minor corrections
  

• 196cfffc
  by James Ennis at 2018-07-27T11:13:50Z

  README.md: Direct users towards the tutorial before existing projects:
  
    This patch also removes displaying the projects that live within
    the buildstream-examples repository, as we wish to deprecate this
    repo and migrate the two examples.
    - #438 and #439
  
    In addition to this, both examples do not 'build' in their current
    state.
  

• 2b93574e
  by James Ennis at 2018-07-27T11:13:50Z

  main_install.rst: Add note to refer Windows and macOS users to docker
  

• 63e2320e
  by Josh Smith at 2018-07-27T11:16:55Z

  .gitignore: Ignore .pytest_cache/
  

• 6ea97b17
  by Javier Jardón at 2018-07-27T11:54:20Z

  Merge branch 'jennis/doc_improvements' into 'master'
  
  Refer readers to our tutorial before referring them to existing bst projects
  
  See merge request BuildStream/buildstream!578

• 8b46e874
  by Jonathan Maw at 2018-07-27T12:24:56Z

  project: Parse and store mirrors
  

• a2e9c62a
  by Jonathan Maw at 2018-07-27T12:24:56Z

  Move _ALIAS_SEPARATOR into utils
  
  The separator is useful in source files other than _project.py
  

• 7c993ac0
  by Jonathan Maw at 2018-07-27T12:24:56Z

  source: Store the url aliases or use an override
  
  This is part of a later plan to implement mirroring without forcing
  everyone to update their sources. We use the expected calls to
  Source.translate_url() when running Source.configure() to extract the
  aliases from the URL. Multiple aliases must be extracted because
  sources exist that may fetch from multiple aliases (for example, git
  submodules)
  
  Later, we want to substitute another URI where the alias normally reads
  from the project - We accomplish this by re-instantiating the Source
  with the alias overrides passed as an argument to the constructor.
  

• 2889003c
  by Jonathan Maw at 2018-07-27T12:24:56Z

  source: When fetching, try to fetch from mirrors first
  
  **KLUDGE WARNING**: This involves making the source store its "meta"
  object so that it's possible to create a copy of the source inside the
  fetch queue, instead of back when the pipeline was being loaded.
  
  This adds the SourceFetcher class, which is intended for sources that
  fetch from multiple URLs (e.g. the git source and its submodules)
  
  Fix when fetching
  

• f81e8e7b
  by Jonathan Maw at 2018-07-27T12:24:56Z

  Set default mirror via command-line or user config
  
  In user config (buildstream.conf), it is set with the "default-mirror"
  field.
  
  On the command-line, it is set with "--default-mirror"
  

• 755ed898
  by Jonathan Maw at 2018-07-27T12:24:56Z

  Add mirrored source tracking
  

• 19c01a56
  by Jonathan Maw at 2018-07-27T12:24:56Z

  bzr.py: Improve mirror support
  
  This fixes:
  * Bzr repositories pulling from the branch they were created with.
  * Bzr's _ensure_mirror() not actually checking that it successfully
    mirrored the ref.
  

• 84872141
  by Jonathan Maw at 2018-07-27T12:24:56Z

  git.py: Improve mirror support
  
  When implementing fetching from mirrors, I encountered some problems
  with the git source:
  
  1. The mirror URL was using translate_url()'s output, so if a different
     alias was used, then fetching from the mirror would go to a different
     directory, and be inaccessible.
  2. After fixing that, fetching was unable to pull from a URL other than
     the one used at repository creation, meaning it wouldn't actually
     pull from the mirror.
  

• bd51a0b2
  by Jonathan Maw at 2018-07-27T12:24:56Z

  testutils: Add a helper to copy a testutils repo
  
  This is helpful if you want to test what happens when you have one repo
  that has diverged from another. By copying the repo you're sure they
  start with shared history.
  
  This is especially useful when mirroring.
  

• 909120ab
  by Jonathan Maw at 2018-07-27T12:24:56Z

  tests: Add mirrored fetching and tracking tests
  

• 1cbc2e17
  by Jonathan Maw at 2018-07-27T12:24:56Z

  doc: Add documentation for source mirroring
  

• 202d9d26
  by Jonathan Maw at 2018-07-27T12:24:56Z

  NEWS: Add item for source mirroring
  

• 2b23898d
  by Jonathan Maw at 2018-07-27T12:24:56Z

  doc: Add tutorials for setting up git and tar mirrors
  

• bd4d0355
  by Jonathan Maw at 2018-07-27T13:08:00Z

  Merge branch '328-support-for-downloading-sources-from-mirrors' into 'master'
  
  Resolve "Support for downloading sources from mirrors"
  
  Closes #328
  
  See merge request BuildStream/buildstream!404

• cb4693b2
  by Josh Smith at 2018-07-27T14:00:26Z

  Merge branch 'Qinusty/pytest_cache_gitignore'
  

• 8a96679a
  by Josh Smith at 2018-07-27T14:01:07Z

  Merge branch 'master' of gitlab.com:BuildStream/buildstream
  

• f5c8ff61
  by Josh Smith at 2018-07-27T14:10:45Z

  _context.py: Cache size is now restricted to available disk space
  
  This address issue #491.
  
  When attempting to run buildstream with a configuration specifying
  a cache quota larger than your available disk space, buildstream
  will alert the user and exit.
  
  Note:
  This takes into consideration your current cache usage and
  therefore restricts the overall size of your artifact cache folder.
  

• 32ddb544
  by Qinusty at 2018-07-27T14:57:30Z

  Merge branch 'Qinusty/491' into 'master'
  
  Cache quota is now restricted to available disk space
  
  See merge request BuildStream/buildstream!563

• 2f59328a
  by Ed Baunton at 2018-07-27T15:05:31Z

  element.py (docs): dashes not underscores for build and install root
  
  Fix small error in docs in relation to `get_variable` documentation.
  

26 changed files:

• .gitignore
• NEWS
• README.rst
• buildstream/__init__.py
• buildstream/_artifactcache/cascache.py
• buildstream/_context.py
• buildstream/_frontend/app.py
• buildstream/_frontend/cli.py
• buildstream/_loader/loader.py
• buildstream/_project.py
• buildstream/_versions.py
• buildstream/element.py
• buildstream/plugins/sources/bzr.py
• buildstream/plugins/sources/git.py
• buildstream/source.py
• buildstream/utils.py
• + doc/source/examples/git-mirror.rst
• + doc/source/examples/tar-mirror.rst
• doc/source/format_project.rst
• doc/source/main_install.rst
• doc/source/using_config.rst
• doc/source/using_examples.rst
• tests/completions/completions.py
• + tests/frontend/mirror.py
• + tests/frontend/project/sources/fetch_source.py
• tests/testutils/repo/repo.py

Changes:

.gitignore

---

...	...	@@ -15,6 +15,7 @@ tmp
15	15	.coverage
16	16	.coverage.*
17	17	.cache
	18	+.pytest_cache/
18	19	*.bst/
19	20
20	21	# Pycache, in case buildstream is ran directly from within the source

NEWS

---

...	...	@@ -5,6 +5,10 @@ buildstream 1.3.1
5	5	o Add a `--tar` option to `bst checkout` which allows a tarball to be
6	6	created from the artifact contents.
7	7
	8	+ o Fetching and tracking will consult mirrors defined in project config,
	9	+ and the preferred mirror to fetch from can be defined in the command
	10	+ line or user config.
	11	+
8	12	=================
9	13	buildstream 1.1.4
10	14	=================

README.rst

---

...	...	@@ -25,7 +25,7 @@ BuildStream offers the following advantages:
25	25
26	26	* **Declarative build instructions/definitions**
27	27
28		- BuildStream provides a a flexible and extensible framework for the modelling
	28	+ BuildStream provides a flexible and extensible framework for the modelling
29	29	of software build pipelines in a declarative YAML format, which allows you to
30	30	manipulate filesystem data in a controlled, reproducible sandboxed environment.
31	31
...	...	@@ -61,25 +61,29 @@ How does BuildStream work?
61	61	==========================
62	62	BuildStream operates on a set of YAML files (.bst files), as follows:
63	63
64		-* loads the YAML files which describe the target(s) and all dependencies
65		-* evaluates the version information and build instructions to calculate a build
	64	+* Loads the YAML files which describe the target(s) and all dependencies.
	65	+* Evaluates the version information and build instructions to calculate a build
66	66	graph for the target(s) and all dependencies and unique cache-keys for each
67		- element
68		-* retrieves elements from cache if they are already built, or builds them in a
69		- sandboxed environment using the instructions declared in the .bst files
70		-* transforms/configures and/or deploys the resulting target(s) based on the
	67	+ element.
	68	+* Retrieves previously built elements (artifacts) from a local/remote cache, or
	69	+ builds the elements in a sandboxed environment using the instructions declared
	70	+ in the .bst files.
	71	+* Transforms/configures and/or deploys the resulting target(s) based on the
71	72	instructions declared in the .bst files.
72	73
73	74
74	75	How can I get started?
75	76	======================
76		-The easiest way to get started is to explore some existing .bst files, for example:
	77	+To start using BuildStream, first,
	78	+`install <https://buildstream.gitlab.io/buildstream/main_install.html>`_
	79	+BuildStream onto your machine and then follow our
	80	+`tutorial <https://buildstream.gitlab.io/buildstream/using_tutorial.html>`_.
	81	+
	82	+We also recommend exploring some existing BuildStream projects:
77	83
78	84	* https://gitlab.gnome.org/GNOME/gnome-build-meta/
79	85	* https://gitlab.com/freedesktop-sdk/freedesktop-sdk
80	86	* https://gitlab.com/baserock/definitions
81		-* https://gitlab.com/BuildStream/buildstream-examples/tree/master/build-x86image
82		-* https://gitlab.com/BuildStream/buildstream-examples/tree/master/netsurf-flatpak
83	87
84	88	If you have any questions please ask on our `#buildstream <irc://irc.gnome.org/buildstream>`_ channel in `irc.gnome.org <irc://irc.gnome.org>`_
85	89

buildstream/__init__.py

---

...	...	@@ -29,7 +29,7 @@ if "_BST_COMPLETION" not in os.environ:
29	29	from .utils import UtilError, ProgramNotFoundError
30	30	from .sandbox import Sandbox, SandboxFlags
31	31	from .plugin import Plugin
32		- from .source import Source, SourceError, Consistency
	32	+ from .source import Source, SourceError, Consistency, SourceFetcher
33	33	from .element import Element, ElementError, Scope
34	34	from .buildelement import BuildElement
35	35	from .scriptelement import ScriptElement

buildstream/_artifactcache/cascache.py

---

...	...	@@ -240,7 +240,8 @@ class CASCache(ArtifactCache):
240	240
241	241	except grpc.RpcError as e:
242	242	if e.code() != grpc.StatusCode.NOT_FOUND:
243		- raise
	243	+ raise ArtifactError("Failed to pull artifact {}: {}".format(
	244	+ element._get_brief_display_key(), e)) from e
244	245
245	246	return False
246	247
...	...	@@ -285,6 +286,7 @@ class CASCache(ArtifactCache):
285	286
286	287	except grpc.RpcError as e:
287	288	if e.code() != grpc.StatusCode.NOT_FOUND:
	289	+ # Intentionally re-raise RpcError for outer except block.
288	290	raise
289	291
290	292	missing_blobs = {}

buildstream/_context.py

---

...	...	@@ -197,29 +197,55 @@ class Context():
197	197	"\nValid values are, for example: 800M 10G 1T 50%\n"
198	198	.format(str(e))) from e
199	199
200		- # If we are asked not to set a quota, we set it to the maximum
201		- # disk space available minus a headroom of 2GB, such that we
202		- # at least try to avoid raising Exceptions.
	200	+ # Headroom intended to give BuildStream a bit of leeway.
	201	+ # This acts as the minimum size of cache_quota and also
	202	+ # is taken from the user requested cache_quota.
203	203	#
204		- # Of course, we might still end up running out during a build
205		- # if we end up writing more than 2G, but hey, this stuff is
206		- # already really fuzzy.
207		- #
208		- if cache_quota is None:
209		- stat = os.statvfs(artifactdir_volume)
210		- # Again, the artifact directory may not yet have been
211		- # created
212		- if not os.path.exists(self.artifactdir):
213		- cache_size = 0
214		- else:
215		- cache_size = utils._get_dir_size(self.artifactdir)
216		- cache_quota = cache_size + stat.f_bsize * stat.f_bavail
217		-
218	204	if 'BST_TEST_SUITE' in os.environ:
219	205	headroom = 0
220	206	else:
221	207	headroom = 2e9
222	208
	209	+ stat = os.statvfs(artifactdir_volume)
	210	+ available_space = (stat.f_bsize * stat.f_bavail)
	211	+
	212	+ # Again, the artifact directory may not yet have been created yet
	213	+ #
	214	+ if not os.path.exists(self.artifactdir):
	215	+ cache_size = 0
	216	+ else:
	217	+ cache_size = utils._get_dir_size(self.artifactdir)
	218	+
	219	+ # Ensure system has enough storage for the cache_quota
	220	+ #
	221	+ # If cache_quota is none, set it to the maximum it could possibly be.
	222	+ #
	223	+ # Also check that cache_quota is atleast as large as our headroom.
	224	+ #
	225	+ if cache_quota is None: # Infinity, set to max system storage
	226	+ cache_quota = cache_size + available_space
	227	+ if cache_quota < headroom: # Check minimum
	228	+ raise LoadError(LoadErrorReason.INVALID_DATA,
	229	+ "Invalid cache quota ({}): ".format(utils._pretty_size(cache_quota)) +
	230	+ "BuildStream requires a minimum cache quota of 2G.")
	231	+ elif cache_quota > cache_size + available_space: # Check maximum
	232	+ raise LoadError(LoadErrorReason.INVALID_DATA,
	233	+ ("Your system does not have enough available " +
	234	+ "space to support the cache quota specified.\n" +
	235	+ "You currently have:\n" +
	236	+ "- {used} of cache in use at {local_cache_path}\n" +
	237	+ "- {available} of available system storage").format(
	238	+ used=utils._pretty_size(cache_size),
	239	+ local_cache_path=self.artifactdir,
	240	+ available=utils._pretty_size(available_space)))
	241	+
	242	+ # Place a slight headroom (2e9 (2GB) on the cache_quota) into
	243	+ # cache_quota to try and avoid exceptions.
	244	+ #
	245	+ # Of course, we might still end up running out during a build
	246	+ # if we end up writing more than 2G, but hey, this stuff is
	247	+ # already really fuzzy.
	248	+ #
223	249	self.cache_quota = cache_quota - headroom
224	250	self.cache_lower_threshold = self.cache_quota / 2
225	251
...	...	@@ -259,7 +285,7 @@ class Context():
259	285	# Shallow validation of overrides, parts of buildstream which rely
260	286	# on the overrides are expected to validate elsewhere.
261	287	for _, overrides in _yaml.node_items(self._project_overrides):
262		- _yaml.node_validate(overrides, ['artifacts', 'options', 'strict'])
	288	+ _yaml.node_validate(overrides, ['artifacts', 'options', 'strict', 'default-mirror'])
263	289
264	290	profile_end(Topics.LOAD_CONTEXT, 'load')
265	291

buildstream/_frontend/app.py

---

...	...	@@ -202,7 +202,8 @@ class App():
202	202	# Load the Project
203	203	#
204	204	try:
205		- self.project = Project(directory, self.context, cli_options=self._main_options['option'])
	205	+ self.project = Project(directory, self.context, cli_options=self._main_options['option'],
	206	+ default_mirror=self._main_options.get('default_mirror'))
206	207	except LoadError as e:
207	208
208	209	# Let's automatically start a `bst init` session in this case

buildstream/_frontend/cli.py

---

...	...	@@ -217,6 +217,8 @@ def print_version(ctx, param, value):
217	217	help="Elements must be rebuilt when their dependencies have changed")
218	218	@click.option('--option', '-o', type=click.Tuple([str, str]), multiple=True, metavar='OPTION VALUE',
219	219	help="Specify a project option")
	220	+@click.option('--default-mirror', default=None,
	221	+ help="The mirror to fetch from first, before attempting other mirrors")
220	222	@click.pass_context
221	223	def cli(context, **kwargs):
222	224	"""Build and manipulate BuildStream projects

buildstream/_loader/loader.py

---

...	...	@@ -513,7 +513,7 @@ class Loader():
513	513	if self._fetch_subprojects:
514	514	if ticker:
515	515	ticker(filename, 'Fetching subproject from {} source'.format(source.get_kind()))
516		- source.fetch()
	516	+ source._fetch()
517	517	else:
518	518	detail = "Try fetching the project with `bst fetch {}`".format(filename)
519	519	raise LoadError(LoadErrorReason.SUBPROJECT_FETCH_NEEDED,

buildstream/_project.py

---

...	...	@@ -19,7 +19,7 @@
19	19
20	20	import os
21	21	import multiprocessing # for cpu_count()
22		-from collections import Mapping
	22	+from collections import Mapping, OrderedDict
23	23	from pluginbase import PluginBase
24	24	from . import utils
25	25	from . import _cachekey
...	...	@@ -35,9 +35,6 @@ from ._projectrefs import ProjectRefs, ProjectRefStorage
35	35	from ._versions import BST_FORMAT_VERSION
36	36
37	37
38		-# The separator we use for user specified aliases
39		-_ALIAS_SEPARATOR = ':'
40		-
41	38	# Project Configuration file
42	39	_PROJECT_CONF_FILE = 'project.conf'
43	40
...	...	@@ -70,7 +67,7 @@ class HostMount():
70	67	#
71	68	class Project():
72	69
73		- def __init__(self, directory, context, *, junction=None, cli_options=None):
	70	+ def __init__(self, directory, context, *, junction=None, cli_options=None, default_mirror=None):
74	71
75	72	# The project name
76	73	self.name = None
...	...	@@ -94,6 +91,8 @@ class Project():
94	91	self.base_env_nocache = None # The base nocache mask (list) for the environment
95	92	self.element_overrides = {} # Element specific configurations
96	93	self.source_overrides = {} # Source specific configurations
	94	+ self.mirrors = OrderedDict() # contains dicts of alias-mappings to URIs.
	95	+ self.default_mirror = default_mirror # The name of the preferred mirror.
97	96
98	97	#
99	98	# Private Members
...	...	@@ -133,8 +132,8 @@ class Project():
133	132	# fully qualified urls based on the shorthand which is allowed
134	133	# to be specified in the YAML
135	134	def translate_url(self, url):
136		- if url and _ALIAS_SEPARATOR in url:
137		- url_alias, url_body = url.split(_ALIAS_SEPARATOR, 1)
	135	+ if url and utils._ALIAS_SEPARATOR in url:
	136	+ url_alias, url_body = url.split(utils._ALIAS_SEPARATOR, 1)
138	137	alias_url = self._aliases.get(url_alias)
139	138	if alias_url:
140	139	url = alias_url + url_body
...	...	@@ -202,6 +201,36 @@ class Project():
202	201	self._assert_plugin_format(source, version)
203	202	return source
204	203
	204	+ # get_alias_uri()
	205	+ #
	206	+ # Returns the URI for a given alias, if it exists
	207	+ #
	208	+ # Args:
	209	+ # alias (str): The alias.
	210	+ #
	211	+ # Returns:
	212	+ # str: The URI for the given alias; or None: if there is no URI for
	213	+ # that alias.
	214	+ def get_alias_uri(self, alias):
	215	+ return self._aliases.get(alias)
	216	+
	217	+ # get_alias_uris()
	218	+ #
	219	+ # Returns a list of every URI to replace an alias with
	220	+ def get_alias_uris(self, alias):
	221	+ if not alias or alias not in self._aliases:
	222	+ return [None]
	223	+
	224	+ mirror_list = []
	225	+ for key, alias_mapping in self.mirrors.items():
	226	+ if alias in alias_mapping:
	227	+ if key == self.default_mirror:
	228	+ mirror_list = alias_mapping[alias] + mirror_list
	229	+ else:
	230	+ mirror_list += alias_mapping[alias]
	231	+ mirror_list.append(self._aliases[alias])
	232	+ return mirror_list
	233	+
205	234	# _load():
206	235	#
207	236	# Loads the project configuration file in the project directory.
...	...	@@ -249,7 +278,7 @@ class Project():
249	278	'aliases', 'name',
250	279	'artifacts', 'options',
251	280	'fail-on-overlap', 'shell',
252		- 'ref-storage', 'sandbox'
	281	+ 'ref-storage', 'sandbox', 'mirrors',
253	282	])
254	283
255	284	# The project name, element path and option declarations
...	...	@@ -290,6 +319,10 @@ class Project():
290	319	#
291	320	self.options.process_node(config)
292	321
	322	+ # Override default_mirror if not set by command-line
	323	+ if not self.default_mirror:
	324	+ self.default_mirror = _yaml.node_get(overrides, str, 'default-mirror', default_value=None)
	325	+
293	326	#
294	327	# Now all YAML composition is done, from here on we just load
295	328	# the values from our loaded configuration dictionary.
...	...	@@ -414,6 +447,21 @@ class Project():
414	447
415	448	self._shell_host_files.append(mount)
416	449
	450	+ mirrors = _yaml.node_get(config, list, 'mirrors', default_value=[])
	451	+ for mirror in mirrors:
	452	+ allowed_mirror_fields = [
	453	+ 'name', 'aliases'
	454	+ ]
	455	+ _yaml.node_validate(mirror, allowed_mirror_fields)
	456	+ mirror_name = _yaml.node_get(mirror, str, 'name')
	457	+ alias_mappings = {}
	458	+ for alias_mapping, uris in _yaml.node_items(mirror['aliases']):
	459	+ assert isinstance(uris, list)
	460	+ alias_mappings[alias_mapping] = list(uris)
	461	+ self.mirrors[mirror_name] = alias_mappings
	462	+ if not self.default_mirror:
	463	+ self.default_mirror = mirror_name
	464	+
417	465	# _assert_plugin_format()
418	466	#
419	467	# Helper to raise a PluginError if the loaded plugin is of a lesser version then

buildstream/_versions.py

---

...	...	@@ -23,7 +23,7 @@
23	23	# This version is bumped whenever enhancements are made
24	24	# to the `project.conf` format or the core element format.
25	25	#
26		-BST_FORMAT_VERSION = 10
	26	+BST_FORMAT_VERSION = 11
27	27
28	28
29	29	# The base BuildStream artifact version

buildstream/element.py

---

...	...	@@ -349,8 +349,8 @@ class Element(Plugin):
349	349	generated script is run:
350	350
351	351	- All element variables have been exported.
352		- - The cwd is `self.get_variable('build_root')/self.normal_name`.
353		- - $PREFIX is set to `self.get_variable('install_root')`.
	352	+ - The cwd is `self.get_variable('build-root')/self.normal_name`.
	353	+ - $PREFIX is set to `self.get_variable('install-root')`.
354	354	- The directory indicated by $PREFIX is an empty directory.
355	355
356	356	Files are expected to be installed to $PREFIX.

buildstream/plugins/sources/bzr.py

---

...	...	@@ -102,7 +102,7 @@ class BzrSource(Source):
102	102	def track(self):
103	103	with self.timed_activity("Tracking {}".format(self.url),
104	104	silent_nested=True):
105		- self._ensure_mirror()
	105	+ self._ensure_mirror(skip_ref_check=True)
106	106	ret, out = self.check_output([self.host_bzr, "version-info",
107	107	"--custom", "--template={revno}",
108	108	self._get_branch_dir()],
...	...	@@ -214,7 +214,7 @@ class BzrSource(Source):
214	214	yield repodir
215	215	self._atomic_replace_mirrordir(repodir)
216	216
217		- def _ensure_mirror(self):
	217	+ def _ensure_mirror(self, skip_ref_check=False):
218	218	with self._atomic_repodir() as repodir:
219	219	# Initialize repo if no metadata
220	220	bzr_metadata_dir = os.path.join(repodir, ".bzr")
...	...	@@ -223,18 +223,21 @@ class BzrSource(Source):
223	223	fail="Failed to initialize bzr repository")
224	224
225	225	branch_dir = os.path.join(repodir, self.tracking)
	226	+ branch_url = self.url + "/" + self.tracking
226	227	if not os.path.exists(branch_dir):
227	228	# `bzr branch` the branch if it doesn't exist
228	229	# to get the upstream code
229		- branch_url = self.url + "/" + self.tracking
230	230	self.call([self.host_bzr, "branch", branch_url, branch_dir],
231	231	fail="Failed to branch from {} to {}".format(branch_url, branch_dir))
232	232
233	233	else:
234	234	# `bzr pull` the branch if it does exist
235	235	# to get any changes to the upstream code
236		- self.call([self.host_bzr, "pull", "--directory={}".format(branch_dir)],
	236	+ self.call([self.host_bzr, "pull", "--directory={}".format(branch_dir), branch_url],
237	237	fail="Failed to pull new changes for {}".format(branch_dir))
	238	+ if not skip_ref_check and not self._check_ref():
	239	+ raise SourceError("Failed to ensure ref '{}' was mirrored".format(self.ref),
	240	+ reason="ref-not-mirrored")
238	241
239	242
240	243	def setup():

buildstream/plugins/sources/git.py

---

...	...	@@ -78,7 +78,7 @@ from io import StringIO
78	78
79	79	from configparser import RawConfigParser
80	80
81		-from buildstream import Source, SourceError, Consistency
	81	+from buildstream import Source, SourceError, Consistency, SourceFetcher
82	82	from buildstream import utils
83	83
84	84	GIT_MODULES = '.gitmodules'
...	...	@@ -88,18 +88,20 @@ GIT_MODULES = '.gitmodules'
88	88	# for the primary git source and also for each submodule it
89	89	# might have at a given time
90	90	#
91		-class GitMirror():
	91	+class GitMirror(SourceFetcher):
92	92
93	93	def __init__(self, source, path, url, ref):
94	94
	95	+ super().__init__()
95	96	self.source = source
96	97	self.path = path
97		- self.url = source.translate_url(url)
	98	+ self.url = url
98	99	self.ref = ref
99		- self.mirror = os.path.join(source.get_mirror_directory(), utils.url_directory_name(self.url))
	100	+ self.mirror = os.path.join(source.get_mirror_directory(), utils.url_directory_name(url))
	101	+ self.mark_download_url(url)
100	102
101	103	# Ensures that the mirror exists
102		- def ensure(self):
	104	+ def ensure(self, alias_override=None):
103	105
104	106	# Unfortunately, git does not know how to only clone just a specific ref,
105	107	# so we have to download all of those gigs even if we only need a couple
...	...	@@ -112,22 +114,47 @@ class GitMirror():
112	114	# system configured tmpdir is not on the same partition.
113	115	#
114	116	with self.source.tempdir() as tmpdir:
115		- self.source.call([self.source.host_git, 'clone', '--mirror', '-n', self.url, tmpdir],
116		- fail="Failed to clone git repository {}".format(self.url),
	117	+ url = self.source.translate_url(self.url, alias_override=alias_override)
	118	+ self.source.call([self.source.host_git, 'clone', '--mirror', '-n', url, tmpdir],
	119	+ fail="Failed to clone git repository {}".format(url),
117	120	fail_temporarily=True)
118	121
119	122	try:
120	123	shutil.move(tmpdir, self.mirror)
121	124	except (shutil.Error, OSError) as e:
122	125	raise SourceError("{}: Failed to move cloned git repository {} from '{}' to '{}'"
123		- .format(self.source, self.url, tmpdir, self.mirror)) from e
	126	+ .format(self.source, url, tmpdir, self.mirror)) from e
	127	+
	128	+ def _fetch(self, alias_override=None):
	129	+ url = self.source.translate_url(self.url, alias_override=alias_override)
	130	+
	131	+ if alias_override:
	132	+ remote_name = utils.url_directory_name(alias_override)
	133	+ _, remotes = self.source.check_output(
	134	+ [self.source.host_git, 'remote'],
	135	+ fail="Failed to retrieve list of remotes in {}".format(self.mirror),
	136	+ cwd=self.mirror
	137	+ )
	138	+ if remote_name not in remotes:
	139	+ self.source.call(
	140	+ [self.source.host_git, 'remote', 'add', remote_name, url],
	141	+ fail="Failed to add remote {} with url {}".format(remote_name, url),
	142	+ cwd=self.mirror
	143	+ )
	144	+ else:
	145	+ remote_name = "origin"
124	146
125		- def fetch(self):
126		- self.source.call([self.source.host_git, 'fetch', 'origin', '--prune'],
127		- fail="Failed to fetch from remote git repository: {}".format(self.url),
	147	+ self.source.call([self.source.host_git, 'fetch', remote_name, '--prune'],
	148	+ fail="Failed to fetch from remote git repository: {}".format(url),
128	149	fail_temporarily=True,
129	150	cwd=self.mirror)
130	151
	152	+ def fetch(self, alias_override=None):
	153	+ self.ensure(alias_override)
	154	+ if not self.has_ref():
	155	+ self._fetch(alias_override)
	156	+ self.assert_ref()
	157	+
131	158	def has_ref(self):
132	159	if not self.ref:
133	160	return False
...	...	@@ -171,13 +198,14 @@ class GitMirror():
171	198
172	199	def init_workspace(self, directory):
173	200	fullpath = os.path.join(directory, self.path)
	201	+ url = self.source.translate_url(self.url)
174	202
175	203	self.source.call([self.source.host_git, 'clone', '--no-checkout', self.mirror, fullpath],
176	204	fail="Failed to clone git mirror {} in directory: {}".format(self.mirror, fullpath),
177	205	fail_temporarily=True)
178	206
179		- self.source.call([self.source.host_git, 'remote', 'set-url', 'origin', self.url],
180		- fail='Failed to add remote origin "{}"'.format(self.url),
	207	+ self.source.call([self.source.host_git, 'remote', 'set-url', 'origin', url],
	208	+ fail='Failed to add remote origin "{}"'.format(url),
181	209	cwd=fullpath)
182	210
183	211	self.source.call([self.source.host_git, 'checkout', '--force', self.ref],
...	...	@@ -277,6 +305,8 @@ class GitSource(Source):
277	305	checkout = self.node_get_member(submodule, bool, 'checkout')
278	306	self.submodule_checkout_overrides[path] = checkout
279	307
	308	+ self.mark_download_url(self.original_url)
	309	+
280	310	def preflight(self):
281	311	# Check if git is installed, get the binary at the same time
282	312	self.host_git = utils.get_host_tool('git')
...	...	@@ -328,31 +358,13 @@ class GitSource(Source):
328	358	.format(self.tracking, self.mirror.url),
329	359	silent_nested=True):
330	360	self.mirror.ensure()
331		- self.mirror.fetch()
	361	+ self.mirror._fetch()
332	362
333	363	# Update self.mirror.ref and node.ref from the self.tracking branch
334	364	ret = self.mirror.latest_commit(self.tracking)
335	365
336	366	return ret
337	367
338		- def fetch(self):
339		-
340		- with self.timed_activity("Fetching {}".format(self.mirror.url), silent_nested=True):
341		-
342		- # Here we are only interested in ensuring that our mirror contains
343		- # the self.mirror.ref commit.
344		- self.mirror.ensure()
345		- if not self.mirror.has_ref():
346		- self.mirror.fetch()
347		-
348		- self.mirror.assert_ref()
349		-
350		- # Here after performing any fetches, we need to also ensure that
351		- # we've cached the desired refs in our mirrors of submodules.
352		- #
353		- self.refresh_submodules()
354		- self.fetch_submodules()
355		-
356	368	def init_workspace(self, directory):
357	369	# XXX: may wish to refactor this as some code dupe with stage()
358	370	self.refresh_submodules()
...	...	@@ -384,6 +396,10 @@ class GitSource(Source):
384	396	if checkout:
385	397	mirror.stage(directory)
386	398
	399	+ def get_source_fetchers(self):
	400	+ self.refresh_submodules()
	401	+ return [self.mirror] + self.submodules
	402	+
387	403	###########################################################
388	404	# Local Functions #
389	405	###########################################################
...	...	@@ -405,6 +421,7 @@ class GitSource(Source):
405	421	# Assumes that we have our mirror and we have the ref which we point to
406	422	#
407	423	def refresh_submodules(self):
	424	+ self.mirror.ensure()
408	425	submodules = []
409	426
410	427	# XXX Here we should issue a warning if either:
...	...	@@ -426,19 +443,6 @@ class GitSource(Source):
426	443
427	444	self.submodules = submodules
428	445
429		- # Ensures that we have mirrored git repositories for all
430		- # the submodules existing at the given commit of the main git source.
431		- #
432		- # Also ensure that these mirrors have the required commits
433		- # referred to at the given commit of the main git source.
434		- #
435		- def fetch_submodules(self):
436		- for mirror in self.submodules:
437		- mirror.ensure()
438		- if not mirror.has_ref():
439		- mirror.fetch()
440		- mirror.assert_ref()
441		-
442	446
443	447	# Plugin entry point
444	448	def setup():

buildstream/source.py

---

...	...	@@ -65,6 +65,33 @@ these methods are mandatory to implement.
65	65
66	66	**Optional**: If left unimplemented, this will default to calling
67	67	:func:`Source.stage() <buildstream.source.Source.stage>`
	68	+
	69	+* :func:`Source.get_source_fetchers() <buildstream.source.Source.get_source_fetchers>`
	70	+
	71	+ Get the objects that are used for fetching.
	72	+
	73	+ **Optional**: This only needs to be implemented for sources that need to
	74	+ download from multiple URLs while fetching (e.g. a git repo and its
	75	+ submodules). For details on how to define a SourceFetcher, see
	76	+ :ref:`SourceFetcher <core_source_fetcher>`.
	77	+
	78	+
	79	+.. _core_source_fetcher:
	80	+
	81	+SourceFetcher - Object for fetching individual URLs
	82	+===================================================
	83	+
	84	+
	85	+Abstract Methods
	86	+----------------
	87	+SourceFetchers expose the following abstract methods. Unless explicitly
	88	+mentioned, these methods are mandatory to implement.
	89	+
	90	+* :func:`SourceFetcher.fetch() <buildstream.source.SourceFetcher.fetch>`
	91	+
	92	+ Fetches the URL associated with this SourceFetcher, optionally taking an
	93	+ alias override.
	94	+
68	95	"""
69	96
70	97	import os
...	...	@@ -114,6 +141,63 @@ class SourceError(BstError):
114	141	super().__init__(message, detail=detail, domain=ErrorDomain.SOURCE, reason=reason, temporary=temporary)
115	142
116	143
	144	+class SourceFetcher():
	145	+ """SourceFetcher()
	146	+
	147	+ This interface exists so that a source that downloads from multiple
	148	+ places (e.g. a git source with submodules) has a consistent interface for
	149	+ fetching and substituting aliases.
	150	+
	151	+ *Since: 1.4*
	152	+ """
	153	+ def __init__(self):
	154	+ self.__alias = None
	155	+
	156	+ #############################################################
	157	+ # Abstract Methods #
	158	+ #############################################################
	159	+ def fetch(self, alias_override=None):
	160	+ """Fetch remote sources and mirror them locally, ensuring at least
	161	+ that the specific reference is cached locally.
	162	+
	163	+ Args:
	164	+ alias_override (str): The alias to use instead of the default one
	165	+ defined by the :ref:`aliases <project_source_aliases>` field
	166	+ in the project's config.
	167	+
	168	+ Raises:
	169	+ :class:`.SourceError`
	170	+
	171	+ Implementors should raise :class:`.SourceError` if the there is some
	172	+ network error or if the source reference could not be matched.
	173	+ """
	174	+ raise ImplError("Source fetcher '{}' does not implement fetch()".format(type(self)))
	175	+
	176	+ #############################################################
	177	+ # Public Methods #
	178	+ #############################################################
	179	+ def mark_download_url(self, url):
	180	+ """Identifies the URL that this SourceFetcher uses to download
	181	+
	182	+ This must be called during the fetcher's initialization
	183	+
	184	+ Args:
	185	+ url (str): The url used to download.
	186	+ """
	187	+ # Not guaranteed to be a valid alias yet.
	188	+ # Ensuring it's a valid alias currently happens in Project.get_alias_uris
	189	+ alias, _ = url.split(utils._ALIAS_SEPARATOR, 1)
	190	+ self.__alias = alias
	191	+
	192	+ #############################################################
	193	+ # Private Methods used in BuildStream #
	194	+ #############################################################
	195	+
	196	+ # Returns the alias used by this fetcher
	197	+ def _get_alias(self):
	198	+ return self.__alias
	199	+
	200	+
117	201	class Source(Plugin):
118	202	"""Source()
119	203
...	...	@@ -125,7 +209,7 @@ class Source(Plugin):
125	209	__defaults = {} # The defaults from the project
126	210	__defaults_set = False # Flag, in case there are not defaults at all
127	211
128		- def __init__(self, context, project, meta):
	212	+ def __init__(self, context, project, meta, *, alias_override=None):
129	213	provenance = _yaml.node_get_provenance(meta.config)
130	214	super().__init__("{}-{}".format(meta.element_name, meta.element_index),
131	215	context, project, provenance, "source")
...	...	@@ -135,6 +219,11 @@ class Source(Plugin):
135	219	self.__element_kind = meta.element_kind # The kind of the element owning this source
136	220	self.__directory = meta.directory # Staging relative directory
137	221	self.__consistency = Consistency.INCONSISTENT # Cached consistency state
	222	+ self.__alias_override = alias_override # Tuple of alias and its override to use instead
	223	+ self.__expected_alias = None # A hacky way to store the first alias used
	224	+
	225	+ # FIXME: Reconstruct a MetaSource from a Source instead of storing it.
	226	+ self.__meta = meta # MetaSource stored so we can copy this source later.
138	227
139	228	# Collect the composited element configuration and
140	229	# ask the element to configure itself.
...	...	@@ -284,6 +373,36 @@ class Source(Plugin):
284	373	"""
285	374	self.stage(directory)
286	375
	376	+ def mark_download_url(self, url):
	377	+ """Identifies the URL that this Source uses to download
	378	+
	379	+ This must be called during :func:`~buildstream.plugin.Plugin.configure` if
	380	+ :func:`~buildstream.source.Source.translate_url` is not called.
	381	+
	382	+ Args:
	383	+ url (str): The url used to download
	384	+
	385	+ *Since: 1.4*
	386	+ """
	387	+ alias, _ = url.split(utils._ALIAS_SEPARATOR, 1)
	388	+ self.__expected_alias = alias
	389	+
	390	+ def get_source_fetchers(self):
	391	+ """Get the objects that are used for fetching
	392	+
	393	+ If this source doesn't download from multiple URLs,
	394	+ returning None and falling back on the default behaviour
	395	+ is recommended.
	396	+
	397	+ Returns:
	398	+ list: A list of SourceFetchers. If SourceFetchers are not supported,
	399	+ this will be an empty list.
	400	+
	401	+ *Since: 1.4*
	402	+ """
	403	+
	404	+ return []
	405	+
287	406	#############################################################
288	407	# Public Methods #
289	408	#############################################################
...	...	@@ -300,18 +419,42 @@ class Source(Plugin):
300	419	os.makedirs(directory, exist_ok=True)
301	420	return directory
302	421
303		- def translate_url(self, url):
	422	+ def translate_url(self, url, *, alias_override=None):
304	423	"""Translates the given url which may be specified with an alias
305	424	into a fully qualified url.
306	425
307	426	Args:
308	427	url (str): A url, which may be using an alias
	428	+ alias_override (str): Optionally, an URI to override the alias with. (*Since: 1.4*)
309	429
310	430	Returns:
311	431	str: The fully qualified url, with aliases resolved
312	432	"""
313		- project = self._get_project()
314		- return project.translate_url(url)
	433	+ # Alias overriding can happen explicitly (by command-line) or
	434	+ # implicitly (the Source being constructed with an __alias_override).
	435	+ if alias_override or self.__alias_override:
	436	+ url_alias, url_body = url.split(utils._ALIAS_SEPARATOR, 1)
	437	+ if url_alias:
	438	+ if alias_override:
	439	+ url = alias_override + url_body
	440	+ else:
	441	+ # Implicit alias overrides may only be done for one
	442	+ # specific alias, so that sources that fetch from multiple
	443	+ # URLs and use different aliases default to only overriding
	444	+ # one alias, rather than getting confused.
	445	+ override_alias = self.__alias_override[0]
	446	+ override_url = self.__alias_override[1]
	447	+ if url_alias == override_alias:
	448	+ url = override_url + url_body
	449	+ return url
	450	+ else:
	451	+ # Sneakily store the alias if it hasn't already been stored
	452	+ if not self.__expected_alias and url and utils._ALIAS_SEPARATOR in url:
	453	+ url_alias, _ = url.split(utils._ALIAS_SEPARATOR, 1)
	454	+ self.__expected_alias = url_alias
	455	+
	456	+ project = self._get_project()
	457	+ return project.translate_url(url)
315	458
316	459	def get_project_directory(self):
317	460	"""Fetch the project base directory
...	...	@@ -375,7 +518,45 @@ class Source(Plugin):
375	518	# Wrapper function around plugin provided fetch method
376	519	#
377	520	def _fetch(self):
378		- self.fetch()
	521	+ project = self._get_project()
	522	+ source_fetchers = self.get_source_fetchers()
	523	+ if source_fetchers:
	524	+ for fetcher in source_fetchers:
	525	+ alias = fetcher._get_alias()
	526	+ success = False
	527	+ for uri in project.get_alias_uris(alias):
	528	+ try:
	529	+ fetcher.fetch(uri)
	530	+ # FIXME: Need to consider temporary vs. permanent failures,
	531	+ # and how this works with retries.
	532	+ except BstError as e:
	533	+ last_error = e
	534	+ continue
	535	+ success = True
	536	+ break
	537	+ if not success:
	538	+ raise last_error
	539	+ else:
	540	+ alias = self._get_alias()
	541	+ if not project.mirrors or not alias:
	542	+ self.fetch()
	543	+ return
	544	+
	545	+ context = self._get_context()
	546	+ source_kind = type(self)
	547	+ for uri in project.get_alias_uris(alias):
	548	+ new_source = source_kind(context, project, self.__meta,
	549	+ alias_override=(alias, uri))
	550	+ new_source._preflight()
	551	+ try:
	552	+ new_source.fetch()
	553	+ # FIXME: Need to consider temporary vs. permanent failures,
	554	+ # and how this works with retries.
	555	+ except BstError as e:
	556	+ last_error = e
	557	+ continue
	558	+ return
	559	+ raise last_error
379	560
380	561	# Wrapper for stage() api which gives the source
381	562	# plugin a fully constructed path considering the
...	...	@@ -582,7 +763,7 @@ class Source(Plugin):
582	763	# Wrapper for track()
583	764	#
584	765	def _track(self):
585		- new_ref = self.track()
	766	+ new_ref = self.__do_track()
586	767	current_ref = self.get_ref()
587	768
588	769	if new_ref is None:
...	...	@@ -594,10 +775,48 @@ class Source(Plugin):
594	775
595	776	return new_ref
596	777
	778	+ # Returns the alias if it's defined in the project
	779	+ def _get_alias(self):
	780	+ alias = self.__expected_alias
	781	+ project = self._get_project()
	782	+ if project.get_alias_uri(alias):
	783	+ # The alias must already be defined in the project's aliases
	784	+ # otherwise http://foo gets treated like it contains an alias
	785	+ return alias
	786	+ else:
	787	+ return None
	788	+
597	789	#############################################################
598	790	# Local Private Methods #
599	791	#############################################################
600	792
	793	+ # Tries to call track for every mirror, stopping once it succeeds
	794	+ def __do_track(self):
	795	+ project = self._get_project()
	796	+ # If there are no mirrors, or no aliases to replace, there's nothing to do here.
	797	+ alias = self._get_alias()
	798	+ if not project.mirrors or not alias:
	799	+ return self.track()
	800	+
	801	+ context = self._get_context()
	802	+ source_kind = type(self)
	803	+
	804	+ # NOTE: We are assuming here that tracking only requires substituting the
	805	+ # first alias used
	806	+ for uri in reversed(project.get_alias_uris(alias)):
	807	+ new_source = source_kind(context, project, self.__meta,
	808	+ alias_override=(alias, uri))
	809	+ new_source._preflight()
	810	+ try:
	811	+ ref = new_source.track()
	812	+ # FIXME: Need to consider temporary vs. permanent failures,
	813	+ # and how this works with retries.
	814	+ except BstError as e:
	815	+ last_error = e
	816	+ continue
	817	+ return ref
	818	+ raise last_error
	819	+
601	820	# Ensures a fully constructed path and returns it
602	821	def __ensure_directory(self, directory):
603	822

buildstream/utils.py

---

...	...	@@ -42,6 +42,10 @@ from . import _signals
42	42	from ._exceptions import BstError, ErrorDomain
43	43
44	44
	45	+# The separator we use for user specified aliases
	46	+_ALIAS_SEPARATOR = ':'
	47	+
	48	+
45	49	class UtilError(BstError):
46	50	"""Raised by utility functions when system calls fail.
47	51
...	...	@@ -608,6 +612,27 @@ def _parse_size(size, volume):
608	612	return int(num) * 1024**units.index(unit)
609	613
610	614
	615	+# _pretty_size()
	616	+#
	617	+# Converts a number of bytes into a string representation in KB, MB, GB, TB
	618	+# represented as K, M, G, T etc.
	619	+#
	620	+# Args:
	621	+# size (int): The size to convert in bytes.
	622	+# dec_places (int): The number of decimal places to output to.
	623	+#
	624	+# Returns:
	625	+# (str): The string representation of the number of bytes in the largest
	626	+def _pretty_size(size, dec_places=0):
	627	+ psize = size
	628	+ unit = 'B'
	629	+ for unit in ('B', 'K', 'M', 'G', 'T'):
	630	+ if psize < 1024:
	631	+ break
	632	+ else:
	633	+ psize /= 1024
	634	+ return "{size:g}{unit}".format(size=round(psize, dec_places), unit=unit)
	635	+
611	636	# A sentinel to be used as a default argument for functions that need
612	637	# to distinguish between a kwarg set to None and an unset kwarg.
613	638	_sentinel = object()

doc/source/examples/git-mirror.rst

---

	1	+
	2	+
	3	+Creating and using a git mirror
	4	+'''''''''''''''''''''''''''''''
	5	+This is an example of how to create a git mirror using git's
	6	+`git-http-backend <https://git-scm.com/docs/git-http-backend>`_ and
	7	+`lighttpd <https://redmine.lighttpd.net/projects/1/wiki/TutorialConfiguration>`_.
	8	+
	9	+
	10	+Prerequisites
	11	+=============
	12	+You will need git installed, and git-http-backend must be present. It is assumed
	13	+that the git-http-backend binary exists at `/usr/lib/git-core/git-http-backend`.
	14	+
	15	+You will need `lighttpd` installed, and at the bare minimum has the modules
	16	+`mod_alias`, `mod_cgi`, and `mod_setenv`.
	17	+
	18	+I will be using gnome-modulesets as an example, which can be cloned from
	19	+`http://gnome7.codethink.co.uk/gnome-modulesets.git`.
	20	+
	21	+
	22	+Starting a git http server
	23	+==========================
	24	+
	25	+
	26	+1. Set up a directory containing mirrors
	27	+----------------------------------------
	28	+Choose a suitable directory to hold your mirrors, e.g. `/var/www/git`.
	29	+
	30	+Place the git repositories you want to use as mirrors in the mirror dir, e.g.
	31	+``git clone --mirror http://git.gnome.org/browse/yelp-xsl /var/www/git/yelp-xsl.git``.
	32	+
	33	+
	34	+2. Configure lighttpd
	35	+---------------------
	36	+Write out a lighttpd.conf as follows:
	37	+
	38	+::
	39	+
	40	+ server.document-root = "/var/www/git/"
	41	+ server.port = 3000
	42	+ server.modules = (
	43	+ "mod_alias",
	44	+ "mod_cgi",
	45	+ "mod_setenv",
	46	+ )
	47	+
	48	+ alias.url += ( "/git" => "/usr/lib/git-core/git-http-backend" )
	49	+ $HTTP["url"] =~ "^/git" {
	50	+ cgi.assign = ("" => "")
	51	+ setenv.add-environment = (
	52	+ "GIT_PROJECT_ROOT" => "/var/www/git",
	53	+ "GIT_HTTP_EXPORT_ALL" => ""
	54	+ )
	55	+ }
	56	+
	57	+.. note::
	58	+
	59	+ If you have your mirrors in another directory, replace /var/www/git/ with that directory.
	60	+
	61	+
	62	+3. Start lighttpd
	63	+-----------------
	64	+lighttpd can be invoked with the command-line ``lighttpd -D -f lighttpd.conf``.
	65	+
	66	+
	67	+4. Test that you can fetch from it
	68	+----------------------------------
	69	+We can then clone the mirrored repo using git via http with
	70	+``git clone http://127.0.0.1:3000/git/yelp-xsl``.
	71	+
	72	+.. note::
	73	+
	74	+ If you have set server.port to something other than the default, you will
	75	+ need to replace the '3000' in the command-line.
	76	+
	77	+
	78	+5. Configure the project to use the mirror
	79	+------------------------------------------
	80	+To add this local http server as a mirror, add the following to the project.conf:
	81	+
	82	+.. code:: yaml
	83	+
	84	+ mirrors:
	85	+ - name: local-mirror
	86	+ aliases:
	87	+ git_gnome_org:
	88	+ - http://127.0.0.1:3000/git/
	89	+
	90	+
	91	+6. Test that the mirror works
	92	+-----------------------------
	93	+We can make buildstream use the mirror by setting the alias to an invalid URL, e.g.
	94	+
	95	+.. code:: yaml
	96	+
	97	+ aliases:
	98	+ git_gnome_org: https://www.example.com/invalid/url/
	99	+
	100	+Now, if you build an element that uses the source you placed in the mirror
	101	+(e.g. ``bst build core-deps/yelp-xsl.bst``), you will see that it uses your mirror.
	102	+
	103	+
	104	+.. _lighttpd_git_tar_conf:
	105	+
	106	+Bonus: lighttpd conf for git and tar
	107	+====================================
	108	+For those who have also used the :ref:`tar-mirror tutorial <using_tar_mirror>`,
	109	+a combined lighttpd.conf is below:
	110	+
	111	+::
	112	+
	113	+ server.document-root = "/var/www/"
	114	+ server.port = 3000
	115	+ server.modules = (
	116	+ "mod_alias",
	117	+ "mod_cgi",
	118	+ "mod_setenv",
	119	+ )
	120	+
	121	+ alias.url += ( "/git" => "/usr/lib/git-core/git-http-backend" )
	122	+ $HTTP["url"] =~ "^/git" {
	123	+ cgi.assign = ("" => "")
	124	+ setenv.add-environment = (
	125	+ "GIT_PROJECT_ROOT" => "/var/www/git",
	126	+ "GIT_HTTP_EXPORT_ALL" => ""
	127	+ )
	128	+ } else $HTTP["url"] =~ "^/tar" {
	129	+ dir-listing.activate = "enable"
	130	+ }
	131	+
	132	+
	133	+Further reading
	134	+===============
	135	+If this mirror isn't being used exclusively in a secure network, it is strongly
	136	+recommended you `use SSL <https://redmine.lighttpd.net/projects/1/wiki/HowToSimpleSSL>`_.
	137	+
	138	+This is the bare minimum required to set up a git mirror. A large, public project
	139	+would prefer to set it up using the
	140	+`git protocol <https://git-scm.com/book/en/v1/Git-on-the-Server-Git-Daemon>`_,
	141	+and a security-conscious project would be configured to use
	142	+`git over SSH <https://git-scm.com/book/en/v1/Git-on-the-Server-Getting-Git-on-a-Server#Small-Setups>`_.
	143	+
	144	+Lighttpd is documented on `its wiki <https://redmine.lighttpd.net/projects/lighttpd/wiki>`_.

doc/source/examples/tar-mirror.rst

---

	1	+
	2	+
	3	+.. _using_tar_mirror:
	4	+
	5	+Creating and using a tar mirror
	6	+'''''''''''''''''''''''''''''''
	7	+This is an example of how to create a tar mirror using
	8	+`lighttpd <https://redmine.lighttpd.net/projects/1/wiki/TutorialConfiguration>`_.
	9	+
	10	+
	11	+Prerequisites
	12	+=============
	13	+You will need `lighttpd` installed.
	14	+
	15	+
	16	+I will be using gnome-modulesets as an example, which can be cloned from
	17	+`http://gnome7.codethink.co.uk/gnome-modulesets.git`.
	18	+
	19	+
	20	+Starting a tar server
	21	+=====================
	22	+
	23	+
	24	+1. Set up a directory containing mirrors
	25	+----------------------------------------
	26	+Choose a suitable directory to hold your mirrored tar files, e.g. `/var/www/tar`.
	27	+
	28	+Place the tar files you want to use as mirrors in your mirror dir, e.g.
	29	+
	30	+.. code::
	31	+
	32	+ mkdir -p /var/www/tar/gettext
	33	+ wget -O /var/www/tar/gettext/gettext-0.19.8.1.tar.xz https://ftp.gnu.org/gnu/gettext/gettext-0.19.8.1.tar.xz
	34	+
	35	+
	36	+2. Configure lighttpd
	37	+---------------------
	38	+Write out a lighttpd.conf as follows:
	39	+
	40	+::
	41	+
	42	+ server.document-root = "/var/www/tar/"
	43	+ server.port = 3000
	44	+
	45	+ dir-listing.activate = "enable"
	46	+
	47	+.. note::
	48	+
	49	+ If you have your mirrors in another directory, replace /var/www/tar/ with that directory.
	50	+
	51	+.. note::
	52	+
	53	+ An example lighttpd.conf that works for both git and tar services is available
	54	+ :ref:`here <lighttpd_git_tar_conf>`
	55	+
	56	+
	57	+3. Start lighttpd
	58	+-----------------
	59	+lighttpd can be invoked with the command-line ``lighttpd -D -f lighttpd.conf``.
	60	+
	61	+
	62	+4. Test that you can fetch from it
	63	+----------------------------------
	64	+We can then download the mirrored file with ``wget 127.0.0.1:3000/tar/gettext/gettext-0.19.8.1.tar.xz``.
	65	+
	66	+.. note::
	67	+
	68	+ If you have set server.port to something other than the default, you will need
	69	+ to replace the '3000' in the command-line.
	70	+
	71	+
	72	+5. Configure the project to use the mirror
	73	+------------------------------------------
	74	+To add this local http server as a mirror, add the following to the project.conf:
	75	+
	76	+.. code:: yaml
	77	+
	78	+ mirrors:
	79	+ - name: local-mirror
	80	+ aliases:
	81	+ ftp_gnu_org:
	82	+ - http://127.0.0.1:3000/tar/
	83	+
	84	+
	85	+6. Test that the mirror works
	86	+-----------------------------
	87	+We can make buildstream use the mirror by setting the alias to an invalid URL, e.g.
	88	+
	89	+.. code:: yaml
	90	+
	91	+ aliases:
	92	+ ftp_gnu_org: https://www.example.com/invalid/url/
	93	+
	94	+Now, if you build an element that uses the source you placed in the mirror
	95	+(e.g. ``bst build core-deps/gettext.bst``), you will see that it uses your mirror.
	96	+
	97	+
	98	+Further reading
	99	+===============
	100	+If this mirror isn't being used exclusively in a secure network, it is strongly
	101	+recommended you `use SSL <https://redmine.lighttpd.net/projects/1/wiki/HowToSimpleSSL>`_.
	102	+
	103	+Lighttpd is documented on `its wiki <https://redmine.lighttpd.net/projects/lighttpd/wiki>`_.

doc/source/format_project.rst

---

...	...	@@ -198,6 +198,43 @@ You can also specify a list of caches here; earlier entries in the list
198	198	will have higher priority than later ones.
199	199
200	200
	201	+.. _project_essentials_mirrors:
	202	+
	203	+Mirrors
	204	+~~~~~~~
	205	+A list of mirrors can be defined that couple a location to a mapping of aliases to a
	206	+list of URIs, e.g.
	207	+
	208	+.. code:: yaml
	209	+
	210	+ mirrors:
	211	+ - name: middle-earth
	212	+ aliases:
	213	+ foo:
	214	+ - http://www.middle-earth.com/foo/1
	215	+ - http://www.middle-earth.com/foo/2
	216	+ bar:
	217	+ - http://www.middle-earth.com/bar/1
	218	+ - http://www.middle-earth.com/bar/2
	219	+ - name: oz
	220	+ aliases:
	221	+ foo:
	222	+ - http://www.oz.com/foo
	223	+ bar:
	224	+ - http://www.oz.com/bar
	225	+
	226	+The order that the mirrors (and the URIs therein) are consulted is in the order
	227	+they are defined when fetching, and in reverse-order when tracking.
	228	+
	229	+A default mirror to consult first can be defined via
	230	+:ref:`user config <config_default_mirror>`, or the command-line argument
	231	+:ref:`--default-mirror <invoking_bst>`.
	232	+
	233	+.. note::
	234	+
	235	+ The ``mirrors`` field is available since :ref:`format version 11 <project_format_version>`
	236	+
	237	+
201	238	.. _project_plugins:
202	239
203	240	External plugins

doc/source/main_install.rst

---

1	1	Install
2	2	=======
3		-This section covers how to install BuildStream onto your machine, how to run BuildStream inside a docker image and also how to configure an artifact server.
	3	+This section covers how to install BuildStream onto your machine, how to run
	4	+BuildStream inside a docker image and also how to configure an artifact server.
4	5
	6	+.. note::
	7	+
	8	+ BuildStream is not currently supported natively on macOS and Windows. Windows
	9	+ and macOS users should refer to :ref:`docker`.
5	10
6	11	.. toctree::
7	12	:maxdepth: 2

doc/source/using_config.rst

---

...	...	@@ -89,6 +89,27 @@ modifying some low level component.
89	89	the ``--strict`` and ``--no-strict`` command line options.
90	90
91	91
	92	+.. _config_default_mirror:
	93	+
	94	+Default Mirror
	95	+~~~~~~~~~~~~~~
	96	+When using :ref:`mirrors <project_essentials_mirrors>`, a default mirror can
	97	+be defined to be fetched first.
	98	+The default mirror is defined by its name, e.g.
	99	+
	100	+.. code:: yaml
	101	+
	102	+ projects:
	103	+ project-name:
	104	+ default-mirror: oz
	105	+
	106	+
	107	+.. note::
	108	+
	109	+ It is possible to override this at invocation time using the
	110	+ ``--default-mirror`` command-line option.
	111	+
	112	+
92	113	Default configuration
93	114	---------------------
94	115	The default BuildStream configuration is specified here for reference:

doc/source/using_examples.rst

---

...	...	@@ -10,3 +10,5 @@ maintained and work as expected.
10	10	:maxdepth: 1
11	11
12	12	examples/flatpak-autotools
	13	+ examples/tar-mirror
	14	+ examples/git-mirror

tests/completions/completions.py

---

...	...	@@ -27,6 +27,7 @@ MAIN_OPTIONS = [
27	27	"--colors ",
28	28	"--config ",
29	29	"--debug ",
	30	+ "--default-mirror ",
30	31	"--directory ",
31	32	"--error-lines ",
32	33	"--fetchers ",

tests/frontend/mirror.py

---

	1	+import os
	2	+import pytest
	3	+
	4	+from tests.testutils import cli, create_repo, ALL_REPO_KINDS
	5	+
	6	+from buildstream import _yaml
	7	+
	8	+
	9	+# Project directory
	10	+TOP_DIR = os.path.dirname(os.path.realpath(__file__))
	11	+DATA_DIR = os.path.join(TOP_DIR, 'project')
	12	+
	13	+
	14	+def generate_element(output_file):
	15	+ element = {
	16	+ 'kind': 'import',
	17	+ 'sources': [
	18	+ {
	19	+ 'kind': 'fetch_source',
	20	+ "output-text": output_file,
	21	+ "urls": ["foo:repo1", "bar:repo2"],
	22	+ "fetch-succeeds": {
	23	+ "FOO/repo1": True,
	24	+ "BAR/repo2": False,
	25	+ "OOF/repo1": False,
	26	+ "RAB/repo2": True,
	27	+ "OFO/repo1": False,
	28	+ "RBA/repo2": False,
	29	+ "ooF/repo1": False,
	30	+ "raB/repo2": False,
	31	+ }
	32	+ }
	33	+ ]
	34	+ }
	35	+ return element
	36	+
	37	+
	38	+def generate_project():
	39	+ project = {
	40	+ 'name': 'test',
	41	+ 'element-path': 'elements',
	42	+ 'aliases': {
	43	+ 'foo': 'FOO/',
	44	+ 'bar': 'BAR/',
	45	+ },
	46	+ 'mirrors': [
	47	+ {
	48	+ 'name': 'middle-earth',
	49	+ 'aliases': {
	50	+ 'foo': ['OOF/'],
	51	+ 'bar': ['RAB/'],
	52	+ },
	53	+ },
	54	+ {
	55	+ 'name': 'arrakis',
	56	+ 'aliases': {
	57	+ 'foo': ['OFO/'],
	58	+ 'bar': ['RBA/'],
	59	+ },
	60	+ },
	61	+ {
	62	+ 'name': 'oz',
	63	+ 'aliases': {
	64	+ 'foo': ['ooF/'],
	65	+ 'bar': ['raB/'],
	66	+ }
	67	+ },
	68	+ ],
	69	+ 'plugins': [
	70	+ {
	71	+ 'origin': 'local',
	72	+ 'path': 'sources',
	73	+ 'sources': {
	74	+ 'fetch_source': 0
	75	+ }
	76	+ }
	77	+ ]
	78	+ }
	79	+ return project
	80	+
	81	+
	82	+@pytest.mark.datafiles(DATA_DIR)
	83	+@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
	84	+def test_mirror_fetch(cli, tmpdir, datafiles, kind):
	85	+ bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
	86	+ dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
	87	+ upstream_repodir = os.path.join(str(tmpdir), 'upstream')
	88	+ mirror_repodir = os.path.join(str(tmpdir), 'mirror')
	89	+ project_dir = os.path.join(str(tmpdir), 'project')
	90	+ os.makedirs(project_dir)
	91	+ element_dir = os.path.join(project_dir, 'elements')
	92	+
	93	+ # Create repo objects of the upstream and mirror
	94	+ upstream_repo = create_repo(kind, upstream_repodir)
	95	+ upstream_ref = upstream_repo.create(bin_files_path)
	96	+ mirror_repo = upstream_repo.copy(mirror_repodir)
	97	+ mirror_ref = upstream_ref
	98	+ upstream_ref = upstream_repo.create(dev_files_path)
	99	+
	100	+ element = {
	101	+ 'kind': 'import',
	102	+ 'sources': [
	103	+ upstream_repo.source_config(ref=upstream_ref)
	104	+ ]
	105	+ }
	106	+ element_name = 'test.bst'
	107	+ element_path = os.path.join(element_dir, element_name)
	108	+ full_repo = element['sources'][0]['url']
	109	+ upstream_map, repo_name = os.path.split(full_repo)
	110	+ alias = 'foo-' + kind
	111	+ aliased_repo = alias + ':' + repo_name
	112	+ element['sources'][0]['url'] = aliased_repo
	113	+ full_mirror = mirror_repo.source_config()['url']
	114	+ mirror_map, _ = os.path.split(full_mirror)
	115	+ os.makedirs(element_dir)
	116	+ _yaml.dump(element, element_path)
	117	+
	118	+ project = {
	119	+ 'name': 'test',
	120	+ 'element-path': 'elements',
	121	+ 'aliases': {
	122	+ alias: upstream_map + "/"
	123	+ },
	124	+ 'mirrors': [
	125	+ {
	126	+ 'name': 'middle-earth',
	127	+ 'aliases': {
	128	+ alias: [mirror_map + "/"],
	129	+ },
	130	+ },
	131	+ ]
	132	+ }
	133	+ project_file = os.path.join(project_dir, 'project.conf')
	134	+ _yaml.dump(project, project_file)
	135	+
	136	+ # No obvious ways of checking that the mirror has been fetched
	137	+ # But at least we can be sure it succeeds
	138	+ result = cli.run(project=project_dir, args=['fetch', element_name])
	139	+ result.assert_success()
	140	+
	141	+
	142	+@pytest.mark.datafiles(DATA_DIR)
	143	+def test_mirror_fetch_multi(cli, tmpdir, datafiles):
	144	+ output_file = os.path.join(str(tmpdir), "output.txt")
	145	+ project_dir = str(tmpdir)
	146	+ element_dir = os.path.join(project_dir, 'elements')
	147	+ os.makedirs(element_dir, exist_ok=True)
	148	+ element_name = "test.bst"
	149	+ element_path = os.path.join(element_dir, element_name)
	150	+ element = generate_element(output_file)
	151	+ _yaml.dump(element, element_path)
	152	+
	153	+ project_file = os.path.join(project_dir, 'project.conf')
	154	+ project = generate_project()
	155	+ _yaml.dump(project, project_file)
	156	+
	157	+ result = cli.run(project=project_dir, args=['fetch', element_name])
	158	+ result.assert_success()
	159	+ with open(output_file) as f:
	160	+ contents = f.read()
	161	+ assert "Fetch foo:repo1 succeeded from FOO/repo1" in contents
	162	+ assert "Fetch bar:repo2 succeeded from RAB/repo2" in contents
	163	+
	164	+
	165	+@pytest.mark.datafiles(DATA_DIR)
	166	+def test_mirror_fetch_default_cmdline(cli, tmpdir, datafiles):
	167	+ output_file = os.path.join(str(tmpdir), "output.txt")
	168	+ project_dir = str(tmpdir)
	169	+ element_dir = os.path.join(project_dir, 'elements')
	170	+ os.makedirs(element_dir, exist_ok=True)
	171	+ element_name = "test.bst"
	172	+ element_path = os.path.join(element_dir, element_name)
	173	+ element = generate_element(output_file)
	174	+ _yaml.dump(element, element_path)
	175	+
	176	+ project_file = os.path.join(project_dir, 'project.conf')
	177	+ project = generate_project()
	178	+ _yaml.dump(project, project_file)
	179	+
	180	+ result = cli.run(project=project_dir, args=['--default-mirror', 'arrakis', 'fetch', element_name])
	181	+ result.assert_success()
	182	+ with open(output_file) as f:
	183	+ contents = f.read()
	184	+ print(contents)
	185	+ # Success if fetching from arrakis' mirror happened before middle-earth's
	186	+ arrakis_str = "OFO/repo1"
	187	+ arrakis_pos = contents.find(arrakis_str)
	188	+ assert arrakis_pos != -1, "'{}' wasn't found".format(arrakis_str)
	189	+ me_str = "OOF/repo1"
	190	+ me_pos = contents.find(me_str)
	191	+ assert me_pos != -1, "'{}' wasn't found".format(me_str)
	192	+ assert arrakis_pos < me_pos, "'{}' wasn't found before '{}'".format(arrakis_str, me_str)
	193	+
	194	+
	195	+@pytest.mark.datafiles(DATA_DIR)
	196	+def test_mirror_fetch_default_userconfig(cli, tmpdir, datafiles):
	197	+ output_file = os.path.join(str(tmpdir), "output.txt")
	198	+ project_dir = str(tmpdir)
	199	+ element_dir = os.path.join(project_dir, 'elements')
	200	+ os.makedirs(element_dir, exist_ok=True)
	201	+ element_name = "test.bst"
	202	+ element_path = os.path.join(element_dir, element_name)
	203	+ element = generate_element(output_file)
	204	+ _yaml.dump(element, element_path)
	205	+
	206	+ project_file = os.path.join(project_dir, 'project.conf')
	207	+ project = generate_project()
	208	+ _yaml.dump(project, project_file)
	209	+
	210	+ userconfig = {
	211	+ 'projects': {
	212	+ 'test': {
	213	+ 'default-mirror': 'oz'
	214	+ }
	215	+ }
	216	+ }
	217	+ cli.configure(userconfig)
	218	+
	219	+ result = cli.run(project=project_dir, args=['fetch', element_name])
	220	+ result.assert_success()
	221	+ with open(output_file) as f:
	222	+ contents = f.read()
	223	+ print(contents)
	224	+ # Success if fetching from Oz' mirror happened before middle-earth's
	225	+ oz_str = "ooF/repo1"
	226	+ oz_pos = contents.find(oz_str)
	227	+ assert oz_pos != -1, "'{}' wasn't found".format(oz_str)
	228	+ me_str = "OOF/repo1"
	229	+ me_pos = contents.find(me_str)
	230	+ assert me_pos != -1, "'{}' wasn't found".format(me_str)
	231	+ assert oz_pos < me_pos, "'{}' wasn't found before '{}'".format(oz_str, me_str)
	232	+
	233	+
	234	+@pytest.mark.datafiles(DATA_DIR)
	235	+def test_mirror_fetch_default_cmdline_overrides_config(cli, tmpdir, datafiles):
	236	+ output_file = os.path.join(str(tmpdir), "output.txt")
	237	+ project_dir = str(tmpdir)
	238	+ element_dir = os.path.join(project_dir, 'elements')
	239	+ os.makedirs(element_dir, exist_ok=True)
	240	+ element_name = "test.bst"
	241	+ element_path = os.path.join(element_dir, element_name)
	242	+ element = generate_element(output_file)
	243	+ _yaml.dump(element, element_path)
	244	+
	245	+ project_file = os.path.join(project_dir, 'project.conf')
	246	+ project = generate_project()
	247	+ _yaml.dump(project, project_file)
	248	+
	249	+ userconfig = {
	250	+ 'projects': {
	251	+ 'test': {
	252	+ 'default-mirror': 'oz'
	253	+ }
	254	+ }
	255	+ }
	256	+ cli.configure(userconfig)
	257	+
	258	+ result = cli.run(project=project_dir, args=['--default-mirror', 'arrakis', 'fetch', element_name])
	259	+ result.assert_success()
	260	+ with open(output_file) as f:
	261	+ contents = f.read()
	262	+ print(contents)
	263	+ # Success if fetching from arrakis' mirror happened before middle-earth's
	264	+ arrakis_str = "OFO/repo1"
	265	+ arrakis_pos = contents.find(arrakis_str)
	266	+ assert arrakis_pos != -1, "'{}' wasn't found".format(arrakis_str)
	267	+ me_str = "OOF/repo1"
	268	+ me_pos = contents.find(me_str)
	269	+ assert me_pos != -1, "'{}' wasn't found".format(me_str)
	270	+ assert arrakis_pos < me_pos, "'{}' wasn't found before '{}'".format(arrakis_str, me_str)
	271	+
	272	+
	273	+@pytest.mark.datafiles(DATA_DIR)
	274	+@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
	275	+def test_mirror_track_upstream_present(cli, tmpdir, datafiles, kind):
	276	+ bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
	277	+ dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
	278	+ upstream_repodir = os.path.join(str(tmpdir), 'upstream')
	279	+ mirror_repodir = os.path.join(str(tmpdir), 'mirror')
	280	+ project_dir = os.path.join(str(tmpdir), 'project')
	281	+ os.makedirs(project_dir)
	282	+ element_dir = os.path.join(project_dir, 'elements')
	283	+
	284	+ # Create repo objects of the upstream and mirror
	285	+ upstream_repo = create_repo(kind, upstream_repodir)
	286	+ upstream_ref = upstream_repo.create(bin_files_path)
	287	+ mirror_repo = upstream_repo.copy(mirror_repodir)
	288	+ mirror_ref = upstream_ref
	289	+ upstream_ref = upstream_repo.create(dev_files_path)
	290	+
	291	+ element = {
	292	+ 'kind': 'import',
	293	+ 'sources': [
	294	+ upstream_repo.source_config(ref=upstream_ref)
	295	+ ]
	296	+ }
	297	+
	298	+ element['sources'][0]
	299	+ element_name = 'test.bst'
	300	+ element_path = os.path.join(element_dir, element_name)
	301	+ full_repo = element['sources'][0]['url']
	302	+ upstream_map, repo_name = os.path.split(full_repo)
	303	+ alias = 'foo-' + kind
	304	+ aliased_repo = alias + ':' + repo_name
	305	+ element['sources'][0]['url'] = aliased_repo
	306	+ full_mirror = mirror_repo.source_config()['url']
	307	+ mirror_map, _ = os.path.split(full_mirror)
	308	+ os.makedirs(element_dir)
	309	+ _yaml.dump(element, element_path)
	310	+
	311	+ project = {
	312	+ 'name': 'test',
	313	+ 'element-path': 'elements',
	314	+ 'aliases': {
	315	+ alias: upstream_map + "/"
	316	+ },
	317	+ 'mirrors': [
	318	+ {
	319	+ 'name': 'middle-earth',
	320	+ 'aliases': {
	321	+ alias: [mirror_map + "/"],
	322	+ },
	323	+ },
	324	+ ]
	325	+ }
	326	+ project_file = os.path.join(project_dir, 'project.conf')
	327	+ _yaml.dump(project, project_file)
	328	+
	329	+ result = cli.run(project=project_dir, args=['track', element_name])
	330	+ result.assert_success()
	331	+
	332	+ # Tracking tries upstream first. Check the ref is from upstream.
	333	+ new_element = _yaml.load(element_path)
	334	+ source = new_element['sources'][0]
	335	+ if 'ref' in source:
	336	+ assert source['ref'] == upstream_ref
	337	+
	338	+
	339	+@pytest.mark.datafiles(DATA_DIR)
	340	+@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
	341	+def test_mirror_track_upstream_absent(cli, tmpdir, datafiles, kind):
	342	+ bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
	343	+ dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
	344	+ upstream_repodir = os.path.join(str(tmpdir), 'upstream')
	345	+ mirror_repodir = os.path.join(str(tmpdir), 'mirror')
	346	+ project_dir = os.path.join(str(tmpdir), 'project')
	347	+ os.makedirs(project_dir)
	348	+ element_dir = os.path.join(project_dir, 'elements')
	349	+
	350	+ # Create repo objects of the upstream and mirror
	351	+ upstream_repo = create_repo(kind, upstream_repodir)
	352	+ upstream_ref = upstream_repo.create(bin_files_path)
	353	+ mirror_repo = upstream_repo.copy(mirror_repodir)
	354	+ mirror_ref = upstream_ref
	355	+ upstream_ref = upstream_repo.create(dev_files_path)
	356	+
	357	+ element = {
	358	+ 'kind': 'import',
	359	+ 'sources': [
	360	+ upstream_repo.source_config(ref=upstream_ref)
	361	+ ]
	362	+ }
	363	+
	364	+ element['sources'][0]
	365	+ element_name = 'test.bst'
	366	+ element_path = os.path.join(element_dir, element_name)
	367	+ full_repo = element['sources'][0]['url']
	368	+ upstream_map, repo_name = os.path.split(full_repo)
	369	+ alias = 'foo-' + kind
	370	+ aliased_repo = alias + ':' + repo_name
	371	+ element['sources'][0]['url'] = aliased_repo
	372	+ full_mirror = mirror_repo.source_config()['url']
	373	+ mirror_map, _ = os.path.split(full_mirror)
	374	+ os.makedirs(element_dir)
	375	+ _yaml.dump(element, element_path)
	376	+
	377	+ project = {
	378	+ 'name': 'test',
	379	+ 'element-path': 'elements',
	380	+ 'aliases': {
	381	+ alias: 'http://www.example.com/'
	382	+ },
	383	+ 'mirrors': [
	384	+ {
	385	+ 'name': 'middle-earth',
	386	+ 'aliases': {
	387	+ alias: [mirror_map + "/"],
	388	+ },
	389	+ },
	390	+ ]
	391	+ }
	392	+ project_file = os.path.join(project_dir, 'project.conf')
	393	+ _yaml.dump(project, project_file)
	394	+
	395	+ result = cli.run(project=project_dir, args=['track', element_name])
	396	+ result.assert_success()
	397	+
	398	+ # Check that tracking fell back to the mirror
	399	+ new_element = _yaml.load(element_path)
	400	+ source = new_element['sources'][0]
	401	+ if 'ref' in source:
	402	+ assert source['ref'] == mirror_ref

tests/frontend/project/sources/fetch_source.py

---

	1	+import os
	2	+import sys
	3	+
	4	+from buildstream import Source, Consistency, SourceError, SourceFetcher
	5	+
	6	+# Expected config
	7	+# sources:
	8	+# - output-text: $FILE
	9	+# urls:
	10	+# - foo:bar
	11	+# - baz:quux
	12	+# fetch-succeeds:
	13	+# Foo/bar: true
	14	+# ooF/bar: false
	15	+
	16	+
	17	+class FetchFetcher(SourceFetcher):
	18	+ def __init__(self, source, url):
	19	+ super().__init__()
	20	+ self.source = source
	21	+ self.original_url = url
	22	+ self.mark_download_url(url)
	23	+
	24	+ def fetch(self, alias_override=None):
	25	+ url = self.source.translate_url(self.original_url, alias_override=alias_override)
	26	+ with open(self.source.output_file, "a") as f:
	27	+ success = url in self.source.fetch_succeeds and self.source.fetch_succeeds[url]
	28	+ message = "Fetch {} {} from {}\n".format(self.original_url,
	29	+ "succeeded" if success else "failed",
	30	+ url)
	31	+ f.write(message)
	32	+ if not success:
	33	+ raise SourceError("Failed to fetch {}".format(url))
	34	+
	35	+
	36	+class FetchSource(Source):
	37	+ # Read config to know which URLs to fetch
	38	+ def configure(self, node):
	39	+ self.original_urls = self.node_get_member(node, list, 'urls')
	40	+ self.fetchers = [FetchFetcher(self, url) for url in self.original_urls]
	41	+ self.output_file = self.node_get_member(node, str, 'output-text')
	42	+ self.fetch_succeeds = {}
	43	+ if 'fetch-succeeds' in node:
	44	+ self.fetch_succeeds = {x[0]: x[1] for x in self.node_items(node['fetch-succeeds'])}
	45	+
	46	+ def get_source_fetchers(self):
	47	+ return self.fetchers
	48	+
	49	+ def preflight(self):
	50	+ output_dir = os.path.dirname(self.output_file)
	51	+ if not os.path.exists(output_dir):
	52	+ raise SourceError("Directory '{}' does not exist".format(output_dir))
	53	+
	54	+ def fetch(self):
	55	+ for fetcher in self.fetchers:
	56	+ fetcher.fetch()
	57	+
	58	+ def get_unique_key(self):
	59	+ return {"urls": self.original_urls, "output_file": self.output_file}
	60	+
	61	+ def get_consistency(self):
	62	+ if not os.path.exists(self.output_file):
	63	+ return Consistency.RESOLVED
	64	+
	65	+ with open(self.output_file, "r") as f:
	66	+ contents = f.read()
	67	+ for url in self.original_urls:
	68	+ if url not in contents:
	69	+ return Consistency.RESOLVED
	70	+
	71	+ return Consistency.CACHED
	72	+
	73	+ # We dont have a ref, we're a local file...
	74	+ def load_ref(self, node):
	75	+ pass
	76	+
	77	+ def get_ref(self):
	78	+ return None # pragma: nocover
	79	+
	80	+ def set_ref(self, ref, node):
	81	+ pass # pragma: nocover
	82	+
	83	+
	84	+def setup():
	85	+ return FetchSource

tests/testutils/repo/repo.py

---

...	...	@@ -22,7 +22,7 @@ class Repo():
22	22	# The directory the actual repo will be stored in
23	23	self.repo = os.path.join(self.directory, subdir)
24	24
25		- os.makedirs(self.repo)
	25	+ os.makedirs(self.repo, exist_ok=True)
26	26
27	27	# create():
28	28	#
...	...	@@ -69,3 +69,22 @@ class Repo():
69	69	shutil.copytree(src_path, dest_path)
70	70	else:
71	71	shutil.copy2(src_path, dest_path)
	72	+
	73	+ # copy():
	74	+ #
	75	+ # Creates a copy of this repository in the specified
	76	+ # destination.
	77	+ #
	78	+ # Args:
	79	+ # dest (str): The destination directory
	80	+ #
	81	+ # Returns:
	82	+ # (Repo): A Repo object for the new repository.
	83	+ def copy(self, dest):
	84	+ subdir = self.repo[len(self.directory):].lstrip(os.sep)
	85	+ new_dir = os.path.join(dest, subdir)
	86	+ os.makedirs(new_dir, exist_ok=True)
	87	+ self.copy_directory(self.repo, new_dir)
	88	+ repo_type = type(self)
	89	+ new_repo = repo_type(dest, subdir)
	90	+ return new_repo

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.