[Notes] [Git][BuildStream/buildstream][valentindavid/script-artifact-corruption] 7 commits: using_config.rst: Update our instructions for how we declare caches in user config

[richardmaw-codethink <gitlab mg gitlab com>]

Title: GitLab

richardmaw-codethink pushed to branch valentindavid/script-artifact-corruption at BuildStream / buildstream

Commits:

• 8e64ccef
  by James Ennis at 2018-11-07T11:06:11Z

  using_config.rst: Update our instructions for how we declare caches in user config
  

• 09ab676d
  by James Ennis at 2018-11-07T11:06:11Z

  format_project.rst: Update instructions regarding how we declare remote caches in project.conf
  

• f514124f
  by James Ennis at 2018-11-07T11:06:11Z

  using_configuring_artifact_server.rst: Remove examples and redirect to user config/project config
  

• 029ba17d
  by richardmaw-codethink at 2018-11-07T11:32:20Z

  Merge branch 'jennis/docs_patch' into 'master'
  
  Update our instructions/examples regarding how we declare remote caches
  
  See merge request BuildStream/buildstream!932

• 261e2cd3
  by Jim MacArthur at 2018-11-07T12:02:43Z

  element.py: Use _force_rmtree instead of custom code.
  
  There are arguments for and against using onerror - see issue #735
  for details.
  

• 8931e42c
  by Jim MacArthur at 2018-11-07T12:26:06Z

  Merge branch 'jmac/make_writable_combination' into 'master'
  
  Use _force_rmtree instead of custom code.
  
  Closes #735
  
  See merge request BuildStream/buildstream!927

• 6ccfab0b
  by Valentin David at 2018-11-07T16:28:47Z

  Fix cache corruption by scripts when layout and integration commands are used
  
  Root directory was marked as a non-artifact mount, so not using
  SafeHardLink.  However integration commands executed with write access
  to the root directory.
  
  Fixes #749
  

10 changed files:

• buildstream/element.py
• buildstream/scriptelement.py
• doc/source/format_project.rst
• doc/source/using_config.rst
• doc/source/using_configuring_artifact_server.rst
• + tests/integration/project/elements/script/corruption-image.bst
• + tests/integration/project/elements/script/corruption-integration.bst
• + tests/integration/project/elements/script/corruption.bst
• + tests/integration/project/files/canary
• tests/integration/script.py

Changes:

buildstream/element.py

---

...	...	@@ -1410,16 +1410,9 @@ class Element(Plugin):
1410	1410
1411	1411	finally:
1412	1412	# Staging may produce directories with less than 'rwx' permissions
1413		- # for the owner, which will break tempfile, so we need to use chmod
1414		- # occasionally.
1415		- def make_dir_writable(fn, path, excinfo):
1416		- os.chmod(os.path.dirname(path), 0o777)
1417		- if os.path.isdir(path):
1418		- os.rmdir(path)
1419		- else:
1420		- os.remove(path)
1421		- shutil.rmtree(temp_staging_directory, onerror=make_dir_writable)
1422		-
	1413	+ # for the owner, which breaks tempfile. _force_rmtree will deal
	1414	+ # with these.
	1415	+ utils._force_rmtree(temp_staging_directory)
1423	1416	# Ensure deterministic mtime of sources at build time
1424	1417	vdirectory.set_deterministic_mtime()
1425	1418	# Ensure deterministic owners of sources at build time

buildstream/scriptelement.py

---

...	...	@@ -201,16 +201,17 @@ class ScriptElement(Element):
201	201	# Setup environment
202	202	sandbox.set_environment(self.get_environment())
203	203
	204	+ # Tell the sandbox to mount the install root
	205	+ directories = {'/': False}
	206	+
204	207	# Mark the artifact directories in the layout
205	208	for item in self.__layout:
206		- if item['destination'] != '/':
207		- if item['element']:
208		- sandbox.mark_directory(item['destination'], artifact=True)
209		- else:
210		- sandbox.mark_directory(item['destination'])
	209	+ destination = item['destination']
	210	+ was_artifact = directories.get(destination, False)
	211	+ directories[destination] = item['element'] or was_artifact
211	212
212		- # Tell the sandbox to mount the install root
213		- sandbox.mark_directory(self.__install_root)
	213	+ for directory, artifact in directories.items():
	214	+ sandbox.mark_directory(directory, artifact=artifact)
214	215
215	216	def stage(self, sandbox):
216	217

doc/source/format_project.rst

---

...	...	@@ -190,19 +190,34 @@ for more detail.
190	190	Artifact server
191	191	~~~~~~~~~~~~~~~
192	192	If you have setup an :ref:`artifact server <artifacts>` for your
193		-project then it is convenient to configure this in your ``project.conf``
	193	+project then it is convenient to configure the following in your ``project.conf``
194	194	so that users need not have any additional configuration to communicate
195	195	with an artifact share.
196	196
197	197	.. code:: yaml
198	198
	199	+ #
	200	+ # Artifacts
	201	+ #
199	202	artifacts:
	203	+ # A remote cache from which to download prebuilt artifacts
	204	+ - url: https://foo.com/artifacts:11001
	205	+ server.cert: server.crt
	206	+ # A remote cache from which to upload/download built/prebuilt artifacts
	207	+ - url: https://foo.com/artifacts:11002
	208	+ server-cert: server.crt
	209	+ client-cert: client.crt
	210	+ client-key: client.key
200	211
201		- # A url from which to download prebuilt artifacts
202		- url: https://foo.com/artifacts
	212	+.. note::
	213	+
	214	+ You can also specify a list of different caches here; earlier entries in the
	215	+ list will have higher priority than later ones.
	216	+
	217	+The use of ports are required to distinguish between pull only access and
	218	+push/pull access. For information regarding the server/client certificates
	219	+and keys, please see: :ref:`Key pair for the server <server_authentication>`.
203	220
204		-You can also specify a list of caches here; earlier entries in the list
205		-will have higher priority than later ones.
206	221
207	222	Remote execution
208	223	~~~~~~~~~~~~~~~~

doc/source/using_config.rst

---

...	...	@@ -32,38 +32,75 @@ the supported configurations on a project wide basis are listed here.
32	32
33	33	Artifact server
34	34	~~~~~~~~~~~~~~~
35		-The project you build will often specify a :ref:`remote artifact cache
36		-<artifacts>` already, but you may want to specify extra caches. There are two
37		-ways to do this. You can add one or more global caches:
	35	+Although project's often specify a :ref:`remote artifact cache <artifacts>` in
	36	+their ``project.conf``, you may also want to specify extra caches.
38	37
39		-**Example**
	38	+Assuming that your host/server is reachable on the internet as ``artifacts.com``
	39	+(for example), there are two ways to declare remote caches in your user
	40	+configuration:
	41	+
	42	+1. Adding global caches:
40	43
41	44	.. code:: yaml
42	45
	46	+ #
	47	+ # Artifacts
	48	+ #
43	49	artifacts:
44		- url: https://artifacts.com/artifacts
	50	+ # Add a cache to pull from
	51	+ - url: https://artifacts.com/artifacts:11001
	52	+ server-cert: server.crt
	53	+ # Add a cache to push/pull to/from
	54	+ - url: https://artifacts.com/artifacts:11002
	55	+ server-cert: server.crt
	56	+ client-cert: client.crt
	57	+ client-key: client.key
	58	+ push: true
	59	+ # Add another cache to pull from
	60	+ - url: https://anothercache.com/artifacts:8080
	61	+ server-cert: another_server.crt
	62	+
	63	+.. note::
45	64
46		-Caches listed there will be considered lower priority than those specified
47		-by the project configuration.
	65	+ Caches declared here will be used by **all** BuildStream project's on the user's
	66	+ machine and are considered a lower priority than those specified in the project
	67	+ configuration.
48	68
49		-You can also add project-specific caches:
50	69
51		-**Example**
	70	+2. Specifying caches for a specific project within the user configuration:
52	71
53	72	.. code:: yaml
54	73
55	74	projects:
56	75	project-name:
57	76	artifacts:
58		- - url: https://artifacts.com/artifacts1
59		- - url: ssh://user artifacts com/artifacts2
	77	+ # Add a cache to pull from
	78	+ - url: https://artifacts.com/artifacts:11001
	79	+ server-cert: server.crt
	80	+ # Add a cache to push/pull to/from
	81	+ - url: https://artifacts.com/artifacts:11002
	82	+ server-cert: server.crt
	83	+ client-cert: client.crt
	84	+ client-key: client.key
60	85	push: true
	86	+ # Add another cache to pull from
	87	+ - url: https://ourprojectcache.com/artifacts:8080
	88	+ server-cert: project_server.crt
	89	+
	90	+
	91	+.. note::
	92	+
	93	+ Caches listed here will be considered a higher priority than those specified
	94	+ by the project. Furthermore, for a given list of URLs, earlier entries will
	95	+ have higher priority.
	96	+
	97	+
	98	+Notice that the use of different ports for the same server distinguishes between
	99	+pull only access and push/pull access. For information regarding this and the
	100	+server/client certificates and keys, please see:
	101	+:ref:`Key pair for the server <server_authentication>`.
61	102
62		-Caches listed here will be considered higher priority than those specified
63		-by the project.
64	103
65		-If you give a list of URLs, earlier entries in the list will have higher
66		-priority than later ones.
67	104
68	105	Strict build plan
69	106	~~~~~~~~~~~~~~~~~

doc/source/using_configuring_artifact_server.rst

---

...	...	@@ -98,6 +98,8 @@ Command reference
98	98	:prog: bst-artifact-server
99	99
100	100
	101	+.. _server_authentication:
	102	+
101	103	Key pair for the server
102	104	~~~~~~~~~~~~~~~~~~~~~~~
103	105
...	...	@@ -237,52 +239,12 @@ We can then check if the services are successfully running with:
237	239	For more information on systemd services see:
238	240	`Creating Systemd Service Files <https://www.devdungeon.com/content/creating-systemd-service-files>`_.
239	241
240		-User configuration
241		-~~~~~~~~~~~~~~~~~~
242		-The user configuration for artifacts is documented with the rest
243		-of the :ref:`user configuration documentation <user_config>`.
244		-
245		-Note that for self-signed certificates, the public key fields are mandatory.
246		-
247		-Assuming you have the same setup used in this document, and that your
248		-host is reachable on the internet as ``artifacts.com`` (for example),
249		-then a user can use the following user configuration:
250		-
251		-Pull-only:
252		-
253		-.. code:: yaml
254		-
255		- #
256		- # Artifacts
257		- #
258		- artifacts:
259		-
260		- url: https://artifacts.com:11001
261		-
262		- # Optional server certificate if not trusted by system root certificates
263		- server-cert: server.crt
	242	+Declaring remote artifact caches
	243	+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	244	+Remote artifact caches can be declared within either:
264	245
265		-Pull and push:
266		-
267		-.. code:: yaml
268		-
269		- #
270		- # Artifacts
271		- #
272		- artifacts:
273		-
274		- url: https://artifacts.com:11002
275		-
276		- # Optional server certificate if not trusted by system root certificates
277		- server-cert: server.crt
278		-
279		- # Optional client key pair for authentication
280		- client-key: client.key
281		- client-cert: client.crt
282		-
283		- push: true
284		-
285		-.. note::
	246	+1. The :ref:`project configuration <project_essentials_artifacts>`, or
	247	+2. The :ref:`user configuration <config_artifacts>`.
286	248
287		- Equivalent statements can be delcared in a project's configuration file
288		- (the ``project.conf``).
	249	+Please follow the above links to see examples showing how we declare remote
	250	+caches in both the project configuration and the user configuration, respectively.

tests/integration/project/elements/script/corruption-image.bst

---

	1	+kind: import
	2	+sources:
	3	+- kind: local
	4	+ path: files/canary

tests/integration/project/elements/script/corruption-integration.bst

---

	1	+kind: stack
	2	+
	3	+public:
	4	+ bst:
	5	+ integration-commands:
	6	+ - echo smashed >>/canary
	7	+

tests/integration/project/elements/script/corruption.bst

---

	1	+kind: script
	2	+
	3	+depends:
	4	+- filename: base.bst
	5	+ type: build
	6	+- filename: script/corruption-image.bst
	7	+ type: build
	8	+- filename: script/corruption-integration.bst
	9	+ type: build
	10	+
	11	+variables:
	12	+ install-root: "/"
	13	+
	14	+config:
	15	+ layout:
	16	+ - element: base.bst
	17	+ destination: "/"
	18	+ - element: script/corruption-image.bst
	19	+ destination: "/"
	20	+ - element: script/corruption-integration.bst
	21	+ destination: "/"

tests/integration/project/files/canary

---

1

+alive

tests/integration/script.py

---

...	...	@@ -155,3 +155,32 @@ def test_script_layout(cli, tmpdir, datafiles):
155	155	text = f.read()
156	156
157	157	assert text == "Hi\n"
	158	+
	159	+
	160	+@pytest.mark.datafiles(DATA_DIR)
	161	+def test_regression_cache_corruption(cli, tmpdir, datafiles):
	162	+ project = str(datafiles)
	163	+ checkout_original = os.path.join(cli.directory, 'checkout-original')
	164	+ checkout_after = os.path.join(cli.directory, 'checkout-after')
	165	+ element_name = 'script/corruption.bst'
	166	+ canary_element_name = 'script/corruption-image.bst'
	167	+
	168	+ res = cli.run(project=project, args=['build', canary_element_name])
	169	+ assert res.exit_code == 0
	170	+
	171	+ res = cli.run(project=project, args=['checkout', canary_element_name,
	172	+ checkout_original])
	173	+ assert res.exit_code == 0
	174	+
	175	+ with open(os.path.join(checkout_original, 'canary')) as f:
	176	+ assert f.read() == 'alive\n'
	177	+
	178	+ res = cli.run(project=project, args=['build', element_name])
	179	+ assert res.exit_code == 0
	180	+
	181	+ res = cli.run(project=project, args=['checkout', canary_element_name,
	182	+ checkout_after])
	183	+ assert res.exit_code == 0
	184	+
	185	+ with open(os.path.join(checkout_after, 'canary')) as f:
	186	+ assert f.read() == 'alive\n'

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.