[Notes] [Git][BuildStream/buildstream][bschubert/pipeline] 6 commits: .gitlab-ci.yml: Add tests for python 3.7

[Benjamin Schubert <gitlab mg gitlab com>]

Title: GitLab

Benjamin Schubert pushed to branch bschubert/pipeline at BuildStream / buildstream

Commits:

• 4b544555
  by Chandan Singh at 2019-01-15T21:28:40Z

  .gitlab-ci.yml: Add tests for python 3.7
  
  We already have tests for python 3.5 and 3.6 but not 3.7.
  
  Fixes https://gitlab.com/BuildStream/buildstream/issues/838.
  

• 80fe0d9a
  by Javier Jardón at 2019-01-15T22:48:22Z

  Merge branch 'chandan/python37-tests' into 'master'
  
  .gitlab-ci.yml: Add tests for python 3.7
  
  Closes #838
  
  See merge request BuildStream/buildstream!1074

• c91784ab
  by Chandan Singh at 2019-01-15T22:49:01Z

  conftest.py: Don't use deprecated get_marker() function
  
  Starting from `pytest` version 4.1.0, `Node.get_marker()` has been
  removed, and hence our tests break when running with newer versions of
  `pytest`. It was deprecated since a while back but it has recently been
  removed completely. Use `get_closest_marker()` as a replacement that is
  suggested in the changelog, and seems to work fine for our use case.
  
  See https://github.com/pytest-dev/pytest/pull/4564 for more context on
  the upstream issue.
  
  One way of verifying this change is that this should fix the recently
  added `tests-fedora-update-deps` job, that was failing before due to
  this issue.
  

• ecae4d73
  by Javier Jardón at 2019-01-15T23:23:41Z

  Merge branch 'chandan/fix-pytest-get-marker' into 'master'
  
  conftest.py: Don't use deprecated get_marker() function
  
  See merge request BuildStream/buildstream!1073

• 7ad5fa8d
  by Benjamin Schubert at 2019-01-16T09:29:59Z

  Stop updating state in queue.status()
  
  Statuses of an element can be changed when:
  
  1) It is pulled
  2) It is fetched
  3) It is workspaced and it finished building
  4) One of its dependencies is tracked
  5) One of its dependencies is workspaced and finished building
  
  We can therefore update the statuses at those moments and we don't
  need to check all the time.
  
  This reduces considerably the calls to update_states that are done
  

• c2a10bc8
  by Benjamin Schubert at 2019-01-16T09:29:59Z

  Don't update state before summary. It should be consistent
  
  Since we know when an object's state can change, we don't need
  to update the state of all objects for consistency.
  

7 changed files:

• .gitlab-ci.yml
• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/queues/fetchqueue.py
• buildstream/_scheduler/queues/pullqueue.py
• buildstream/_stream.py
• buildstream/element.py
• conftest.py

Changes:

.gitlab-ci.yml

---

...	...	@@ -60,6 +60,16 @@ tests-ubuntu-18.04:
60	60	image: buildstream/testsuite-ubuntu:18.04-5da27168-32c47d1c
61	61	<<: *tests
62	62
	63	+tests-python-3.7-stretch:
	64	+ image: buildstream/testsuite-python:3.7-stretch-a60f0c39
	65	+ <<: *tests
	66	+
	67	+ variables:
	68	+ # Note that we explicitly specify TOXENV in this case because this
	69	+ # image has both 3.6 and 3.7 versions. python3.6 cannot be removed because
	70	+ # some of our base dependencies declare it as their runtime dependency.
	71	+ TOXENV: py37
	72	+
63	73	overnight-fedora-28-aarch64:
64	74	image: buildstream/testsuite-fedora:aarch64-28-5da27168-32c47d1c
65	75	tags:

buildstream/_scheduler/queues/buildqueue.py

---

...	...	@@ -71,9 +71,6 @@ class BuildQueue(Queue):
71	71	return element._assemble()
72	72
73	73	def status(self, element):
74		- # state of dependencies may have changed, recalculate element state
75		- element._update_state()
76		-
77	74	if not element._is_required():
78	75	# Artifact is not currently required but it may be requested later.
79	76	# Keep it in the queue.

buildstream/_scheduler/queues/fetchqueue.py

---

...	...	@@ -44,9 +44,6 @@ class FetchQueue(Queue):
44	44	element._fetch()
45	45
46	46	def status(self, element):
47		- # state of dependencies may have changed, recalculate element state
48		- element._update_state()
49		-
50	47	if not element._is_required():
51	48	# Artifact is not currently required but it may be requested later.
52	49	# Keep it in the queue.

buildstream/_scheduler/queues/pullqueue.py

---

...	...	@@ -39,9 +39,6 @@ class PullQueue(Queue):
39	39	raise SkipJob(self.action_name)
40	40
41	41	def status(self, element):
42		- # state of dependencies may have changed, recalculate element state
43		- element._update_state()
44		-
45	42	if not element._is_required():
46	43	# Artifact is not currently required but it may be requested later.
47	44	# Keep it in the queue.

buildstream/_stream.py

---

...	...	@@ -28,7 +28,7 @@ import tarfile
28	28	import tempfile
29	29	from contextlib import contextmanager, suppress
30	30
31		-from ._exceptions import StreamError, ImplError, BstError, set_last_task_error
	31	+from ._exceptions import StreamError, ImplError, BstError
32	32	from ._message import Message, MessageType
33	33	from ._scheduler import Scheduler, SchedStatus, TrackQueue, FetchQueue, BuildQueue, PullQueue, PushQueue
34	34	from ._pipeline import Pipeline, PipelineSelection
...	...	@@ -1018,17 +1018,6 @@ class Stream():
1018	1018
1019	1019	_, status = self._scheduler.run(self.queues)
1020	1020
1021		- # Force update element states after a run, such that the summary
1022		- # is more coherent
1023		- try:
1024		- for element in self.total_elements:
1025		- element._update_state()
1026		- except BstError as e:
1027		- self._message(MessageType.ERROR, "Error resolving final state", detail=str(e))
1028		- set_last_task_error(e.domain, e.reason)
1029		- except Exception as e: # pylint: disable=broad-except
1030		- self._message(MessageType.BUG, "Unhandled exception while resolving final state", detail=str(e))
1031		-
1032	1021	if status == SchedStatus.ERROR:
1033	1022	raise StreamError()
1034	1023	elif status == SchedStatus.TERMINATED:

buildstream/element.py

---

...	...	@@ -197,6 +197,7 @@ class Element(Plugin):
197	197
198	198	self.__runtime_dependencies = [] # Direct runtime dependency Elements
199	199	self.__build_dependencies = [] # Direct build dependency Elements
	200	+ self.__reverse_build_dependencies = [] # Direct reverse dependency Elements
200	201	self.__sources = [] # List of Sources
201	202	self.__weak_cache_key = None # Our cached weak cache key
202	203	self.__strict_cache_key = None # Our cached cache key for strict builds
...	...	@@ -439,6 +440,21 @@ class Element(Plugin):
439	440	if should_yield and (recurse or recursed) and scope != Scope.BUILD:
440	441	yield self
441	442
	443	+ def reverse_build_dependencies(self):
	444	+ """Get all reverse dependencies for the given element in a topological order.
	445	+
	446	+ Currently, dependencies might be returned multiple times.
	447	+ """
	448	+ # FIXME: we should return each entry only once in topological order
	449	+ def recurse_rdeps(element):
	450	+ yield element
	451	+
	452	+ for rdep in element.__reverse_build_dependencies:
	453	+ yield from recurse_rdeps(rdep)
	454	+
	455	+ for rdep in self.__reverse_build_dependencies:
	456	+ yield from recurse_rdeps(rdep)
	457	+
442	458	def search(self, scope, name):
443	459	"""Search for a dependency by name
444	460
...	...	@@ -930,6 +946,7 @@ class Element(Plugin):
930	946	for meta_dep in meta.build_dependencies:
931	947	dependency = Element._new_from_meta(meta_dep)
932	948	element.__build_dependencies.append(dependency)
	949	+ dependency.__reverse_build_dependencies.append(element)
933	950
934	951	return element
935	952
...	...	@@ -1306,6 +1323,9 @@ class Element(Plugin):
1306	1323
1307	1324	self._update_state()
1308	1325
	1326	+ for reverse_dep in self.reverse_build_dependencies():
	1327	+ reverse_dep._update_state()
	1328	+
1309	1329	# _track():
1310	1330	#
1311	1331	# Calls track() on the Element sources
...	...	@@ -1503,6 +1523,9 @@ class Element(Plugin):
1503	1523	self._update_state()
1504	1524
1505	1525	if self._get_workspace() and self._cached_success():
	1526	+ for rdep in self.reverse_build_dependencies():
	1527	+ rdep._update_state()
	1528	+
1506	1529	assert utils._is_main_process(), \
1507	1530	"Attempted to save workspace configuration from child process"
1508	1531	#

conftest.py

---

...	...	@@ -32,7 +32,7 @@ def pytest_addoption(parser):
32	32
33	33
34	34	def pytest_runtest_setup(item):
35		- if item.get_marker('integration') and not item.config.getvalue('integration'):
	35	+ if item.get_closest_marker('integration') and not item.config.getvalue('integration'):
36	36	pytest.skip('skipping integration test')
37	37
38	38

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.