[Notes] [Git][BuildStream/buildstream][tristan/fix-cache-exclusivity-1.2] 3 commits: _scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE

[Tristan Van Berkom <gitlab mg gitlab com>]

Title: GitLab

Tristan Van Berkom pushed to branch tristan/fix-cache-exclusivity-1.2 at BuildStream / buildstream

Commits:

• b8ade1b0
  by Tristan Van Berkom at 2018-09-05T12:21:20Z

  _scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE
  
  This is what the whole resource.py thing was created for, the
  cleanup job must have exclusive access to the cache, while the pull
  and build jobs which result in adding new artifacts, must only require
  shared access.
  
  This is a part of #623
  

• 158aca86
  by Tristan Van Berkom at 2018-09-05T12:21:22Z

  _scheduler/resources.py: Set CACHE max resources to 0
  
  This means there is no cap for shared resource requests.
  
  Together with the previous commit, this causes the cleanup
  job and the pull/build jobs to all require unlimited shared
  access to the CACHE resource; while only the cleanup job
  requires exclusive access to the resource.
  
  This is a part of #623
  

• 65eb56b3
  by Tristan Van Berkom at 2018-09-05T12:21:25Z

  _scheduler/scheduler.py: Make CacheSizeJob() not require the CACHE exclusively
  
  This runs after every pull, and does not need the cache exclusively,
  only the cleanup job requires the cache exclusively.
  
  Without this, every time a cache_size job is queued, all pull and
  build jobs need to complete before cache_size job can run exclusively,
  which is not good.
  
  This is a part of #623
  

4 changed files:

• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/queues/pullqueue.py
• buildstream/_scheduler/resources.py
• buildstream/_scheduler/scheduler.py

Changes:

buildstream/_scheduler/queues/buildqueue.py

---

...	...	@@ -28,7 +28,7 @@ class BuildQueue(Queue):
28	28
29	29	action_name = "Build"
30	30	complete_name = "Built"
31		- resources = [ResourceType.PROCESS]
	31	+ resources = [ResourceType.PROCESS, ResourceType.CACHE]
32	32
33	33	def process(self, element):
34	34	element._assemble()

buildstream/_scheduler/queues/pullqueue.py

---

...	...	@@ -29,7 +29,7 @@ class PullQueue(Queue):
29	29
30	30	action_name = "Pull"
31	31	complete_name = "Pulled"
32		- resources = [ResourceType.DOWNLOAD]
	32	+ resources = [ResourceType.DOWNLOAD, ResourceType.CACHE]
33	33
34	34	def process(self, element):
35	35	# returns whether an artifact was downloaded or not

buildstream/_scheduler/resources.py

---

...	...	@@ -8,7 +8,7 @@ class ResourceType():
8	8	class Resources():
9	9	def __init__(self, num_builders, num_fetchers, num_pushers):
10	10	self._max_resources = {
11		- ResourceType.CACHE: 1,
	11	+ ResourceType.CACHE: 0,
12	12	ResourceType.DOWNLOAD: num_fetchers,
13	13	ResourceType.PROCESS: num_builders,
14	14	ResourceType.UPLOAD: num_pushers

buildstream/_scheduler/scheduler.py

---

...	...	@@ -330,7 +330,6 @@ class Scheduler():
330	330	job = CacheSizeJob(self, 'cache_size', 'cache_size/cache_size',
331	331	resources=[ResourceType.CACHE,
332	332	ResourceType.PROCESS],
333		- exclusive_resources=[ResourceType.CACHE],
334	333	complete_cb=self._run_cleanup)
335	334	self.schedule_jobs([job])
336	335

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.