[Notes] [Git][BuildStream/buildstream][danielsilverstone-ct/maybe-reduce-fd-leaks] jobs.py: Reduce FD leaks from queues and process objects

[Daniel <gitlab mg gitlab com>]

Title: GitLab

Daniel pushed to branch danielsilverstone-ct/maybe-reduce-fd-leaks at BuildStream / buildstream

Commits:

• 0c24eab7
  by Daniel Silverstone at 2018-09-04T16:50:26Z

  jobs.py: Reduce FD leaks from queues and process objects
  
  The garbage collector can take too long to get around to cleaning
  up the Queue and Process instances in completed Job instances.  As
  such, FDs tend to leak and in very large projects this can result
  in running out of FDs before a build, fetch, track, or other process
  can complete.  This patch reduces the chance of that by only creating
  the queue when it's needed, and forcing the queue and process instances
  to be deleted when the parent is finished with them.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

1 changed file:

• buildstream/_scheduler/jobs/job.py

Changes:

buildstream/_scheduler/jobs/job.py

---

...	...	@@ -109,7 +109,7 @@ class Job():
109	109	# Private members
110	110	#
111	111	self._scheduler = scheduler # The scheduler
112		- self._queue = multiprocessing.Queue() # A message passing queue
	112	+ self._queue = None # A message passing queue
113	113	self._process = None # The Process object
114	114	self._watcher = None # Child process watcher
115	115	self._listening = False # Whether the parent is currently listening
...	...	@@ -130,6 +130,8 @@ class Job():
130	130	#
131	131	def spawn(self):
132	132
	133	+ self._queue = multiprocessing.Queue()
	134	+
133	135	self._tries += 1
134	136	self._parent_start_listening()
135	137
...	...	@@ -552,6 +554,9 @@ class Job():
552	554	self.parent_complete(returncode == RC_OK, self._result)
553	555	self._scheduler.job_completed(self, returncode == RC_OK)
554	556
	557	+ # Force the deletion of the queue and process objects to try and clean up FDs
	558	+ self._queue = self._process = None
	559	+
555	560	# _parent_process_envelope()
556	561	#
557	562	# Processes a message Envelope deserialized form the message queue.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.