[issue14119] Ability to adjust queue size in Executors

Antoine Pitrou added the comment: Issue 29595 has a more complete PR, so closing this issue as duplicate. -- nosy: +pitrou resolution: -> duplicate stage: patch review -> resolved status: open -> closed superseder: -> Expose max_queue_size in ThreadPoolExecutor __

[issue14119] Ability to adjust queue size in Executors

Michael Hrivnak added the comment: My project also has a use case for this, very similar to the others described. Here's what we want: with ThreadPoolExecutor(queue_size=500) as executor: for item in parse_a_long_list_of_work(somefile.xml): executor.submit(Job(item)) I do not want to pa

[issue14119] Ability to adjust queue size in Executors

Changes by Camilla Montonen : -- nosy: +Winterflower ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://ma

[issue14119] Ability to adjust queue size in Executors

Vinay Anantharaman added the comment: I did a code reading myself and I noticed that task_done is not called as well. Is there a reason? -- nosy: +Vinay Anantharaman ___ Python tracker

[issue14119] Ability to adjust queue size in Executors

Patrik Dufresne added the comment: Any update on this subject ? Also had to monkey patch the implementation to avoid consuming all the system memory. -- nosy: +Patrik Dufresne ___ Python tracker _

[issue14119] Ability to adjust queue size in Executors

Alexander Mohr added the comment: adding support for internal queue size is critical to avoid chewing through all your memory when you have a LOT of tasks. I just hit this issue myself. If we could have a simple parameter to set the max queue size this would help tremendously! -- no

[issue14119] Ability to adjust queue size in Executors

Victor Varvariuc added the comment: Hi! Looks like your pseudocode will work as a workaround instead of monkey-patching! Still the my suggestion to add the line to code stays. self._count should be always equal to the length of self._work_queue? If yes, why duplication. If no - which one to us

[issue14119] Ability to adjust queue size in Executors

Brian Quinlan added the comment: Can't you accomplish what you want using add_done_callback? e.g. # Pseudocode class MyExecutor(ThreadPoolExecutor): def __init__(self): self._count = 0 def _decrement(self): with self._some_lock: self._count -= 1 def submit(self, fn, *args,

[issue14119] Ability to adjust queue size in Executors

Victor Varvariuc added the comment: Hi Brian, In one my projects I had to monkey-patch module `concurrent.futures.thread:60`( http://hg.python.org/cpython/file/37caaf21f827/Lib/concurrent/futures/thread.py#l60) with: def _worker(executor_reference, work_queue): try: while True:

[issue14119] Ability to adjust queue size in Executors

Brian Quinlan added the comment: Hi Victor, I don't understand your problem. Could you be very specific in your description? -- ___ Python tracker ___ __

[issue14119] Ability to adjust queue size in Executors

Victor Varvariuc added the comment: Maybe I should have created another issue for this, but without this issue being solved, the new issue will not help much. Here it is: http://hg.python.org/cpython/file/37caaf21f827/Lib/concurrent/futures/thread.py#l63 After running an work item `work_queue.ta

[issue14119] Ability to adjust queue size in Executors

Nam Nguyen added the comment: +1 That was actually what I did. I replaced the internal queue with another one whose limit was properly set. If you are busy to write one, let me find some time to create another patch. -- ___ Python tracker

[issue14119] Ability to adjust queue size in Executors

Brian Quinlan added the comment: I've had people request that they be able control the order of processed work submissions. So a more general way to solve your problem might be to make the two executors take an optional Queue argument in their constructors. You'd have to explain in detail in

[issue14119] Ability to adjust queue size in Executors

Nam Nguyen added the comment: I used the ThreadPoolExecutor to process a large number (bounded) of input/output jobs. Because there were too many of them and the worker threads could not process them fast enough to drain them from the queue, the queue kept increasing in size. It was okay for

[issue14119] Ability to adjust queue size in Executors

Brian Quinlan added the comment: The queue that you identified i.e. self._call_queue = multiprocessing.Queue(self._max_workers + EXTRA_QUEUED_CALLS) does not get considered during submit() - are you sure that it somehow causes submit() to block. Could

[issue14119] Ability to adjust queue size in Executors

Nam Nguyen added the comment: Currently, ProcessionPoolExecutor has this line in its constructor: self._call_queue = multiprocessing.Queue(self._max_workers + EXTRA_QUEUED_CALLS) where EXTRA_QUEUED_CALLS is 1. And yes, it would be best to expor

[issue14119] Ability to adjust queue size in Executors

Brian Quinlan added the comment: Hey Nam, I'm not sure that I understand. You want ThreadPoolExecutor.submit to block if there are too many work items in the queue? Are you sure that this happens currently with ProcessPoolExecutor? I can't see why it would. -- __

[issue14119] Ability to adjust queue size in Executors

R. David Murray added the comment: Brian: ping. Since this is an enhancement, if you are going to accept it it would be nice to get it into 3.3, which means committing it before June 23rd. -- nosy: +r.david.murray stage: -> patch review versions: +Python 3.3

[issue14119] Ability to adjust queue size in Executors

Changes by Brian Quinlan : -- assignee: -> bquinlan nosy: +bquinlan ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscr

[issue14119] Ability to adjust queue size in Executors

Changes by Nam Nguyen : -- keywords: +patch Added file: http://bugs.python.org/file24666/executor-queue-size.diff ___ Python tracker ___ _

[issue14119] Ability to adjust queue size in Executors

Nam Nguyen added the comment: By the way, ProcessPoolExecutor actually sets its queue size to a reasonable number but ThreadPoolExecutor does not. -- ___ Python tracker ___ ___

[issue14119] Ability to adjust queue size in Executors

New submission from Nam Nguyen : I am running into a memory consumption issue with concurrent.futures module. Its Executors do not have a public API to adjust their queue size. and the queues are created unbounded initially. It would be helpful to have some public method or a parameter at con