On Dec 10, 2010, at 10:51 AM, Thomas Nagy wrote:
--- El vie, 10/12/10, Brian Quinlan escribió:
On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote:
I have a process running for a long time, and which
may use futures of different max_workers count. I think it
is not too far-fetched to create a new futures object each
time. Yet, the execution becomes slower after each call, for
example with http://freehackers.org/~tnagy/futures_test.py:
"""
import concurrent.futures
from queue import Queue
import datetime
class counter(object):
def __init__(self, fut):
self.fut = fut
def run(self):
def
look_busy(num, obj):
tot = 0
for x in range(num):
tot += x
obj.out_q.put(tot)
start =
datetime.datetime.utcnow()
self.count = 0
self.out_q =
Queue(0)
for x in
range(1000):
self.count += 1
self.fut.submit(look_busy, self.count,
self)
while
self.count:
self.count -= 1
self.out_q.get()
delta =
datetime.datetime.utcnow() - start
print(delta.total_seconds())
fut =
concurrent.futures.ThreadPoolExecutor(max_workers=20)
for x in range(100):
# comment the following line
fut =
concurrent.futures.ThreadPoolExecutor(max_workers=20)
c = counter(fut)
c.run()
"""
The runtime grows after each step:
0.216451
0.225186
0.223725
0.222274
0.230964
0.240531
0.24137
0.252393
0.249948
0.257153
...
Is there a mistake in this piece of code?
There is no mistake that I can see but I suspect that the
circular references that you are building are causing the
ThreadPoolExecutor to take a long time to be collected. Try
adding:
c = counter(fut)
c.run()
+ fut.shutdown()
Even if that fixes your problem, I still don't fully
understand this because I would expect the runtime to fall
after a while as ThreadPoolExecutors are collected.
The shutdown call is indeed a good fix :-) Here is the time response
of the calls to counter() when shutdown is not called:
http://www.freehackers.org/~tnagy/runtime_futures.png
FWIW, I think that you are confusion the term "future" with
"executor". A future represents a single work item. An executor
creates futures and schedules their underlying work.
Hmmm....that is very suspicious - it looks like the
ThreadPoolExecutors are not being collected. If you are feeling bored
you could figure out why not :-)
After trying to stop the program by using CTRL+C, the following
error may appear, after which the process cannot be interrupted:
"""
19:18:12 /tmp/build> python3.2 futures_test.py
0.389657
0.417173
0.416513
0.421424
0.449666
0.482273
^CTraceback (most recent call last):
File "futures_test.py", line 36, in <module>
c.run()
File "futures_test.py", line 22, in run
self.fut.submit(look_busy, self.count, self)
File "/usr/local/lib/python3.2/concurrent/futures/thread.py", line
114, in submit
self._work_queue.put(w)
File "/usr/local/lib/python3.2/queue.py", line 135, in put
self.not_full.acquire()
KeyboardInterrupt
"""
It is not expected, is it?
It isn't surprising. Python lock acquisitions are not interruptible
and anytime you interrupt a program that manipulates locks you may
kill the code that was going to cause the lock to be released.
Cheers,
Brian
Thomas
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
