Raymond Hettinger wrote:
> FWIW, I've been working on a way to simplify the use of queues with daemon
> consumer threads
>
> Sometimes, I launch one or more consumer threads that wait for a task to
> enter a
> queue and then work on the task. A recurring problem is that I sometimes need
> to
> know if all of the tasks have been completed so I can exit or do something
> with
> the result.
>
> If each thread only does a single task, I can use t.join() to wait until the
> task is done. However, if the thread stays alive and waits for more Queue
> entries, then there doesn't seem to be a good way to tell when all the
> processing is done.
The pattern I use is to use the None object to indicate that no more tasks are
coming - just add it to your queue once for every worker thread that exists and
have your worker threads exit when they get a task of None.
Your setup code to start the threads and add tasks to the queue wouldn't need
to
change; once all the tasks have been enqueued you'd wait for completion like
this:
# Tell all the workers to quit
for t in worker_threads():
q.put(None)
# Wait for all the workers to quit
for t in worker_threads():
t.join()
do_work_on_results()
The worker thread looks like this:
while 1:
task = q.get()
if task is None:
break
do_work(task)
-Dave
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
