[issue21423] concurrent.futures.ThreadPoolExecutor should accept an initializer argument
New submission from Andreas van Cranenburgh: It would be useful if concurrent.futures.ThreadPoolExecutor took an initializer argument, like multiprocessing.Pool. This is useful for example to load a large dataset once upon initialization of each worker process, without have to pass the dataset as an argument with every job submission, which requires serialization. concurrent.futures has some advantages over multiprocessing such as detecting killed processes ( http://bugs.python.org/issue9205 ), so it would be good if the advantages of both can be combined. It appears that the following fork of concurrent.futures has added these arguments: https://github.com/enthought/encore/blob/7101984bc384da8e7975876ca2809cc0103c3efc/encore/concurrent/futures/enhanced_thread_pool_executor.py -- components: Library (Lib) messages: 217846 nosy: andreasvc priority: normal severity: normal status: open title: concurrent.futures.ThreadPoolExecutor should accept an initializer argument versions: Python 3.5 ___ Python tracker <http://bugs.python.org/issue21423> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21423] concurrent.futures.ThreadPoolExecutor should accept an initializer argument
Andreas van Cranenburgh added the comment: Yes I did mean ProcessPoolExecutor, but indeed, it's good to have for threads as well. I could try to make a patch if it is likely that it would be accepted. -- ___ Python tracker <http://bugs.python.org/issue21423> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21423] concurrent.futures.ThreadPoolExecutor should accept an initializer argument
Andreas van Cranenburgh added the comment: Here's a patch. I have added initializer and initargs keywords to both ThreadPoolExecutor and ProcessPoolExecutor, with the same semantics as multiprocessing.Pool. I couldn't figure out what to do if the initializer fails with a ProcessPoolExecutor: how to properly send the traceback back? I also haven't gotten around to figure out how to write tests. I haven't added unitializers, don't know if they would be useful. -- keywords: +patch Added file: http://bugs.python.org/file35179/pool_initializer.patch ___ Python tracker <http://bugs.python.org/issue21423> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21423] concurrent.futures.ThreadPoolExecutor should accept an initializer argument
Andreas van Cranenburgh added the comment: Here's a version with tests. Detecting an execption in the initializer works with ProcessPoolExecutor, but not with ThreadPoolExecutor. -- Added file: http://bugs.python.org/file35181/pool_initializer_tests.patch ___ Python tracker <http://bugs.python.org/issue21423> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21423] concurrent.futures.ThreadPoolExecutor should accept an initializer argument
Andreas van Cranenburgh added the comment: Giampaolo, this patch is for ProcessPoolExecutor as well. About keyboard interrupts, if my tests are correct, they work in Python 3.3+ with both multiprocessing and concurrent.futures. (Although for the latter I have to hit ctrl-c twice). -- ___ Python tracker <http://bugs.python.org/issue21423> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
