Carlo Rosati <[email protected]> added the comment:
I figured out that the problem is itertools.tee does not use a
multiprocessing.Manager proxied object for shared state. I was able to create a
workaround tee as follows.
def multiprocessing_tee(iterable, n=2):
"""Write a multiprocessing safe itertools.tee"""
it = iter(iterable)
m = multiprocessing.Manager()
lists = [m.list() for i in range(n)]
def gen(local_list):
keep_m_alive = m
while True:
if not local_list: # when the local list is empty
newval = next(it) # fetch a new value and
for l in lists: # load it to all the lists
l.append(newval)
yield local_list.pop(-1)
return tuple(gen(l) for l in lists)
----------
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue34410>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
