Anders <[email protected]> added the comment:
Note: due to a change in Python 3.8 this example would be a lot less noticeable
if tested. The problem remains the same though.
If you run this snippet with Python 3.7, which is before the thread reuse was
introduced into the ThreadPoolExecutor, each thread will keep around 600mb of
memory in use.
This can be solved by shutting down the ThreadPoolExecutor which this example
does.
Now, the big problem is that asyncio uses a long-running ThreadPoolExecutor,
per default, for run_in_executor. Those threads will stay around forever and
consume memory until the application is shut down.
If you have a job that consumes a lot of memory for a short period of time and
use any long-running ThreadPoolExecutor then the memory will just keep growing
as the job hits various threads that are never cleaned up.
----------
import asyncio
import concurrent
import threading
def prepare_a_giant_list():
d = {}
for i in range(1000):
d[i] = {}
for j in range(1000):
d[i][j] = {}
for k in range(30):
d[i][j][k] = 'a' * 1000
del d
th_num = threading.active_count()
print("Thread number is {}".format(th_num))
async def main():
loop = asyncio.get_running_loop()
with concurrent.futures.ThreadPoolExecutor(max_workers=20) as
async_executor:
await loop.run_in_executor(async_executor, prepare_a_giant_list)
await asyncio.sleep(5)
await loop.run_in_executor(async_executor, prepare_a_giant_list)
await asyncio.sleep(5)
await loop.run_in_executor(async_executor, prepare_a_giant_list)
await asyncio.sleep(5)
await loop.run_in_executor(async_executor, prepare_a_giant_list)
await asyncio.sleep(5)
print('Done!')
await asyncio.sleep(15)
if __name__ == "__main__":
asyncio.run(main())
----------
----------
nosy: +johndoee
versions: +Python 3.6, Python 3.7
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue37909>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
