[issue41699] Potential memory leak with asyncio and run_in_executor

[Sophia Wisdom]

New submission from Sophia Wisdom :

The below example leaks ~20 megabytes of memory. The amount leaked is related 
to both the number of items in the list and the number of times 
`run_in_executor` is called.

```
import asyncio

def leaker():
x = list(range(int(1000)))
1/0

async def function():
loop = asyncio.get_running_loop()
for i in range(1):
loop.run_in_executor(None, leaker)
```
at this point, `ps -o rss -p {pid}` outputs about 10MB

after invoking this:
```
asyncio.run(function())
```
Memory jumps to about 94MB, and doesn't return.

The lists don't show up in `gc.get_objects()`, but the amount of memory leaked 
does increase (though not proportionately) when the lists increase in size.

Note - this doesn't happen if `run_in_executor` is `await`ed immediately, but 
it does still occur if we e.g. put the future in a dictionary and then `await` 
the results later.
The leak still occurs on my machine if the `1/0` is omitted, but not on a 
colleague's.

We're pretty confused as to why this happens, and would appreciate any help.

--
components: asyncio
messages: 376275
nosy: asvetlov, sophia2, yselivanov
priority: normal
severity: normal
status: open
title: Potential memory leak with asyncio and run_in_executor
versions: Python 3.10, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue41699>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41699] Potential memory leak with asyncio and run_in_executor

[Sophia Wisdom]

Sophia Wisdom  added the comment:

It looks like it's not specific to the ThreadPoolExecutor.

```
import asyncio
import concurrent

def leaker_func():
list(range(int(1000)))
# removed 1/0 because this causes issues with the ProcessPoolExecutor

async def function():
loop = asyncio.get_running_loop()
for i in range(1):
loop.run_in_executor(concurrent.futures.ProcessPoolExecutor(), 
leaker_func)
```
10MB at this point

then after executing this:
```
asyncio.run(function())
```
40MB. (~same as ThreadPoolExecutor in python3.10)

--

___
Python tracker 
<https://bugs.python.org/issue41699>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41699] Potential memory leak with asyncio and run_in_executor

[Sophia Wisdom]

Sophia Wisdom  added the comment:

While not calling executor.shutdown() may leave some resources still used, it 
should be small and fixed. Regularly calling executor.shutdown() and then 
instantiating a new ThreadPoolExecutor in order to run an asyncio program does 
not seem like a good API to me.

You mention there appear to be both an event loop and a futures leak -- I think 
I have a good test case for the futures, without using threads at all. This 
seems to be leaking `future._result`s somehow even though their __del__ is 
called.

```
import asyncio
from concurrent.futures import Executor, Future
import gc

result_gcs = 0
suture_gcs = 0

class ResultHolder:
def __init__(self, mem_size):
self.mem = list(range(mem_size)) # so we can see the leak

def __del__(self):
global result_gcs
result_gc += 1

class Suture(Future):
   def __del__(self):
   global suture_gcs
   suture_gcs += 1

class SimpleExecutor(Executor):
def submit(self, fn):
future = Suture()
future.set_result(ResultHolder(1000))
return future

async def function():
loop = asyncio.get_running_loop()
for i in range(1):
loop.run_in_executor(SimpleExecutor(), lambda x:x)

def run():
asyncio.run(function())
print(suture_gcs, result_gcs)
```
10MB

```
> run()
1 1
```
100MB

Both result_gcs and suture_gcs are 1 every time. My best guess for why this 
would happen (for me it doesn't seem to happen without the 
loop.run_in_executor) is the conversion from a concurrent.Future to an 
asyncio.Future, which involves callbacks to check on the result, but that 
doesn't make sense, because the result itself has __del__ called on it but 
somehow it doesn't free the memory!

--

___
Python tracker 
<https://bugs.python.org/issue41699>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com