[issue26259] Memleak when repeated calls to asyncio.queue.Queue.get is performed, without push to queue.

2016-02-01 Thread Jonas Brunsgaard 

New submission from Jonas Brunsgaard:

When making repeated calls to queue.get, memory is building up and is not freed 
until queue.push is called.

I wrote this little program to show my findings. The program will perform a lot 
of calls to queue.get and once every 60 seconds a queue.push is performed. 
Every 15 seconds the memory usage of dictionaries is printet to the console. 
You can find the output below the program

```
import asyncio
from pympler import muppy
from pympler import summary


q = asyncio.Queue()
loop = asyncio.get_event_loop()
closing = False


async def get_with_timeout():
while not closing:
try:
task = asyncio.ensure_future(q.get())
await asyncio.wait_for(task, 0.2)
except asyncio.TimeoutError:
pass

def mem_profiling():
if not closing:
types_ = muppy.filter(muppy.get_objects(), Type=dict)
summary.print_(summary.summarize(types_))
loop.call_later(15, mem_profiling)

def put():
q.put_nowait(None)
loop.call_later(60, put)

put()
tasks = [asyncio.ensure_future(get_with_timeout()) for _ in range(1)]
mem_profiling()

try:
loop.run_forever()
except KeyboardInterrupt:
closing = True
loop.run_until_complete(
asyncio.ensure_future(asyncio.wait(tasks)))
finally:
loop.close()
```

Output:

   types |   # objects |   total size
 | === | 
http://bugs.python.org/file41772/poc.py

___
Python tracker 
<http://bugs.python.org/issue26259>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26259] Memleak when repeated calls to asyncio.queue.Queue.get is performed, without push to queue.

2016-02-01 Thread Jonas Brunsgaard 

Jonas Brunsgaard added the comment:

In my particular case, I developed an application close to beanstalkd, but with 
redis as "engine". I did create a callbackback reader class for users to 
subclass, the callbackreader is checking every second, on every 
tube(queue.Object). If new data has arrived for processing (this subroutine is 
using queue.get with wait_for). Maybe asyncio.Condition would have been the 
better choice. The easy solution was to check if the queue was empty and skip 
the read (get call) if there was nothing in the queue.

Before my fix, over a week the program would take up 10 Gigs of memory in our 
staging environment if nothing was touched, so I was assigned to investigate 
the cause. I think the current behavior is undesirable and cumbersome to see 
through, and if not changed there should be some kind of note in the 
documentation, so other good python folks will have a better chance to 
understand the behavior without reading the cpython asyncio queue 
implementation.

--

___
Python tracker 
<http://bugs.python.org/issue26259>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26259] Memleak when repeated calls to asyncio.queue.Queue.get is performed, without push to queue.

2016-02-02 Thread Jonas Brunsgaard 

Jonas Brunsgaard added the comment:

You are right that get_nowait() is the correct api for my use case, using 
get_nowait() nothing is pushed to the internal _getters deque. The reason for 
my us of get() is that job futures are created one place in the code and then 
thrown in a processing function that will yield the job future. This design was 
created to handle all exceptions in processing(), but I agree that get_nowait 
would have been the superior solution.

I do not have time on my hands right now to take on the challenge of writing a 
patch, but I might take it up later, sound fun ;)

Good day to you sir, and thank you for the feedback.

--

___
Python tracker 
<http://bugs.python.org/issue26259>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26259] Memleak when repeated calls to asyncio.queue.Queue.get is performed, without push to queue.

2016-02-02 Thread Jonas Brunsgaard 

Jonas Brunsgaard added the comment:

Okay I thoroughly read the code again. Can you describe the architectural 
changes to the code regarding a patch, I will do a proposal. But I have to know 
we are on the same page, so I do not waste my and your time :)

--

___
Python tracker 
<http://bugs.python.org/issue26259>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com