[issue11849] glibc allocator doesn't release all free()ed memory

Bob Kline added the comment: > ... jemalloc can reduce memory usage ... Thanks for the tip. I downloaded the source and successfully built the DLL, then went looking for a way to get it loaded. Unfortunately, DLL injection, which is needed to use this allocator in Python, seems to be much bet

[issue11849] glibc allocator doesn't release all free()ed memory

INADA Naoki added the comment: FYI, jemalloc can reduce memory usage, especially when application is multithreaded. https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html https://zapier.com/engineering/celery-python-jemalloc/ -- nosy: +inada.naoki

[issue11849] glibc allocator doesn't release all free()ed memory

Bob Kline added the comment: Thanks for your responses to my comments. I'm working as hard as I can to get my customer's systems migrated into the Python 3 world, and I appreciate the efforts of the community to provide incentives (such as the resolution for this failure) for developers to up

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou added the comment: Well, memory fragmentation can happen with any allocation scheme, and it's possible even Python 3 isn't immune to this. Backporting performance improvements is a strain on our resources and also constitutes a maintenance threat (what if the bug hides in the

[issue11849] glibc allocator doesn't release all free()ed memory

Bob Kline added the comment: Sorry, I should have used the language of the patch author ("the resolution"). Without the resolution, Python 2.7 eventually runs out of memory and crashes for some correctly written user code. -- ___ Python tracker

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou added the comment: It's not really a fix, it's an improvement, and as such doesn't belong in 2.7. Using malloc() and free() is not a bug in itself. -- ___ Python tracker _

[issue11849] glibc allocator doesn't release all free()ed memory

Bob Kline added the comment: Would it be inappropriate for this fix to be applied to 2.7? -- nosy: +bkline ___ Python tracker ___ __

[issue11849] glibc allocator doesn't release all free()ed memory

STINNER Victor added the comment: Extract of the "workaround" section: "You could also run your Python jobs using Jython, which uses the Java JVM and does not exhibit this behavior. Likewise, you could upgrade to Python 3.3 ," Which contains a link to this issu

[issue11849] glibc allocator doesn't release all free()ed memory

Tim Peters added the comment: [@haypo] > http://python.dzone.com/articles/diagnosing-memory-leaks-python > Great job! Using mmap() for arenas is the best solution for this issue. ? I read the article, and they stopped when they found "there seemed to be a ton of tiny little objects around, lik

[issue11849] glibc allocator doesn't release all free()ed memory

STINNER Victor added the comment: I just found this issue from this article: http://python.dzone.com/articles/diagnosing-memory-leaks-python Great job! Using mmap() for arenas is the best solution for this issue. I did something similar on a completly different project (also using its own dedi

[issue11849] glibc allocator doesn't release all free()ed memory

Roundup Robot added the comment: New changeset e7aa72e6aad4 by Antoine Pitrou in branch 'default': Better resolution for issue #11849: Ensure that free()d memory arenas are really released http://hg.python.org/cpython/rev/e7aa72e6aad4 -- ___ Python

[issue11849] glibc allocator doesn't release all free()ed memory

Charles-François Natali added the comment: > Hmm, quite slow indeed, are you sure you're not running in debug mode? > Well, yes, but it's no faster with a non-debug build: my laptop is really crawling :-) > If the performance regression is limited to read(), I don't think it's > really an issu

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou added the comment: > On my box: > default: > $ ./python -m timeit -s "n=30; f=open('/tmp/10MB.bin', 'rb'); > b=bytearray(n)" "f.seek(0);f.readinto(b)" > 1000 loops, best of 3: 640 usec per loop > > default without patch ("$ hg revert -r 68258 Objects/obmalloc.c && make"): > $

[issue11849] glibc allocator doesn't release all free()ed memory

Charles-François Natali added the comment: > However, there's still another strange regression: > > $ ./python -m timeit \ > -s "n=30; f=open('10MB.bin', 'rb', buffering=0); b=bytearray(n)" \ > "f.seek(0);f.readinto(b)" > > -> default branch: > 1 loops, best of 3: 43 usec per loop >

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou added the comment: Ah, sorry, false alarm. "b[:] = b" actually makes a temporary copy of the bytearray when assigning to itself (!). However, there's still another strange regression: $ ./python -m timeit \ -s "n=30; f=open('10MB.bin', 'rb', buffering=0); b=bytearray(n)"

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou added the comment: > I see you're comparing 3.2 and default: could you run the same > benchmark on default with and without the patch ? Same results: -> default branch: 1000 loops, best of 3: 364 usec per loop -> default branch with patch reverted: 1 loops, best of 3: 185 use

[issue11849] glibc allocator doesn't release all free()ed memory

Charles-François Natali added the comment: > For the record, this seems to make large allocations slower: > > -> with patch: > $ ./python -m timeit "b'x'*20" > 1 loops, best of 3: 27.2 usec per loop > > -> without patch: > $ ./python -m timeit "b'x'*20" > 10 loops, best of 3: 7.4

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou added the comment: More surprising is that, even ignoring the allocation cost, other operations on the memory area seem more expensive: $ ./python -m timeit -s "b=bytearray(50)" "b[:] = b" -> python 3.3: 1000 loops, best of 3: 367 usec per loop -> python 3.2: 1 loops, be

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou added the comment: For the record, this seems to make large allocations slower: -> with patch: $ ./python -m timeit "b'x'*20" 1 loops, best of 3: 27.2 usec per loop -> without patch: $ ./python -m timeit "b'x'*20" 10 loops, best of 3: 7.4 usec per loop Not sure

[issue11849] glibc allocator doesn't release all free()ed memory

Changes by Antoine Pitrou : -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker ___ _

[issue11849] glibc allocator doesn't release all free()ed memory

Roundup Robot added the comment: New changeset f8a697bc3ca8 by Antoine Pitrou in branch 'default': Issue #11849: Make it more likely for the system allocator to release http://hg.python.org/cpython/rev/f8a697bc3ca8 -- nosy: +python-dev ___ Python tra

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou added the comment: Patch looks fine to me, thank you. -- stage: -> patch review versions: -Python 2.7, Python 3.1, Python 3.2 ___ Python tracker ___ ___

[issue11849] glibc allocator doesn't release all free()ed memory

Changes by Charles-François Natali : Removed file: http://bugs.python.org/file21858/pymem.diff ___ Python tracker ___ ___ Python-bugs-list mai

[issue11849] glibc allocator doesn't release all free()ed memory

Changes by Charles-François Natali : Removed file: http://bugs.python.org/file21696/gc_trim.diff ___ Python tracker ___ ___ Python-bugs-list m

[issue11849] glibc allocator doesn't release all free()ed memory

Charles-François Natali added the comment: > I guess the final patch will have to guard the mallopt() call with some > #ifdef? Yes. See attached patch pymalloc_frag.diff It's the first time I'm playing with autotools, so please review this part really carefully ;-) > (also, I suppose a porta

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou added the comment: This is a very interesting patch, thank you. I've tested it on Mandriva 64-bit and it indeed fixes the free() issue on the XML workload. I see no regression on pybench, stringbench or json/pickle benchmarks. I guess the final patch will have to guard the mall

[issue11849] glibc allocator doesn't release all free()ed memory

Charles-François Natali added the comment: I've had some time to look at this, and I've written a quick demo patch that should - hopefully - fix this, and reduce memory fragmentation. A little bit of background first: - a couple years ago (probably true when pymalloc was designed and merged),

[issue11849] glibc allocator doesn't release all free()ed memory

Changes by Dave Malcolm : -- nosy: +dmalcolm ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python

[issue11849] glibc allocator doesn't release all free()ed memory

Charles-Francois Natali added the comment: > It isn't better. Requests above 256B are directly handled by malloc, so MALLOC_MMAP_THRESHOLD_ should in fact be set to 256 (with 1024 I guess that on 64-bit every mid-sized dictionnary gets allocated with brk). -- ___

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou added the comment: > > The MALLOC_MMAP_THRESHOLD improvement is less visible here: > > > > Are you running on 64-bit ? Yes. > If yes, it could be that you're exhausting M_MMAP_MAX (malloc falls > back to brk when there are too many mmap mappings). > You could try with > MALLOC_

[issue11849] glibc allocator doesn't release all free()ed memory

Charles-Francois Natali added the comment: > The MALLOC_MMAP_THRESHOLD improvement is less visible here: > Are you running on 64-bit ? If yes, it could be that you're exhausting M_MMAP_MAX (malloc falls back to brk when there are too many mmap mappings). You could try with MALLOC_MMAP_THRESHOLD

[issue11849] glibc allocator doesn't release all free()ed memory

kaifeng added the comment: Sorry for the later update. Valgrind shows there is no memory leak (see attached valgrind.log). The following code, while True: XML(gen_xml()) has an increasing memory usage in the first 5~8 iterations, and waves around a constant level afterwards. So I

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou added the comment: > By the way, I noticed that dictionnaries are never allocated through > pymalloc, since a new dictionnary takes more than 256B... On 64-bit builds indeed. pymalloc could be improved to handle allocations up to 512B. Want to try and write a patch? --

[issue11849] glibc allocator doesn't release all free()ed memory

Antoine Pitrou added the comment: The MALLOC_MMAP_THRESHOLD improvement is less visible here: $ MALLOC_MMAP_THRESHOLD_=1024 ../opt/python issue11849_test.py *** Python 3.3.0 alpha --- USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND 0 antoine 7703 0.0 0.1 57756