[issue24085] large memory overhead when pyc is recompiled

Change by INADA Naoki : -- resolution: -> wont fix stage: -> resolved status: open -> closed ___ Python tracker ___ ___ Python-bug

[issue24085] large memory overhead when pyc is recompiled

Serhiy Storchaka added the comment: VmRSS for different versions: malloc jmalloc 2.7: 237316 kB 90524 kB 3.4: 53888 kB 14768 kB 3.5: 51396 kB 14908 kB 3.6: 90692 kB 31776 kB 3.7: 130952 kB 28296 kB 3.8: 130284 kB 27644 kB --

[issue24085] large memory overhead when pyc is recompiled

INADA Naoki added the comment: since anon_city_hoods has massive constants, compiler_add_const makes dict larger and larger. It creates many large tuples too. I suspect it makes glibc malloc unhappy. Maybe, we can improve pymalloc for medium and large objects, by porting strategy from jemal

[issue24085] large memory overhead when pyc is recompiled

INADA Naoki added the comment: In case repro2, unreturned memory is in glibc malloc. jemalloc mitigates this issue. There are some fragmentation in pymalloc, but I think it's acceptable level. $ python3 -B repro2.py ready 1079124 VmHWM: 1079124 kB VmRSS: 83588 kB $ LD_PRELOAD=/usr/lib/

[issue24085] large memory overhead when pyc is recompiled

Jonathan G. Underwood added the comment: Thanks to both Serhiy Storchaka and David Murray - indeed you're both correct, and that is the issue in 21074, and the workaround from there of declaring a variable for that size fixes the problem. -- ___ P

[issue24085] large memory overhead when pyc is recompiled

Serhiy Storchaka added the comment: Jonathan, this is a different problem, and it is fixed in 3.6+ (see issue21074). -- ___ Python tracker ___

[issue24085] large memory overhead when pyc is recompiled

R. David Murray added the comment: That's presumably due to the compile-time constant-expression optimization. Have you tried bytes(0x100)? I don't think that gets treated as a constant by the optimizer (but I could be wrong since a bunch of things ahve been added to it lately). -

[issue24085] large memory overhead when pyc is recompiled

Jonathan G. Underwood added the comment: Seeing a very similar problem - very high memory useage during byte compilation. Consider the very simple code in a file: ``` def test_huge(): try: huge = b'\0' * 0x1 # this allocates 4GB of memory! except MemoryError:

[issue24085] large memory overhead when pyc is recompiled

Buck Evan added the comment: @serhiy.storchaka This is a very stable piece of a legacy code base, so we're not keen to refactor it so dramatically, although we could. We've worked around this issue by compiling pyc files ahead of time and taking extra care that they're preserved through deplo

[issue24085] large memory overhead when pyc is recompiled

Serhiy Storchaka added the comment: The use of PYTHONDONTWRITEBYTECODE is not a workaround because it makes your to have memory overhead unconditionally. The compiler needs more momory than require compiled data itself. If this is an issue, I suggest to use different representation for the dat

[issue24085] large memory overhead when pyc is recompiled

Changes by Antoine Pitrou : -- nosy: +benjamin.peterson, brett.cannon, georg.brandl, ncoghlan ___ Python tracker ___ ___ Python-bugs-l

[issue24085] large memory overhead when pyc is recompiled

Buck Evan added the comment: New data: The memory consumption seems to be in the compiler rather than the marshaller: ``` $ PYTHONDONTWRITEBYTECODE=1 python -c 'import repro' 16032 $ PYTHONDONTWRITEBYTECODE=1 python -c 'import repro' 16032 $ PYTHONDONTWRITEBYTECODE=1 python -c 'import repro' 1

[issue24085] large memory overhead when pyc is recompiled

Changes by Geoffrey Spear : -- components: +Interpreter Core nosy: +geoffreyspear type: -> resource usage versions: +Python 3.5 ___ Python tracker ___ __

[issue24085] large memory overhead when pyc is recompiled

Antoine Pitrou added the comment: (by the way, my numbers are with Python 3.5 - the in-development version - on 64-bit Linux) -- ___ Python tracker ___ _

[issue24085] large memory overhead when pyc is recompiled

Antoine Pitrou added the comment: Ok, I can reproduce: $ rm -r __pycache__/; ./python repro2.py ready 1047656 VmHWM: 1047656 kB VmRSS: 50660 kB $ ./python repro2.py ready 77480 VmHWM: 77480 kB VmRSS: 15664 kB My guess is that memory fragmentation prevents the RSS mark to dro

[issue24085] large memory overhead when pyc is recompiled

Changes by Anthony Sottile : Added file: http://bugs.python.org/file39259/anon_city_hoods.tar.gz ___ Python tracker ___ ___ Python-bugs-list m

[issue24085] large memory overhead when pyc is recompiled

Anthony Sottile added the comment: Attached is repro2.py (slightly different so my editor doesn't hate itself when editing the file) I'll attach the other file in another comment since it seems I can only do one at a time -- Added file: http://bugs.python.org/file39257/repro2.py

[issue24085] large memory overhead when pyc is recompiled

Antoine Pitrou added the comment: Is there any chance you can upload a script that's large enough to exhibit the problem? (perhaps with anonymized data if there's something sensitive in there) -- ___ Python tracker

[issue24085] large memory overhead when pyc is recompiled

Anthony Sottile added the comment: Ah, then 3.4 still has the problem: $ rm -rf __pycache__/ *.pyc; python3.4 repro.py ready 1112892 VmHWM: 1112892 kB VmRSS:127196 kB asottile@work:/tmp$ python3.4 repro.py ready 77468 VmHWM: 77468 kB VmRSS: 65228 kB --

[issue24085] large memory overhead when pyc is recompiled

Antoine Pitrou added the comment: Note under 3.x, you need to "rm -r __pycache__", not "rm *.pyc", since the pyc files are now stored in the __pycache__ subdirectory. -- ___ Python tracker

[issue24085] large memory overhead when pyc is recompiled

Anthony Sottile added the comment: 3.4 seems happier: asottile@work:/tmp$ rm *.pyc; python3.4 repro.py ready 77472 VmHWM: 77472 kB VmRSS: 65228 kB asottile@work:/tmp$ python3.4 repro.py ready 77472 VmHWM: 77472 kB VmRSS: 65232 kB The nasty result above is from 2.7: $ python

[issue24085] large memory overhead when pyc is recompiled

Antoine Pitrou added the comment: Which Python version is that? Can you try with 3.4 or 3.5? (is it under GNU/Linux?) > This file is significantly larger than the one attached, not sure > if it makes much of a difference. Python doesn't make a difference internally, but perhaps it has some impa

[issue24085] large memory overhead when pyc is recompiled

Anthony Sottile added the comment: I'm still seeing a very large difference: asottile@work:/tmp$ python repro.py ready 72604 VmHWM: 72604 kB VmRSS: 60900 kB asottile@work:/tmp$ rm *.pyc; python repro.py ready 1077232 VmHWM: 1077232 kB VmRSS:218040 kB This file is significantly

[issue24085] large memory overhead when pyc is recompiled

Antoine Pitrou added the comment: > Adding `import gc; gc.collect()` doesn't change the outcome afaict Of course it doesn't. The memory has already been released. "ru_maxrss" is the maximum memory consumption during the whole process lifetime. Add the following at the end of your script (Linux)

[issue24085] large memory overhead when pyc is recompiled

Anthony Sottile added the comment: Adding `import gc; gc.collect()` doesn't change the outcome afaict -- nosy: +asottile ___ Python tracker ___ __

[issue24085] large memory overhead when pyc is recompiled

Antoine Pitrou added the comment: This is transitory memory consumption. Once the source is compiled to bytecode, memory consumption falls down to its previous level. Do you care that much about it? -- nosy: +pitrou ___ Python tracker

[issue24085] large memory overhead when pyc is recompiled

Buck Evan added the comment: Also, we've reproduced this in both linux and osx. -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue24085] large memory overhead when pyc is recompiled

New submission from Buck Evan: In the attached example I show that there's a significant memory overhead present whenever a pre-compiled pyc is not present. This only occurs with more than 5225 objects (dictionaries in this case) allocated. At 13756 objects, the mysterious pyc overhead is 50% o