[issue31558] gc.freeze() - an API to mark objects as uncollectable

Change by Eric Snow : -- nosy: +eric.snow ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Change by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Łukasz Langa added the comment: New changeset c30b55b96c0967e3a8b3b86f25eb012a97f360a5 by Łukasz Langa in branch 'master': bpo-31558: Update NEWS and ACKS (#4013) https://github.com/python/cpython/commit/c30b55b96c0967e3a8b3b86f25eb012a97f360a5 -- ___

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Change by Łukasz Langa : -- pull_requests: +3987 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Change by Łukasz Langa : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___ P

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Łukasz Langa added the comment: New changeset c75edabbb65ca2bb29e51f8d1eb2c780e5890982 by Łukasz Langa (brainfvck) in branch 'master': bpo-31558: Add gc.freeze() (#3705) https://github.com/python/cpython/commit/c75edabbb65ca2bb29e51f8d1eb2c780e5890982 --

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Łukasz Langa added the comment: Based on Inadasan's, Antoine's, Neil's, and Barry's review, I'm merging the change to 3.7. -- ___ Python tracker ___ __

[issue31558] gc.freeze() - an API to mark objects as uncollectable

INADA Naoki added the comment: > So my understand is that gc frees some objects and makes some memory pages > becomes available to allocate in child process. Allocation on the shared > memory pages will cause the copy-on-write even without gc. > > Though this behavior may have better name? OK

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Zekun Li added the comment: So what we did is: We keep gc **disabled** on parent process and freeze after warmup, enable gc on child process. The reason not to do a full collection is mentioned in previous comments/original ticket - (I called it) memory fragmentation. The observation is - W

[issue31558] gc.freeze() - an API to mark objects as uncollectable

INADA Naoki added the comment: Should gc.freeze() do gc.collect() right before freezing? Or should we document `gc.collect(); gc.freeze();` idiom? I don't like `gc.freeze(collect=False)`. So if there is at least one use case of `gc.freeze()` without `gc.collect()`, I'm +1 on former (current pu

[issue31558] gc.freeze() - an API to mark objects as uncollectable

INADA Naoki added the comment: >> This is only useful if the parent process has a lot of memory that's never >> used by the child processes right? Otherwise, you would lose via refcounting >> COWs. > > What we saw in prod is that memory fragmentation caused by gc is the main > reason of share

[issue31558] gc.freeze() - an API to mark objects as uncollectable

INADA Naoki added the comment: As Instagram's report, disabling cycler GC really helps even if there is refcont. All application have some cold data: imported but never used modules, functions. -- ___ Python tracker

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Zekun Li added the comment: > This is only useful if the parent process has a lot of memory that's never > used by the child processes right? Otherwise, you would lose via refcounting > COWs. What we saw in prod is that memory fragmentation caused by gc is the main reason of shared memory sh

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Serhiy Storchaka added the comment: What about msg302790? -- nosy: +serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mai

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Łukasz Langa added the comment: Alright Python people, I don't see anybody being against the idea on the thread. Can we get a review of the linked PR? I don't think it would be good form for me to accept it. -- ___ Python tracker

[issue31558] gc.freeze() - an API to mark objects as uncollectable

STINNER Victor added the comment: (my previous msg303283 was for the bpo-11063, I removed it, sorry for the spam.) -- ___ Python tracker ___ ___

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Change by STINNER Victor : -- Removed message: https://bugs.python.org/msg303283 ___ Python tracker ___ ___ Python-bugs-list mailing

[issue31558] gc.freeze() - an API to mark objects as uncollectable

STINNER Victor added the comment: I ran two benchmarks on my Fedora 26. * new = master (commit a106aec2ed6ba171838ca7e6ba43c4e722bbecd1) * ref = commit 8d59aca4a953b097a9b02b0ecafef840e4ac5855 git co master ./python -m perf timeit -s 'import sys, uuid' "del sys.modules['uuid']; import uuid; u

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Antoine Pitrou added the comment: Le 25/09/2017 à 20:55, Neil Schemenauer a écrit : > > I think the basic idea makes a lot of sense, i.e. have a generation that is > never collected. An alternative way to implement it would be to have an > extra generation, e.g. rather than just 0, 1, 2 also

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Barry A. Warsaw added the comment: I like the idea of a gen 4 that never gets collected. This would have been useful for the original problem that inspired me to add the `generation` argument to `gc.collect()`. The nice thing about this, is just as you suggest: you could force a collection o

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Neil Schemenauer added the comment: I think the basic idea makes a lot of sense, i.e. have a generation that is never collected. An alternative way to implement it would be to have an extra generation, e.g. rather than just 0, 1, 2 also have generation 3. The collection would by default neve

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Changes by Antoine Pitrou : -- nosy: +pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyth

[issue31558] gc.freeze() - an API to mark objects as uncollectable

INADA Naoki added the comment: AFAIK, Python shutdown process calls full GC. Don't touching permanent generation makes shutdown faster. On the other hand, there are some downside: * Some object may be not freed while shutdown. It looks like "leak" for application embedding Python interpreter.

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Changes by Raymond Hettinger : -- nosy: +rhettinger ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://ma

[issue31558] gc.freeze() - an API to mark objects as uncollectable

INADA Naoki added the comment: Nice idea! I think it helps not only sharing more memory for forking application, but also long running large application. There are many static objects which is tracked by GC. It makes full GC time long. And CPU cache is filled by unused data. For example, web w

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Changes by Raymond Hettinger : -- nosy: +davin, tim.peters ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: htt

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Benjamin Peterson added the comment: This is only useful if the parent process has a lot of memory that's never used by the child processes right? Otherwise, you would lose via refcounting COWs. -- nosy: +benjamin.peterson ___ Python tracker

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Changes by Barry A. Warsaw : -- nosy: +barry ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyth

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Changes by Łukasz Langa : -- components: +Interpreter Core ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: htt

[issue31558] gc.freeze() - an API to mark objects as uncollectable

Changes by Roundup Robot : -- keywords: +patch pull_requests: +3690 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscr

[issue31558] gc.freeze() - an API to mark objects as uncollectable

New submission from Łukasz Langa: When you're forking many worker processes off of a parent process, the resulting children are initially very cheap in memory. They share memory pages with the base process until a write happens [1]_. Sadly, the garbage collector in Python touches every object