Hey Inada, thanks for the feedback

> Generally speaking, fork is a legacy API. It is too difficult to know which 
> library is fork-safe, even for stdlibs.

Yes, this is something that Instagram has to go into great lengths to make sure 
that we get the entire execution into a state where it's safe to fork. It 
works, but it's hard to maintain. We'd rather have a simpler model!

> I hope per-interpreter GIL replaces fork use cases.

We hope so too, hence the big push towards having immutable shared state across 
the interpreters. For large applications like Instagram, this is a must, 
otherwise copying state into every interpreter would be too costly.

> Anyway, I don't believe stopping refcounting will fix the CoW issue yet. See 
> this article [1] again.

That article is five years old so it doesn't reflect the current state of the 
system! We have continuous profiling and monitoring of Copy on Writes and after 
introducing the techniques described in this PEP, we have largely fixed the 
majority of scenarios where this happens.

You are right in the fact that just addressing reference counting will not fix 
all CoW issues. The trick here is also to leverage the permanent GC generation 
used for the `gc.freeze` API. That is, if you have a container that it's known 
to be immortal, it should be pushed into the permanent GC generation. This will 
guarantee that the GC itself will not change the GC headers of said instance.

Thus, if you immortalize your heap before forking (using the techniques in: 
https://github.com/python/cpython/pull/31489) then you'll end up removing the 
vast majority of scenarios where CoW takes place. I can look into writing a new 
technical article for Instagram with more up to date info but this might take 
time to get through!

Now, I said that we've largely fixed the CoW issue because there are still 
places where it happens such as: free lists, the small object allocator, etc. 
But these are relatively small compared to the ones coming from reference 
counts and the GC head mutations.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TBXKYD6OOR7I5QAMTE3VAJT5YCDISOET/
Code of Conduct: http://python.org/psf/codeofconduct/

Reply via email to