[Python-Dev] Following up stories about beginning contribution
Hello everyone, Following up my request for such stories in early August[1], I have received 10 such stories which may be shared publicly, and several more in private. As a first step, I've create a repository[2] with all of the publicly available stories, so that they are easily accessible and readable from a single location. I have made sure to collected written consent from all authors to use these stories publicly. Hence, everything there is published under the CC 4.0 license, with attribution to the original author and a link to the original post/message/tweet. *If any author is uncomfortable with these publication terms, please let me know ASAP!* My next step will be to read through the stories again recognize common and obviously critical problems or patterns. I'll publish those findings and plan to follow up with a structured survey in hope of collecting information and opinions specific to those issues from a larger number of people. Then, hopefully, we'll have a solid foundation for making improvements. .. [1]: https://mail.python.org/archives/list/core-mentors...@python.org/message/JSTTXJRLS2A6W5NCBDOHNW7RSRKVIL2F/ .. [2]: https://github.com/taleinat/python-contribution-feedback ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HYHVSPZTLCDMHWVA2P6T3ITVEQHPZQFR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Belatedly accepting PEP 579 (design discussion leading to vectorcall)
I approve [PEP 579], with the understanding that acceptance of an Informational PEP doesn't really mean that much. PEP 579 charts the design issues addressed in PEP 575, PEP 580, PEP 590 (and hopefully more upcoming proposals). As noted in PEP 1: Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementers are free to ignore Informational PEPs or follow their advice. While there is no consensus on whether the issues or the solutions in PEP 579 are valid, the list is still useful to guide further design. I plan to note that in the PEP itself to inform drive-by readers: https://github.com/python/peps/pull/1233/files I should have done this ages ago, but it fell off my TODO list. Thanks to Brett for reminding me! [PEP 579]: https://www.python.org/dev/peps/pep-0579/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6DWOBYFL4VNPFZHVU2IUYYMS7NSEUNMR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Implementation of PEP-0604
Cool ! I'm very happy you take time to do this job. Your idea to use a shadowobject objet to promote just-in-time is very good for me, and can be used for others extensions. Thanks Philippe Prados Le sam. 9 nov. 2019 à 00:12, Richard Eames a écrit : > Hi Philippe and list, > > Long time lurker here, I've had a few spare cycles at $work this week that > I spent exploring an alternate implementation to PEP604 starting with the > implementation from Philippe. I don't say "reference implementation", > because it doesn't implement PEP604 faithfully, but it does try to > implement it's goal. My thinking was that the "runtime" component to it > only needs to support the isinstance/issubclass methods and so can be > pretty minimal without need to import all the typing.py machinery. At which > point, when actual type checking was done (anything in typing.py or mypy), > the `union` type can be special cased, or promoted to a `typing.Union`. > > I started by creating a built-in "union" type, that was a thin wrapper > around a tuple, such that type.__or__ will return `union(lhs, rhs)` which > stored the two items in a 2-length tuple. I also changed the methods in > `typing.Union` to return a `union`. However, I hit my first issue: `int | > int | str` would then return `union(int, union(int, str))`, whereas it > should deduplicate the types. So, I implemented logic to detect and merge > union | union, but changed the underlying data to a plain flat tuple. And > the next issue I hit was that some of the other `typing.py` types weren't > being detected as valid types (probably due to my limited understanding of > the PyObject datastructure/flags), specifically, `int | "forwardref"`, I > needed a way to signal to typing.py that I had a forwardref without > importing it. This is where I changed my approach to the problem. > > I wanted to find a way to have these metatypes (Union/TypeVar/ForwardRef) > without re-implementing them in c, or importing them. My Idea was to change > the builtin type to what I called a "shadow type". In the C layer it's just > holds a tuple of objects/parameters, and has a type field to say what it > is, then when in the type checking/typing.py layer, if it detects an object > is a "shadowobject", then it promotes it to a full type. What I liked about > this approach, is that for isinstance/issubclass, when it see a > "shadowobject" it can just grab the underlying types from the > shadowobject->params tuple and no import magic needs to happen. All of the > type magic can be left to mypy/typing.py without much knowledge of the > shadowobject. I've mostly got through this implementation with all but one > test passing. > > At this point, I've got as far as I can go without help, so I'm posting > here, looking for feedback or discussion. I expect that maybe the next step > should be me commenting on the PEP604 itself for implementation concerns > regarding `_GenericAlias`? > > Implementation is located at > https://github.com/Naddiseo/cpython/tree/PEP604 > > Richard > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/P56ZE6USNIGCQU5PIOLKC65VHD5DM37U/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FLRUBE4VNWCFUN6HOBXB7TZRRXXDWCZN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Pass the Python thread state to internal C functions
Hi,
Are you ok to modify internal C functions to pass explicitly tstate?
--
I started to modify internal C functions to pass explicitly "tstate"
when calling C functions: the Python thread state (PyThreadState).
Example of C code (after my changes):
if (_Py_EnterRecursiveCall(tstate, " while calling a Python object")) {
return NULL;
}
PyObject *result = (*call)(callable, args, kwargs);
_Py_LeaveRecursiveCall(tstate);
return _Py_CheckFunctionResult(tstate, callable, result, NULL);
In Python 3.8, the tstate is implicit:
if (Py_EnterRecursiveCall(" while calling a Python object")) {
return NULL;
}
PyObject *result = (*call)(callable, args, kwargs);
Py_LeaveRecursiveCall();
return _Py_CheckFunctionResult(callable, result, NULL);
There are different reasons to pass explicitly tstate, but my main
motivation is to rework Python code base to move away from implicit
global states to states passed explicitly, to implement the PEP 554
"Multiple Interpreters in the Stdlib". In short, the final goal is to
run multiple isolated Python interpreters in the same process: run
pure Python code on multiple CPUs in parallel with a single process
(whereas multiprocessing runs multiple processes).
Currently, subinterpreters are a hack: they still share a lot of
things, the code base is not ready to implement isolated interpreters
with one "GIL" (interpreter lock) per interpreter, and to run multiple
interpreters in parallel. Many _PyRuntimeState fields (the global
_PyRuntime variable) should be moved to PyInterpreterState (or maybe
PyThreadState): per interpreter.
Another simpler but more annoying example are Py_None and Py_True
singletons which are globals. We cannot share these singletons between
interpreters because updating their reference counter would be a
performance bottleneck. If we put a "superglobal-GIL" to ensure that
Py_None reference counter remains consistent, it would basically
"serialize" all threads, rather than running them in parallel.
The idea of passing tstate to internal C functions is to prepare code
to get the per-interpreter None from tstate.
tstate is basically the "root" to access all states which are per
interpreter. For example, PyInterpreterState can be read from
tstate->interp.
Right now, tstate is only passed to a few functions, but you should
expect to see it passed to way more functions later, once more
structures will be moved to PyInterpreterState.
--
On my latest merged PR 17052 ("Add _PyObject_VectorcallTstate()"),
Mark Shannon wrote: "I don't see how this could ever be faster, nor do
I see how it is more correct."
https://github.com/python/cpython/pull/17052#issuecomment-552538438
Currently, tstate is get using these internal APIs:
#define _PyRuntimeState_GetThreadState(runtime) \
((PyThreadState*)_Py_atomic_load_relaxed(&(runtime)->gilstate.tstate_current))
#define _PyThreadState_GET() _PyRuntimeState_GetThreadState(&_PyRuntime)
or using public APIs:
PyAPI_FUNC(PyThreadState *) PyThreadState_Get(void);
#define PyThreadState_GET() PyThreadState_Get()
I dislike _PyThreadState_GET() for 2 reasons:
* it relies on the _PyRuntime global variable: I would prefer to avoid
global variables
* it uses an atomic operation which can become a perofrmance issue
when more and more code will require tstate
--
An alternative would be to use PyGILState_GetThisThreadState() which
uses a thread local state (TLS) variable to get the Python thread
state ("tstate"), rather that _PyRuntime atomic variable. Except that
the PyGILState API doesn't support subinterpreters yet :-(
https://bugs.python.org/issue15751 "Support subinterpreters in the GIL
state API" is open since 2012.
Note: While the GIL is released, _PyThreadState_GET() is NULL, whereas
PyGILState_GetThisThreadState() is non-NULL.
--
Links:
* https://pythoncapi.readthedocs.io/runtime.html : my notes on moving
globals to per interpreter states
* https://bugs.python.org/issue36710
* https://bugs.python.org/issue38644
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/PQBGECVGVYFTVDLBYURLCXA3T7IPEHHO/
Code of Conduct: http://python.org/psf/codeofconduct/
