Re: [Python-Dev] Encoding of PyFrameObject members
Le 7 févr. 2015 22:34, "Greg Ewing" a écrit : with --shared) > You might be able to use Py_AddPendingCall to schedule > what you want done outside the context of the signal > handler. I don't how it could work. You have to increment the reference counting, but also maybe increment references to other frames. Again there is no guarantee that python structures are consistent in a signal handler. While a faulthandler is only called once, a profiler is called very frequently, up to once per python instruction. Unlikely bugs become very likely. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
Le 8 févr. 2015 05:39, "Gregory P. Smith" a écrit : > From there, in your signal handler you must try to acquire the newly exposed keymutex and (...) Stop! Acquiring a lock in a signal handler is just a crazy idea. It's very far from an async signal-safe function. > So until those are fixed (hooray for Antoine's PEP!), ... I wrote the PEP with Charles François Natali, but Charles wrote the whole implementation. Antoine also helped us and approved the PEP. It's the french connection! Just to say that Charles did most of the work. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
I'm working on vmprof (github.com/vmprof/vmprof-python) which works for both cpython and pypy (pypy has special support, cpython is patched on-the fly) On Sun, Feb 8, 2015 at 6:39 AM, Gregory P. Smith wrote: > To get at the Python thread state from a signal handler (using 2.7 as a > reference here; but i don't believe 3.4 has changed this part much) you need > to modify the interpreter to expose pystate.c's "autoTLSkey" and thread.c's > "struct key" as well as "keyhead" and "keymutex". > > From there, in your signal handler you must try to acquire the newly exposed > keymutex and do nothing if you were unable to get it. If you did acquire it > (rare not to), you can walk the keyhead list looking for autoTLSkey to find > the current valid thread state. > > I had an intern (hi Elena!) write a signal sampling based low overhead > Python CPU profiler based on that last summer. I believe there are still > bugs to shaken out (if they are even possible to fix... Armin's comments are > true: signal handler code is super limited). I am stating this here because > I want someone to pester me at PyCon if I haven't released our work as a > proof of concept by then. The important take away: From what I could figure > out, you need to modify the CPython interpreter to be more amenable to such > introspection. > > A downside of a signal based profiler: *ALL* of the EINTR mishandling bugs > within the Python interpreter, stdlib, and your own code will show up in > your application. So until those are fixed (hooray for Antoine's PEP!), it > may not be practical for use on production processes which is sort of the > entire point of a low overhead sampling profiler... > > I'd like to get a buildbot setup that runs the testsuite while a continual > barrage of signals are being generated. We really don't stress test that > stuff (as evidence by the EINTR mishandling issues that are rampant) as > non-fatal signals are so rare for most things... until they aren't. > > As a side note and encouragement: I wonder what PyPy could do for > dynamically enabled and disabled low overhead CPU profiling. (take that as a > hint that I want someone else to get extremely creative!) > > -gps > > On Sat Feb 07 2015 at 1:34:26 PM Greg Ewing > wrote: >> >> Maciej Fijalkowski wrote: >> > However, you can't access thread >> > locals from signal handlers (since in some cases it mallocs, thread >> > locals are built lazily if you're inside the .so, e.g. if python is >> > built with --shared) >> >> You might be able to use Py_AddPendingCall to schedule >> what you want done outside the context of the signal >> handler. >> >> The call will be made by the main thread, though, >> so if you need to access the frame of whatever thread >> was running when the signal occured, you will have >> to track down its PyThreadState somehow and get the >> frame from there. Not sure what would be involved >> in doing that. >> >> -- >> Greg >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
Hi Francis Feel free to steal most of vmprof code, it should generally work without requiring to patch cpython (python 3 patches appreciated :-). As far as timer goes - it seems not to be going anywhere, I would rather use a background thread or something On Sun, Feb 8, 2015 at 10:03 PM, Francis Giraldeau wrote: > 2015-02-08 4:01 GMT-05:00 Maciej Fijalkowski : >> >> I'm working on vmprof (github.com/vmprof/vmprof-python) which works >> for both cpython and pypy (pypy has special support, cpython is >> patched on-the fly) > > > This looks interesting. I'm working on a profiler that is similar, but not > based on timer. Instead, the signal is generated when an hardware > performance counter overflows. It required a special linux kernel module, > and the tracepoint is recorded using LTTng-UST. > > https://github.com/giraldeau/perfuser > https://github.com/giraldeau/perfuser-modules > https://github.com/giraldeau/python-profile-ust > > This is of course very experimental, requires a special setup, an I don't > even know if it's going to produce good results. I'll report the results in > the coming weeks. > > Cheers, > > Francis Giraldeau ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
2015-02-08 4:01 GMT-05:00 Maciej Fijalkowski : > I'm working on vmprof (github.com/vmprof/vmprof-python) which works > for both cpython and pypy (pypy has special support, cpython is > patched on-the fly) This looks interesting. I'm working on a profiler that is similar, but not based on timer. Instead, the signal is generated when an hardware performance counter overflows. It required a special linux kernel module, and the tracepoint is recorded using LTTng-UST. https://github.com/giraldeau/perfuser https://github.com/giraldeau/perfuser-modules https://github.com/giraldeau/python-profile-ust This is of course very experimental, requires a special setup, an I don't even know if it's going to produce good results. I'll report the results in the coming weeks. Cheers, Francis Giraldeau ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [RELEASE] Python 3.4.3rc1 is now available
On behalf of the Python development community and the Python 3.4 release team, I'm happy to announce the availability of Python 3.4.3rc1. Python 3.4.3rc1 has many bugfixes and other small improvements over 3.4.2. You can download it here: https://www.python.org/download/releases/3.4.3 Not done yet, //arry/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [RELEASE] Python 3.5.0a1 is now available
On behalf of the Python development community and the Python 3.5 release team, I'm also pleased to announce the availability of Python 3.5.0a1. Python 3.5.0a1 is the first alpha release of Python 3.5, which will be the next major release of Python. Python 3.5 is still under heavy development, and is far from complete. This is a preview release, and its use is not recommended for production settings. You can download it here: https://www.python.org/download/releases/3.5.0a1 Happy hacking, //arry/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
On 9 February 2015 at 09:11, Maciej Fijalkowski wrote: > Hi Francis > > Feel free to steal most of vmprof code, it should generally work > without requiring to patch cpython (python 3 patches appreciated :-). > As far as timer goes - it seems not to be going anywhere, I would > rather use a background thread or something What about setting a flag when the signal arrives and checking it at the next bytecode evaluation or something? -Rob ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
This is exactly how signals are implemented in Python :-) Victor Le lundi 9 février 2015, Robert Collins a écrit : > On 9 February 2015 at 09:11, Maciej Fijalkowski > wrote: > > Hi Francis > > > > Feel free to steal most of vmprof code, it should generally work > > without requiring to patch cpython (python 3 patches appreciated :-). > > As far as timer goes - it seems not to be going anywhere, I would > > rather use a background thread or something > > What about setting a flag when the signal arrives and checking it at > the next bytecode evaluation or something? > > -Rob > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
On Sun Feb 08 2015 at 12:21:36 AM Victor Stinner wrote: > > Le 8 févr. 2015 05:39, "Gregory P. Smith" a écrit : > > From there, in your signal handler you must try to acquire the newly > exposed keymutex and (...) > > Stop! Acquiring a lock in a signal handler is just a crazy idea. It's very > far from an async signal-safe function. > I'd normally agree... In this case, sem_trywait() is non-blocking. It is unfortunately not on the official POSIX list of async-signal-safe functions (of the SEMs, only sem_post() is) but many implementations actually are safe, documented or not. https://github.com/lattera/glibc/blob/master/nptl/sysdeps/unix/sysv/linux/x86_64/sem_trywait.S - appears safe. So long as the underlying semaphore is implemented using an atomic instruction on the target platform I wouldn't expect any implementation of sem_trywait to be async signal unsafe. (nor would I depend on that without checking) -gps > > So until those are fixed (hooray for Antoine's PEP!), ... > > I wrote the PEP with Charles François Natali, but Charles wrote the whole > implementation. Antoine also helped us and approved the PEP. It's the > french connection! Just to say that Charles did most of the work. > > Victor > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
