Re: [Python-Dev] Signals, threads, blocking C functions
On Saturday 02 September 2006 22:10, Gustavo Carneiro wrote: > According to [1], all python needs to do to avoid this problem is > block all signals in all but the main thread; then we can guarantee > signal handlers are always called from the main thread, and pygtk > doesn't need a timeout. > But I would really prefer the first alternative, as it could be > fixed within python 2.5; no need to wait for 2.6. Assuming "the first alternative" is the "just block all signals in all but the main thread" option, there is absolutely no chance of this going into 2.5. Signals and threads combined are an complete *nightmare* of platform-specific behaviour. I'm -1000 on trying to change this code now, _after_ the first release candidate. To say that "that path lies madness" is like saying "Pacific Ocean large, wet, full of fish". -- Anthony Baxter <[EMAIL PROTECTED]> It's never too late to have a happy childhood. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Error while building 2.5rc1 pythoncore_pgo on VC8
Fredrik Lundh wrote: >> That error mentioned in that post was in "pythoncore" module. >> My error is while compiling "pythoncore_pgo" module. > > iirc, that's a partially experimental alternative build for playing > with performance guided optimizations. are you sure you need > that module ? Oh yes, it's a 30% improvement in pystone, for free. -- Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
"Gustavo Carneiro" <[EMAIL PROTECTED]> writes: > According to [1], all python needs to do to avoid this problem is > block all signals in all but the main thread; Argh, no: then people who call system() from non-main threads end up running subprocesses with all signals masked, which breaks other things in very mysterious ways. Been there... No time to read the rest of the post, maybe in a few days... Cheers, mwh -- Agh, the braindamage! It's not unlike the massively non-brilliant decision to use the period in abbreviations as well as a sentence terminator. Had these people no imagination at _all_? -- Erik Naggum, comp.lang.lisp ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
On 9/4/06, Michael Hudson <[EMAIL PROTECTED]> wrote: > "Gustavo Carneiro" <[EMAIL PROTECTED]> writes: > > > According to [1], all python needs to do to avoid this problem is > > block all signals in all but the main thread; > > Argh, no: then people who call system() from non-main threads end up > running subprocesses with all signals masked, which breaks other > things in very mysterious ways. Been there... That's a very good point; I wasn't aware that child processes inherited the signals mask from their parent processes. > No time to read the rest of the post, maybe in a few days... Don't worry. From the feedback received so far it seems that any proposed solution has to wait for Python 2.6 :-( I am now thinking of something along these lines: typedef void (*PyPendingCallNotify)(void *user_data); PyAPI_FUNC(void) Py_AddPendingCallNotify(PyPendingCallNotify callback, void *user_data); PyAPI_FUNC(void) Py_RemovePendingCallNotify(PyPendingCallNotify callback, void *user_data); Regards. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
"Gustavo Carneiro" <[EMAIL PROTECTED]> wrote: > > That's a very good point; I wasn't aware that child processes > inherited the signals mask from their parent processes. That's one of the few places where POSIX does describe what happens. Well, usually. You really don't want to know what happens when you call something revolting, like csh or a setuid program. This particular mess is why I had to write my own nohup - the new POSIX interfaces broke the existing one, and it remains broken today on almost all systems. > I am now thinking of something along these lines: > typedef void (*PyPendingCallNotify)(void *user_data); > PyAPI_FUNC(void) Py_AddPendingCallNotify(PyPendingCallNotify callback, > void *user_data); > PyAPI_FUNC(void) Py_RemovePendingCallNotify(PyPendingCallNotify > callback, void *user_data); Why would that help? The problems are semantic, not syntactic. Anthony Baxter isn't exaggerating the problem, despite what you may think from his posting. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
Chris McDonough <[EMAIL PROTECTED]> wrote: > > Would adding an API for sigprocmask help here? No. sigprocmask is a large part of the problem. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
On Tuesday 05 September 2006 00:05, Nick Maclaren wrote: > Anthony Baxter isn't exaggerating the problem, despite what you may > think from his posting. If the SF bugtracker had a better search interface, you could see why I have such a bleak view of this area of Python. What's there now *mostly* works (I exclude freakshows like certain versions of HP/UX, AIX, SCO and the like). It took a hell of a lot of effort to get it to this point. threads + signals == tears. Anthony ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
On 9/4/06, Nick Maclaren <[EMAIL PROTECTED]> wrote: > "Gustavo Carneiro" <[EMAIL PROTECTED]> wrote: > > I am now thinking of something along these lines: > > typedef void (*PyPendingCallNotify)(void *user_data); > > PyAPI_FUNC(void) Py_AddPendingCallNotify(PyPendingCallNotify callback, > > void *user_data); > > PyAPI_FUNC(void) Py_RemovePendingCallNotify(PyPendingCallNotify > > callback, void *user_data); > > Why would that help? The problems are semantic, not syntactic. > > Anthony Baxter isn't exaggerating the problem, despite what you may > think from his posting. You guys are tough customers to please. I am just trying to solve a problem here, not create a new one; you have to believe me. OK, let's review what we know about current python, signals, and threads: 1. Python launches threads without touching sigprocmask; 2. Python installs signal handlers for all signals; 3. Signals can be delivered to any thread, let's assume (because of point #1 and not others not mentioned) that we have no control over which threads receive which signals, might as well be random for all we know; 4. Python signal handlers do almost nothing: just sets a flag, and calls Py_AddPendingCall, to postpone the job of handling a signal until a "safer" time. 5. The function Py_MakePendingCalls() should eventually get called at a "safer" time by user or python code. 6. It follows that until Py_MakePendingCalls() is called, the signal will not be handled at all! Now, back to explaining the problem. 1. In PyGTK we have a gobject.MainLoop.run() method, which blocks essentially forever in a poll() system call, and only wakes if/when it has to process timeout or IO event; 2. When we only have one thread, we can guarantee that e.g. SIGINT will always be caught by the thread running the g_main_loop_run(), so we know poll() will be interrupted and a EINTR will be generated, giving us control temporarily back to check for python signals; 3. When we have multiple thread, we cannot make this assumption, so instead we install a timeout to periodically check for signals. We want to get rid of timeouts. Now my idea: add a Python API to say: "dear Python, please call me when you start having pending calls, even if from a signal handler context, ok?" >From that point on, signals will get handled by Python, python calls PyGTK, PyGTK calls a special API to safely wake up the main loop even from a thread or signal handler, then main loop checks for signal by calling PyErr_CheckSignals(), it is handled by Python, and the process lives happily ever after, or die trying. I sincerely hope my explanation was satisfactory this time. Best regards. PS: there's a "funny" comment in Py_AddPendingCall that suggests it is not very safe against reentrancy problems: /* XXX Begin critical section */ /* XXX If you want this to be safe against nested XXX asynchronous calls, you'll have to work harder! */ Are signal handlers guaranteed to not be interrupted by another signal, at least? What about threads? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
On Tuesday 05 September 2006 00:52, Gustavo Carneiro wrote: > 3. Signals can be delivered to any thread, let's assume (because > of point #1 and not others not mentioned) that we have no control over > which threads receive which signals, might as well be random for all > we know; Note that some Unix variants only deliver signals to the main thread (or so the manpages allege, anyway). Anthony ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
On Mon, 04 Sep 2006 15:05:56 +0100, Nick Maclaren <[EMAIL PROTECTED]> wrote: >"Gustavo Carneiro" <[EMAIL PROTECTED]> wrote: >> >> That's a very good point; I wasn't aware that child processes >> inherited the signals mask from their parent processes. > >That's one of the few places where POSIX does describe what happens. >Well, usually. You really don't want to know what happens when you >call something revolting, like csh or a setuid program. This >particular mess is why I had to write my own nohup - the new POSIX >interfaces broke the existing one, and it remains broken today on >almost all systems. > >> I am now thinking of something along these lines: >> typedef void (*PyPendingCallNotify)(void *user_data); >> PyAPI_FUNC(void) Py_AddPendingCallNotify(PyPendingCallNotify callback, >> void *user_data); >> PyAPI_FUNC(void) Py_RemovePendingCallNotify(PyPendingCallNotify >> callback, void *user_data); > >Why would that help? The problems are semantic, not syntactic. > >Anthony Baxter isn't exaggerating the problem, despite what you may >think from his posting. > Dealing with threads and signals is certainly hairy. However, that barely has anything to do with what Gustavo is talking about. By the time Gustavo's proposed API springs into action, the threads already exist and the signal is already being handled by one. So, let's forget about threads and signals for a moment. The problem to be solved is that one piece of code wants to communicate a piece of information to another piece of code. The first piece of code is in Python itself. The second piece of code could be from any third-party library, and Python has no way of knowing about it - now. Gustavo is suggesting adding a registration API so that these third-party libraries can tell Python that they exist and are interested in this piece of information. Simple, no? PyGTK would presumably implement its pending call callback by writing a byte to a pipe which it is also passing to poll(). This lets them handle signals in a very timely manner without constantly waking up from poll() to see if Python wants to do any work. This is far from a new idea - it's basically the bog standard way of handling this situation. It strikes me as a very useful API to add to Python (although at this point in the 2.5 release process, not to 2.5, sorry Gustavo). Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
Gustavo Carneiro wrote: > OK, let's review what we know about current python, signals, and threads: > > 1. Python launches threads without touching sigprocmask; > 2. Python installs signal handlers for all signals; > 3. Signals can be delivered to any thread, let's assume (because > of point #1 and not others not mentioned) that we have no control over > which threads receive which signals, might as well be random for all > we know; > 4. Python signal handlers do almost nothing: just sets a flag, > and calls Py_AddPendingCall, to postpone the job of handling a signal > until a "safer" time. > 5. The function Py_MakePendingCalls() should eventually get > called at a "safer" time by user or python code. > 6. It follows that until Py_MakePendingCalls() is called, the > signal will not be handled at all! > > Now, back to explaining the problem. > > 1. In PyGTK we have a gobject.MainLoop.run() method, which blocks > essentially forever in a poll() system call, and only wakes if/when it > has to process timeout or IO event; > 2. When we only have one thread, we can guarantee that e.g. > SIGINT will always be caught by the thread running the > g_main_loop_run(), so we know poll() will be interrupted and a EINTR > will be generated, giving us control temporarily back to check for > python signals; > 3. When we have multiple thread, we cannot make this assumption, > so instead we install a timeout to periodically check for signals. > > We want to get rid of timeouts. Now my idea: add a Python API to say: > "dear Python, please call me when you start having pending calls, > even if from a signal handler context, ok?" What can be safely done from a signal handler context is *very* limited. Calling back arbitrary Python code is certainly not safe. Reliable asynchronous interruption of arbitrary code is a difficult problem, but POSIX and POSIX implementations botch it particularly badly. I don't know how to implement what you want here, but I'd endorse the comments of Nick Maclaren and Antony Baxter against making precipitate changes. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
"Gustavo Carneiro" <[EMAIL PROTECTED]> wrote: > > You guys are tough customers to please. I am just trying to solve a > problem here, not create a new one; you have to believe me. Oh, I believe you. Look at it this way. You are trying to resolve the problem that your farm is littered with cluster bombs, and your cows keep blowing their legs off. Your solution is effectively saying "well, let's travel around and pick them all up then". > We want to get rid of timeouts. Now my idea: add a Python API to say: > "dear Python, please call me when you start having pending calls, > even if from a signal handler context, ok?" Yes, I know. I have been there and done that, both academically and (observing, as a consultant) to the vendor. And that was on a system that was a damn sight better engineered than any of the main ones that Python runs on today. I have attempted to do much EASIER tasks under both Unix and (earlier) versions of Microsoft Windows, and failed dismally because the system wasn't up to it. > From that point on, signals will get handled by Python, python calls > PyGTK, PyGTK calls a special API to safely wake up the main loop even > from a thread or signal handler, then main loop checks for signal by > calling PyErr_CheckSignals(), it is handled by Python, and the process > lives happily ever after, or die trying. The first thing that will happen to that beautiful theory when it goes out into Unix County or Microsoft City is that a gang of ugly facts will find it and beat it into a pulp. > I sincerely hope my explanation was satisfactory this time. Oh, it was last time. It isn't that that is the problem. > Are signal handlers guaranteed to not be interrupted by another > signal, at least? What about threads? No and no. In theory, what POSIX says about blocking threads should be reliable; in my experience, it almost is, except under precisely the circumstances that you most want it to work. Look, I am agreeing that your basic design is right. What I am saying is that (a) you cannot make delivery reliable and abolish timeouts and (b) that it is such a revoltingly system-dependent mess that I would much rather Python didn't fiddle with it. Do you know how signalling is misimplemented at the hardware level? And that it is possible for a handler to be called with any of its critical pointers (INCLUDING the global code and data pointers) in undefined states? Do you know how to program round that sort of thing? I can answer "yes" to all three - for my sins, which must be many and grievous, for that to be the case :-( Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
Jean-Paul Calderone wrote: > PyGTK would presumably implement its pending call callback by writing a > byte to a pipe which it is also passing to poll(). But doing that in a signal handler context invokes undefined behaviour according to POSIX. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
On Mon, 04 Sep 2006 17:24:56 +0100, David Hopwood <[EMAIL PROTECTED]> wrote: >Jean-Paul Calderone wrote: >> PyGTK would presumably implement its pending call callback by writing a >> byte to a pipe which it is also passing to poll(). > >But doing that in a signal handler context invokes undefined behaviour >according to POSIX. write(2) is explicitly listed as async-signal safe in IEEE Std 1003.1, 2004. Was this changed in a later edition? Otherwise, I don't understand what you mean by this. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > On Mon, 04 Sep 2006 17:24:56 +0100, > David Hopwood <[EMAIL PROTECTED] > der.co.uk> wrote: > >Jean-Paul Calderone wrote: > >> PyGTK would presumably implement its pending call callback by writing a > >> byte to a pipe which it is also passing to poll(). > > > >But doing that in a signal handler context invokes undefined behaviour > >according to POSIX. > > write(2) is explicitly listed as async-signal safe in IEEE Std 1003.1, 2004. > Was this changed in a later edition? Otherwise, I don't understand what you > mean by this. Try looking at the C90 or C99 standard, for a start :-( NOTHING may safely be done in a real signal handler, except possibly setting a value of type static volatile sig_atomic_t. And even that can be problematic. And note that POSIX defers to C on what the C languages defines. So, even if the function is async-signal-safe, the code that calls it can't be! POSIX's lists are complete fantasy, anyway. Look at the one that defines thread-safety, and then try to get your mind around what exit being thread-safe actually implies (especially with regard to atexit functions). Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
Jean-Paul Calderone wrote: > On Mon, 04 Sep 2006 17:24:56 +0100, David Hopwood <[EMAIL PROTECTED]> wrote: > >>Jean-Paul Calderone wrote: >> >>>PyGTK would presumably implement its pending call callback by writing a >>>byte to a pipe which it is also passing to poll(). >> >>But doing that in a signal handler context invokes undefined behaviour >>according to POSIX. > > write(2) is explicitly listed as async-signal safe in IEEE Std 1003.1, 2004. I stand corrected. I must have misremembered this. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
On Mon, 04 Sep 2006 18:18:41 +0100, Nick Maclaren <[EMAIL PROTECTED]> wrote: >Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: >> On Mon, 04 Sep 2006 17:24:56 +0100, >> David Hopwood <[EMAIL PROTECTED] >> der.co.uk> wrote: >> >Jean-Paul Calderone wrote: >> >> PyGTK would presumably implement its pending call callback by writing a >> >> byte to a pipe which it is also passing to poll(). >> > >> >But doing that in a signal handler context invokes undefined behaviour >> >according to POSIX. >> >> write(2) is explicitly listed as async-signal safe in IEEE Std 1003.1, 2004. >> Was this changed in a later edition? Otherwise, I don't understand what you >> mean by this. > >Try looking at the C90 or C99 standard, for a start :-( > >NOTHING may safely be done in a real signal handler, except possibly >setting a value of type static volatile sig_atomic_t. And even that >can be problematic. And note that POSIX defers to C on what the C >languages defines. So, even if the function is async-signal-safe, >the code that calls it can't be! > >POSIX's lists are complete fantasy, anyway. Look at the one that >defines thread-safety, and then try to get your mind around what >exit being thread-safe actually implies (especially with regard to >atexit functions). > Thanks for expounding. Given that it is basically impossible to do anything useful in a signal handler according to the relevant standards (does Python's current signal handler even avoid relying on undefined behavior?), how would you suggest addressing this issue? It seems to me that it is actually possible to do useful things in a signal handler, so long as one accepts that doing so is relying on platform specific behavior. How hard would it be to implement this for the platforms Python supports, rather than for a hypothetical standards-exact platform? Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > > Thanks for expounding. Given that it is basically impossible to do > anything useful in a signal handler according to the relevant standards > (does Python's current signal handler even avoid relying on undefined > behavior?), how would you suggest addressing this issue? Much as you are doing, and I described, but the first step would be to find out what 'most' Python people need for signal handling in threaded programs. This is because there is an unavoidable conflict between portability/reliability and functionality. I would definitely block all signals in threads, except for those that are likely to be generated ON the thread (SIGFPE etc.) It is a very good idea not to touch the handling of several of those, because doing so can cause chaos. I would have at least two 'standard' handlers, one of which would simply set a flag and return, and the other of which would abort. Now, NEITHER is a very useful specification, but providing ANY information is risky, which is why it is critical to know what people need. I would not TRUST the blocking of signals, so would set up handlers even when I blocked them, and would do the minimum fiddling in the main thread compatible with decent functionality. I would provide a call to test if the signal flag was set, and another to test and clear it. This would be callable ONLY from the main thread, and that would be checked. It is possible to do better, but that starts needing serious research. > It seems to me that it is actually possible to do useful things in a > signal handler, so long as one accepts that doing so is relying on > platform specific behavior. Unfortunately, that is wrong. That was true under MVS and VMS, but in Unix and Microsoft systems, the problem is that the behaviour is both platform and circumstance-dependent. What you can do reliably depends mostly on what is going on at the time. For example, on many Unix and Microsoft platforms, signals received while you are in the middle of certain functions or system calls, or certain particular signals (often SIGFPE), call the C handler with a bad set of global pointers or similar. I believe that this is one of reasons (perhaps the main one) that some such failures so often cause debuggers to be unable to find the stack pointer. I have tracked a few of those down, and have occasionally identified the cause (and even got it fixed!), but it is a murderous task, and I know of few other people who have ever succeeded. > How hard would it be to implement this for the platforms Python supports, > rather than for a hypothetical standards-exact platform? I have seen this effect on OSF/1, IRIX, Solaris, Linux and versions of Microsoft Windows. I have never used a modern BSD, haven't used HP-UX since release 9, and haven't used Microsoft systems seriously in years (though I did hang my new laptop in its GUI fairly easily). As I say, this isn't so much a platform issue as a circumstance one. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Cross-platform math functions?
Hi - I'm curious if there is any interest in the Python community to achieve better cross-platform math behavior. A quick test[1] shows a non-surprising difference between the platform implementations. Question: Is there any interest in changing the behavior to produce identical results across platforms (for example by utilizing fdlibm [2])? Since I have need for a set of cross-platform math functions I'll probably start with a math-compatible fdlibm module (unless somebody has done that already ;-) Cheers, - Andreas [1] Using Python 2.4: >>> import math >>> math.cos(1.0e32) WinXP:-0.39929634612021897 LinuxX86: -0.49093671143542561 [2] http://www.netlib.org/fdlibm/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
In GLib we have a child watch notification feature that relies on
the following signal handler:
static void
g_child_watch_signal_handler (int signum)
{
child_watch_count ++;
if (child_watch_init_state == CHILD_WATCH_INITIALIZED_THREADED)
{
write (child_watch_wake_up_pipe[1], "B", 1);
}
else
{
/* We count on the signal interrupting the poll in the same thread.
*/
}
}
Now, we've had this API for a long time already (at least 2.5
years). I'm pretty sure it works well enough on most *nix systems.
Event if it works 99% of the times, it's way better than *failing*
*100%* of the times, which is what happens now with Python.
All I ask is an API to add a callback that Python signal handlers
call, from signal context. That much I'm sure is safe. What happens
from there on will be out of Python's hands, so Python
purist^H^H^H^H^H^H developers cannot be blamed for anything that
happens next. You can laugh at PyGTK and GLib all you want for having
"unsafe signal handling", I don't care.
Regards.
On 9/4/06, Nick Maclaren <[EMAIL PROTECTED]> wrote:
> Jean-Paul Calderone <[EMAIL PROTECTED]> wrote:
> >
> > Thanks for expounding. Given that it is basically impossible to do
> > anything useful in a signal handler according to the relevant standards
> > (does Python's current signal handler even avoid relying on undefined
> > behavior?), how would you suggest addressing this issue?
>
> Much as you are doing, and I described, but the first step would be
> to find out what 'most' Python people need for signal handling in
> threaded programs. This is because there is an unavoidable conflict
> between portability/reliability and functionality.
>
> I would definitely block all signals in threads, except for those that
> are likely to be generated ON the thread (SIGFPE etc.) It is a very
> good idea not to touch the handling of several of those, because doing
> so can cause chaos.
>
> I would have at least two 'standard' handlers, one of which would simply
> set a flag and return, and the other of which would abort. Now, NEITHER
> is a very useful specification, but providing ANY information is risky,
> which is why it is critical to know what people need.
>
> I would not TRUST the blocking of signals, so would set up handlers even
> when I blocked them, and would do the minimum fiddling in the main
> thread compatible with decent functionality.
>
> I would provide a call to test if the signal flag was set, and another
> to test and clear it. This would be callable ONLY from the main thread,
> and that would be checked.
>
> It is possible to do better, but that starts needing serious research.
>
> > It seems to me that it is actually possible to do useful things in a
> > signal handler, so long as one accepts that doing so is relying on
> > platform specific behavior.
>
> Unfortunately, that is wrong. That was true under MVS and VMS, but
> in Unix and Microsoft systems, the problem is that the behaviour is
> both platform and circumstance-dependent. What you can do reliably
> depends mostly on what is going on at the time.
>
> For example, on many Unix and Microsoft platforms, signals received
> while you are in the middle of certain functions or system calls, or
> certain particular signals (often SIGFPE), call the C handler with a
> bad set of global pointers or similar. I believe that this is one of
> reasons (perhaps the main one) that some such failures so often cause
> debuggers to be unable to find the stack pointer.
>
> I have tracked a few of those down, and have occasionally identified
> the cause (and even got it fixed!), but it is a murderous task, and
> I know of few other people who have ever succeeded.
>
> > How hard would it be to implement this for the platforms Python supports,
> > rather than for a hypothetical standards-exact platform?
>
> I have seen this effect on OSF/1, IRIX, Solaris, Linux and versions
> of Microsoft Windows. I have never used a modern BSD, haven't used
> HP-UX since release 9, and haven't used Microsoft systems seriously
> in years (though I did hang my new laptop in its GUI fairly easily).
>
> As I say, this isn't so much a platform issue as a circumstance one.
>
>
> Regards,
> Nick Maclaren,
> University of Cambridge Computing Service,
> New Museums Site, Pembroke Street, Cambridge CB2 3QH, England.
> Email: [EMAIL PROTECTED]
> Tel.: +44 1223 334761Fax: +44 1223 334679
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cross-platform math functions?
[Andreas Raab] > I'm curious if there is any interest in the Python community to achieve > better cross-platform math behavior. A quick test[1] shows a > non-surprising difference between the platform implementations. > Question: Is there any interest in changing the behavior to produce > identical results across platforms (for example by utilizing fdlibm > [2])? Since I have need for a set of cross-platform math functions I'll > probably start with a math-compatible fdlibm module (unless somebody has > done that already ;-) Package a Python wrapper and see how popular it becomes. Some reasons against trying to standardize on fdlibm were explained here: http://mail.python.org/pipermail/python-list/2005-July/290164.html Bottom line is I suspect that when it comes to bit-for-bit reproducibility, fewer people care about that x-platform than care about it x-language on the box they use. Nothing wrong with different modules for people with different desires. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
[Tim Peters] >> Speaking of which, I saw no feedback on the proposed patch in >> >> http://mail.python.org/pipermail/python-dev/2006-August/068502.html >> >> so I'll just check that in tomorrow. [Anthony Baxter] > This should also be backported to release24-maint and release23-maint. Let me > know if you can't do the backport... Done in rev 51711 on the 2.5 branch. Done in rev 51715 on the 2.4 branch. Done in rev 51716 on the trunk, although in the LONG_MIN way (which is less obscure, but a more "radical" code change). I don't care about the 2.3 branch, so leaving that to someone who does. Merge rev 51711 from the 2.5 branch. It will generate a conflict on Misc/NEWS. Easiest to revert Misc/NEWS then and just copy/paste the little blurb from 2.5 news at the appropriate place: """ - Overflow checking code in integer division ran afoul of new gcc optimizations. Changed to be more standard-conforming. """ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
On 9/4/06, Nick Maclaren <[EMAIL PROTECTED]> wrote: > Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > > On Mon, 04 Sep 2006 17:24:56 +0100, > > David Hopwood <[EMAIL PROTECTED] > > der.co.uk> wrote: > > >Jean-Paul Calderone wrote: > > >> PyGTK would presumably implement its pending call callback by writing a > > >> byte to a pipe which it is also passing to poll(). > > > > > >But doing that in a signal handler context invokes undefined behaviour > > >according to POSIX. > > > > write(2) is explicitly listed as async-signal safe in IEEE Std 1003.1, 2004. > > Was this changed in a later edition? Otherwise, I don't understand what you > > mean by this. > > Try looking at the C90 or C99 standard, for a start :-( > > NOTHING may safely be done in a real signal handler, except possibly > setting a value of type static volatile sig_atomic_t. And even that > can be problematic. And note that POSIX defers to C on what the C > languages defines. So, even if the function is async-signal-safe, > the code that calls it can't be! I don't believe that is true. It says (or atleast SUSv3 says) that: """ 3.26 Async-Signal-Safe Function A function that may be invoked, without restriction, from signal-catching functions. No function is async-signal-safe unless explicitly described as such.""" Sure, it doesn't give me a warm-fuzzy feeling of knowing why it works, but we can expect that it magically does. My understanding is that threading in general is the same way... Of course that doesn't preclude bugs in the various implementations, but those trump the standards anyway. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
On 9/4/06, Gustavo Carneiro <[EMAIL PROTECTED]> wrote: > Now, we've had this API for a long time already (at least 2.5 > years). I'm pretty sure it works well enough on most *nix systems. > Event if it works 99% of the times, it's way better than *failing* > *100%* of the times, which is what happens now with Python. Failing 99% of the time is as bad as failing 100% of the time, if your goal is to eliminate the short timeout on poll(). 1% is quite a lot, and it would probably have an annoying tendency to trigger repeatedly when the user does certain things (not reproducible by you of course). That said, I do hope we can get 100%, or at least enough nines that we can increase the timeout significantly. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] TRUNK IS UNFROZEN, available for 2.6 work if you are so inclined
On 8/18/06, Georg Brandl <[EMAIL PROTECTED]> wrote: > > I'd like to commit this. It fixes bug 1542051. > > Index: Objects/exceptions.c ... Georg, Did you still want to fix this? I don't remember anything happening with it. I don't see where _PyObject_GC_TRACK is called, so I'm not sure why _PyObject_GC_UNTRACK is necessary. You should probably add the patch to the bug report and we can discuss there. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] no remaining issues blocking 2.5 release
Gustavo, Did you still want this addressed? Anthony and I made some comments on the bug/patch, but nothing has been updated. n -- On 8/15/06, Gustavo Niemeyer <[EMAIL PROTECTED]> wrote: > > If you have issues, respond ASAP! The release candidate is planned to > > be cut this Thursday/Friday. There are only a few more days before > > code freeze. A branch will be made when the release candidate is cut. > > I'd like to see problem #1531862 fixed. The bug is clear and the > fix should be trivial. I can commit a fix tonight, if the subprocess > module author/maintainer is unavailable to check it out. > > -- > Gustavo Niemeyer > http://niemeyer.net > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] 2.5 status
There are 3 bugs currently listed in PEP 356 as blocking: http://python.org/sf/1551432 - __unicode__ breaks on exception classes http://python.org/sf/1550938 - improper exception w/relative import http://python.org/sf/1541697 - sgmllib regexp bug causes hang Does anyone want to fix the sgmlib issue? If not, we should revert this week before c2 is cut. I'm hoping that we will have *no changes* in 2.5 final from c2. Should there be any bugs/patches added to or removed from the list? The buildbots are currently humming along, but I believe all 3 versions (2.4, 2.5, and 2.6) are fine. Test out 2.5c1+ and report all bugs! n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cross-platform math functions?
Tim Peters wrote: > Package a Python wrapper and see how popular it becomes. Some reasons > against trying to standardize on fdlibm were explained here: > >http://mail.python.org/pipermail/python-list/2005-July/290164.html Thanks, these are good points. About speed, do you have any good benchmarks available? In my experience fdlibm is quite reasonable for speed in the context of use by dynamic languages (i.e., counting allocation overheads, lookup and send performance etc) but since I'm not a Python expert I'd appreciate some help with realistic benchmarks. > Bottom line is I suspect that when it comes to bit-for-bit > reproducibility, fewer people care about that x-platform than care > about it x-language on the box they use. Nothing wrong with different > modules for people with different desires. Agreed. Thus my question if someone had already done this ;-) Cheers, - Andreas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
