[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?
I don't know if it is pertinent to this at all, but I raised https://bugs.python.org/issue9 in which the faulthandler module can lead to a segfault inside Py_TRASHCAN_SAFE_BEGIN. Would that be avoided if frameobject.c was changed to use Py_TRASHCAN_BEGIN / END? Duncan. On Tue, 2021-08-17 at 12:00 +0200, Łukasz Langa wrote: > Hi everybody, > I'd like to revive this thread as I feel like we have to do something > here but some consensus is needed first. > > To recap, the current state of things is as follows: > - in March 2000 (d724b23420f) Christian Tismer contributed the > "trashcan" patch that added Py_TRASHCAN_SAFE_BEGIN > and Py_TRASHCAN_SAFE_END macros which allow destroying nested objects > non-recursively. > - in May 2019 (GH-11841 of BPO-35983) Antoine Pitrou merged a change > by Jeroen Demeyer which made Py_TRASHCAN_SAFE_BEGIN/END > (unintentionally?) backwards incompatible; this was released in > Python 3.8.0. > - by the way, GH-11841 introduced a new pair of macros (because they > have different signatures) called simply Py_TRASHCAN_BEGIN and > Py_TRASHCAN_END. > - by that time there was already a follow-up PR open (GH-12607) to > improve backwards compatibility of the macros, as well as introduce > tests for them; this was never merged. > - in Feb 2020 (0fa4f43db08) Victor Stinner removed the trashcan > mechanism from the limited C API (note: not ABI, those are macros) > since it accesses fields of structs not exposed in the limited C API; > this was released in Python 3.9.0. > - in May 2020 Irit noticed that the backwards incompatibility (BPO- > 40608) causes segfaults for C API code that worked fine with Python > 3.7. Using the new macros requires code changes but doesn't crash. > > Now, there are a couple of things we can do here: > Option 1: Finish GH-12607 to fix the old macros, keeping in mind this > will restore compatibility lost with Python 3.8 - 3.10 only for users > of 3.11+ > Option 2: Review and merge GH-20104 that reverts the macro changes > that make old client code segfault -- unclear what else this needs > and again, that would only fix it for users of 3.11+ > Option 3: Abandon GH-12607 and GH-20104, instead declaring the old > macros deprecated for 3.11 and remove them in 3.13 > > I personally agree with Irit, voting +1 for Option 3 since the old > macros were soft-deprecated already by introducing new macros in 3.8, > and more importantly made incompatible with pre-3.8 usage. > > Let's talk on how to proceed. > > - Ł -- Duncan Grisby ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SB6ODKKHANAHDXST2W73RNDZSCZQFOHS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?
On Thu, 2021-08-19 at 11:41 +0100, Irit Katriel wrote: > Well, in order to verify that we fixed the problem you reported we > need to be able to test the fix and see that the problem is gone. > Right now we have two theories of what the cause could be, because > there are two known bugs in that area. But you may have stumbled upon > an unrelated third bug. > > The best scenario is that we close a bpo issue because we can > reproduce the bug and see that it has been fixed. When that's not > possible, there will come a point where we will close it and say "the > code has changed since you observed the problem and we fixed things > that could have caused it. Please open a new issue if you still see > this bug". That's fair. I'll do my best to reproduce it and see which (if either) change resolves it. Duncan. -- Duncan Grisby ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YKKJ4RD376UEC7LYN6BAMHVTK55XHYXU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?
On Tue, 2021-08-17 at 12:14 +0100, Irit Katriel wrote: > > On Tue, Aug 17, 2021 at 12:05 PM Duncan Grisby > wrote: > > I don't know if it is pertinent to this at all, but I > > raised https://bugs.python.org/issue9 in which the faulthandler > > module can lead to a segfault inside Py_TRASHCAN_SAFE_BEGIN. Would > > that be avoided if frameobject.c was changed to use > > Py_TRASHCAN_BEGIN / END? > > > > > We just changed frameobject.c to use the new macros. Can you check? > https://github.com/python/cpython/pull/27683 Thanks. Victor Stinner has now commented on the defect I raised to say that it is actually a bug in the traceback module, in that it should not be using Py_DECREF at all inside a signal handler. The fact that it leads to use of the old trashcan macros is not the cause of the crash that I saw. I therefore don't think it's necessary for me to check with the updated frameobject.c. Do you agree? I can try to reproduce the situation, but I can't just quickly test it because the infrastructure where the crash was seen is now being used for other things. Duncan. -- Duncan Grisby ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VLJ6JJ5T6PHUTKEWONXI2P2FTEOJLUPH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] (no subject)
Hi, I posted this to comp.lang.python, but got no response, so I thought I would consult the wise people here... I have encountered a problem with the re module. I have a multi-threaded program that does lots of regular expression searching, with some relatively complex regular expressions. Occasionally, events can conspire to mean that the re search takes minutes. That's bad enough in and of itself, but the real problem is that the re engine does not release the interpreter lock while it is running. All the other threads are therefore blocked for the entire time it takes to do the regular expression search. Is there any fundamental reason why the re module cannot release the interpreter lock, for at least some of the time it is running? The ideal situation for me would be if it could do most of its work with the lock released, since the software is running on a multi processor machine that could productively do other work while the re is being processed. Failing that, could it at least periodically release the lock to give other threads a chance to run? A quick look at the code in _sre.c suggests that for most of the time, no Python objects are being manipulated, so the interpreter lock could be released. Has anyone tried to do that? Thanks, Duncan. -- -- Duncan Grisby -- -- [EMAIL PROTECTED] -- -- http://www.grisby.org -- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: Regular expressions
On Thursday 24 November, Donovan Baarda wrote: > I don't know if this will help, but in my experience compiling re's > often takes longer than matching them... are you sure that it's the > match and not a compile that is taking a long time? Are you using > pre-compiled re's or are you dynamically generating strings and using > them? It's definitely matching time. The res are all pre-compiled. [...] > > A quick look at the code in _sre.c suggests that for most of the time, > > no Python objects are being manipulated, so the interpreter lock could > > be released. Has anyone tried to do that? > > probably not... not many people would have several-minutes-to-match > re's. > > I suspect it would be do-able... I suggest you put together a patch and > submit it on SF... The thing that scares me about doing that is that there might be single-threadedness assumptions in the code that I don't spot. It's the kind of thing where a patch could appear to work fine, but them mysteriously fail due to some occasional race condition. Does anyone know if there is there any global state in _sre that would prevent it being re-entered, or know for certain that there isn't? Cheers, Duncan. -- -- Duncan Grisby -- -- [EMAIL PROTECTED] -- -- http://www.grisby.org -- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SRE should release the GIL (was: no subject)
On Monday 28 November, Guido van Rossum wrote: > On 11/24/05, Duncan Grisby <[EMAIL PROTECTED]> wrote: > > I have encountered a problem with the re module. I have a > > multi-threaded program that does lots of regular expression searching, > > with some relatively complex regular expressions. Occasionally, events > > can conspire to mean that the re search takes minutes. That's bad > > enough in and of itself, but the real problem is that the re engine > > does not release the interpreter lock while it is running. All the > > other threads are therefore blocked for the entire time it takes to do > > the regular expression search. > > Rather than trying to fight the GIL, I suggest that you let a regex > expert look at your regex(es) and the input that causes the long > running times. As Fredrik suggested, certain patterns are just > inefficient but can be rewritten more efficiently. There are plenty of > regex experts on c.l.py. Part of the problem is certainly inefficient regexes, and we have improved things to some extent by changing some of them. Unfortunately, the regexes come from user input, so we can't be certain that our users aren't going to do stupid things. It's not too bad if a stupid regex slows things down for a bit, but it is bad if it causes the whole application to freeze for minutes at a time. > Unless you have a multi-CPU box, the performance of your app isn't > going to improve by releasing the GIL -- it only affects the > responsiveness of other threads. We do have a multi-CPU box. Even with good regexes, regex matching takes up a significant proportion of the time spent processing in our application, so being able to release the GIL will hopefully increase performance overall as well as increasing responsiveness. We are currently testing our application with the patch to sre that Eric posted. Once we get on to some performance tests, we'll post the results of whether releasing the GIL does make a measurable difference for us. Cheers, Duncan. -- -- Duncan Grisby -- -- [EMAIL PROTECTED] -- -- http://www.grisby.org -- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
