Re: [Python-Dev] Inconsistency of PyModule_AddObject()
On 28.04.16 01:24, Case Van Horsen wrote: On Wed, Apr 27, 2016 at 11:06 AM, Serhiy Storchaka wrote: I think it is better to have relation with PyModule_AddIntConstant() etc than with PyObject_SetAttrString. My patch doesn't introduce new public function, but changes the behavior of the old function. This needs minimal changes to user code that mostly use PyModule_AddObject() incorrectly (not blaming authors). How will this impact code that uses PyModule_AddObject() correctly? No impact except emitting a deprecation warning at build time. But we can remove a deprecation warning and add it in future release if this is annoying. But are you sure, that your code uses PyModule_AddObject() correctly? Only two modules in the stdlib (_json and _tkinter) used it correctly. Other modules have bugs even in tries to use PyModule_AddObject() correctly for some operations. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
Serhiy Storchaka gmail.com> writes: > No impact except emitting a deprecation warning at build time. But we > can remove a deprecation warning and add it in future release if this is > annoying. > > But are you sure, that your code uses PyModule_AddObject() correctly? > Only two modules in the stdlib (_json and _tkinter) used it correctly. > Other modules have bugs even in tries to use PyModule_AddObject() > correctly for some operations. Could you perhaps stop labeling this as a bug? Usually we are talking about a *single* "leak" that a) does not even show up in Valgrind and b) only occurs under severe memory pressure when the OOM-killer is already waiting. I'm honestly mystified by your terminology and it's beginning to feel that you need to justify this patch at all costs. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
Serhiy Storchaka gmail.com> writes: > But are you sure, that your code uses PyModule_AddObject() correctly? > Only two modules in the stdlib (_json and _tkinter) used it correctly. > Other modules have bugs even in tries to use PyModule_AddObject() > correctly for some operations. For the list, this is the extent of this horrible "bug": diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -5804,8 +5804,7 @@ PyObject_CallObject((PyObject *)&PyDecContext_Type, NULL)); init_basic_context(basic_context_template); Py_INCREF(basic_context_template); -CHECK_INT(PyModule_AddObject(m, "BasicContext", - basic_context_template)); +CHECK_INT(-1); $ valgrind --suppressions=Misc/valgrind-python.supp ./python -c "import decimal" [...] ==16945== LEAK SUMMARY: ==16945==definitely lost: 0 bytes in 0 blocks [...] Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
On Thu, Apr 28, 2016, at 05:05, Stefan Krah wrote: > $ valgrind --suppressions=Misc/valgrind-python.supp ./python -c "import > decimal" > > [...] > ==16945== LEAK SUMMARY: > ==16945==definitely lost: 0 bytes in 0 blocks > Well, the obvious flaw with your test case is that a reference is retained forever in the C static variable basic_context_template. Now, it is arguable that this may be a reasonably common pattern, and that this doesn't actually constitute misuse of the API (the reference count will be wrong, but the object itself is immortal anyway, so it doesn't matter if it's 2 or 1 since it can't be 0 even with correct usage) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
On 28.04.16 11:38, Stefan Krah wrote: Serhiy Storchaka gmail.com> writes: No impact except emitting a deprecation warning at build time. But we can remove a deprecation warning and add it in future release if this is annoying. But are you sure, that your code uses PyModule_AddObject() correctly? Only two modules in the stdlib (_json and _tkinter) used it correctly. Other modules have bugs even in tries to use PyModule_AddObject() correctly for some operations. Could you perhaps stop labeling this as a bug? Usually we are talking about a *single* "leak" that a) does not even show up in Valgrind and b) only occurs under severe memory pressure when the OOM-killer is already waiting. I'm honestly mystified by your terminology and it's beginning to feel that you need to justify this patch at all costs. I say this is a bug because 1. PyModule_AddObject() behavior doesn't match the documentation. 2. Most code that use PyModule_AddObject() doesn't work as intended. Since the bahavior of PyModule_AddObject() contradicts the documentation and is contrintuitive, we can't blame authors in this. I don't say this is a high-impacting bug, I even agree that there is no need to fix the second part in maintained releases. But this is a bug unless you propose different definition for a bug. What can we do with this? 1. Change the documentation of PyModule_AddObject(). I think this is not questionable, and Berker provided a patch in http://bugs.python.org/issue26868 . 2. Update examples in the documentation to correctly handle errors of PyModule_AddObject(). This is more questionable, due to the case (3c) below and because correct error handling code distracts attention from main purpose of examples. 3. One of alternatives: 3a) Fix almost all usages of PyModule_AddObject() in stdlib extension modules. This is hundreds occurrences in over a half-hundred files. 3b) Allow to change the behavior of PyModule_AddObject() to match most authors expectations. This needs to add only one line to switch on new behavior in most files. 3c) Ignore issue. In this case we can not check the result of PyModule_AddObject() at all. But I afraid that correct fixing issues with subinterpreters will need us to return to this issue. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
Random832 fastmail.com> writes: > On Thu, Apr 28, 2016, at 05:05, Stefan Krah wrote: > > $ valgrind --suppressions=Misc/valgrind-python.supp ./python -c "import > > decimal" > > > > [...] > > ==16945== LEAK SUMMARY: > > ==16945==definitely lost: 0 bytes in 0 blocks > > > > Well, the obvious flaw with your test case is that a reference is > retained forever in the C static variable basic_context_template. For actual users of Valgrind this is patently obvious and was pretty much the point of my post. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Needs to install python 3.4.4 in RHEL 6
Hi team, I wanted to install python version 3.4.4 in my RHEL 6 system. Can someone give installation process or any reference link from which I can get required steps and download desire package. Thanks, *Nilesh Date* ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
On Thu, Apr 28, 2016, at 10:11, Stefan Krah wrote: > For actual users of Valgrind this is patently obvious and was > pretty much the point of my post. A more relevant point would be that _decimal does *not* use the API in a way *which would be broken by the proposed change*, regardless of whether the way in which it uses it is subjectively correct or can cause leaks. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
Serhiy Storchaka gmail.com> writes: > 2. Most code that use PyModule_AddObject() doesn't work as intended. > Since the bahavior of PyModule_AddObject() contradicts the documentation > and is contrintuitive, we can't blame authors in this. > > I don't say this is a high-impacting bug, I even agree that there is no > need to fix the second part in maintained releases. But this is a bug > unless you propose different definition for a bug. Why do you think that module authors don't know that? For _decimal, I was aware of the strange behavior. Yes, a single reference can "leak" on failure. The problem is that we don't seem to have any common ground here. Do you accept the following? 1) PyModule_AddObject() can only fail if malloc() fails. a) Normally (for small allocations) this is such a serious problem that the whole application fails anyway. b) Say that you're lucky and the application continues. i) The import fails. In some cases ImportError is caught and a fallback is imported (example _pydecimal). In that case you leak an entire DSO and something small like a single context object. What is the practical difference between the two? ii) The import fails and there's no fallback. Usually the application stops, otherwise DSO+small leak again. iii) Retry the import (I have never seen this): while(1): try: import leftpad except (ImportError, MemoryError): continue break You could have a legitimate leak here, but see a). Module initializations are intricate and boring. I suspect that if we promote wide changes across PyPI packages we'll see more additional segfaults than theoretically plugged memory leaks. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
Random832 fastmail.com> writes: > A more relevant point would be that _decimal does *not* use the API in a > way *which would be broken by the proposed change*, regardless of > whether the way in which it uses it is subjectively correct or can cause > leaks. And the ultimate point is that I don't want to spend about a week per year to evaluate the effect of needless code changes on a highly audited module. And no, this isn't theoretical... Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Needs to install python 3.4.4 in RHEL 6
Hi Nilesh, On Thu, Apr 28, 2016 at 7:00 AM, Nilesh Date wrote: > Hi team, > > I wanted to install python version 3.4.4 in my RHEL 6 system. > Can someone give installation process or any reference link from which I can > get required steps and download desire package. You have a couple of options. Option 1: use software collections [1]. As I vaguely understand it (having never used this myself), the rh-python34 package is supported by Red Hat, and is like any other package for the most part. Looking at that page it does look a bit more complex than option 2 to me, but I've built and installed Python several times over the past few years :) Option 2: compile and install yourself. At a minimum, you'll need a c compiler (gcc, icc, or clang are recommended), and development headers for any extension modules that you require (I'd recommend openssl-devel and readline-devel at the least). Then download the source [2], extract it, and run `cd Python-3.4.4 && ./configure && make profile-opt && make test && sudo make install`. That series of commands will give you python installed in `/usr/local/` that has been compiled with profile-guided optimization (PGO) and has passed the full Python test suite. If any but the last step fails, nothing will have changed on your system. [1] https://www.softwarecollections.org/en/scls/rhscl/rh-python34/ [2] https://www.python.org/downloads/source/ Hope this helps, -- Zach ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
Stefan, could you explain which module you are talking about and why it would cost you a week? What is your responsibility here? --Guido (mobile) On Apr 28, 2016 8:28 AM, "Stefan Krah" wrote: > Random832 fastmail.com> writes: > > A more relevant point would be that _decimal does *not* use the API in a > > way *which would be broken by the proposed change*, regardless of > > whether the way in which it uses it is subjectively correct or can cause > > leaks. > > And the ultimate point is that I don't want to spend about a week per year > to evaluate the effect of needless code changes on a highly audited module. > > And no, this isn't theoretical... > > > Stefan Krah > > > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
On 04/28/2016 08:26 AM, Stefan Krah wrote: Random832 writes: A more relevant point would be that _decimal does *not* use the API in a way *which would be broken by the proposed change*, regardless of whether the way in which it uses it is subjectively correct or can cause leaks. And the ultimate point is that I don't want to spend about a week per year to evaluate the effect of needless code changes on a highly audited module. And no, this isn't theoretical... Considering you have to opt-in to the change, why would this be a big deal for you? Or are you saying you'd rather have the PyModule_AddObject deprecated (without removal?), and a new PyWhatever_Whatever to take it's place? -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Anyone want to lead the sprints at PyCon US 2016?
No one stepped forward to lead the sprints this year, so I will put myself as the sprint leader and lean on everyone else who appears to help. :) On Tue, 5 Apr 2016 at 09:36 Brett Cannon wrote: > The call has started to go out for sprint groups to list themselves > online. Anyone want to specifically lead the core sprint this year? If no > one specifically does then I will sign us up and do my usual thing of > pointing people at the devguide and encourage people to ask questions but > not do a lot of hand-holding (I'm expecting to be busy either working on > GitHub migration stuff or doing other things that I have been neglecting > due to my GitHub migration work). > > -- Forwarded message - > From: Ewa Jodlowska > Date: Mon, 4 Apr 2016 at 07:14 > Subject: [PSF-Community] Sprinting at PyCon US 2016 > To: > > > Are you coming to PyCon US? Have you thought about sprinting? > > The coding Sprints are the hidden gem of PyCon, up to 4 days (June 2-5) of > coding with many Python projects and their maintainers. And if you're > coming to PyCon, taking part in the Sprints is easy! > > You don’t need to change your registration* to join the Sprints. There’s > no additional registration fee, and you even get lunch. You do need to > cover the additional lodging and other meals, but that’s it. If you’ve > booked a room through the PyCon registration system, you'll need to contact > the registration team at pycon2...@cteusa.com as soon as possible to > request the extra nights. The sprinting itself (along with lunch every day) > is free, so your only expenses are your room and other meals. > > If you're interested in what projects will be sprinting, just keep an eye > on the sprints page on the PyCon web site at > https://us.pycon.org/2016/community/sprints/ Be sure to check back, as > groups are being added all the time. > > If you haven't sprinted before, or if you just need to brush up on > sprinting tools and techniques, there will again be an 'Intro to Sprinting' > session the evening of June 1, lead by Shauna Gordon-McKeon and other > members of Python community. To grab a free ticket for this session, just > visit > https://www.eventbrite.com/e/introduction-to-open-source-the-pycon-sprints-tickets-22435151141 > . > > *Please note that conference registration is sold out, but you do not need > a conference registration to come to the Sprints. > > ___ > PSF-Community mailing list > psf-commun...@python.org > https://mail.python.org/mailman/listinfo/psf-community > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Release schedule for Python 3.5.2
I've been holding off on the hope that one or two bugs would get fixes. But those seem to have stalled. So I think it's time that we pushed out a 3.5.2. Maybe announcing a schedule will light a fire under some rumps. I put "Spring 2016" as the release date for 3.5.2 on the 3.5 release schedule PEP. Officially, spring ends--and summer begins--Tuesday June 21 at 12:24am EDT. However on the off chance that the PyCon sprints are productive, I want to hold off until those are done, and maybe give it a couple extra days for the dust to settle. Last sprint day is Sunday June 5th. So, bottom line, the RC will happen during spring, but the final release will technically be during summer. 3.5.2 RC 1 - tag Sat June 11, release Sun June 12 3.5.2 Final - tag Sat June 25, release Sun June 26 Any problems with that? Speak up now. //arry/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
