Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Zitat von Christian Heimes : On 30.08.2014 17:22, Alex Gaynor wrote: The Windows certificate store is used by ``load_default_certs``: * https://github.com/python/cpython/blob/master/Lib/ssl.py#L379-L381 * https://docs.python.org/3.4/library/ssl.html#ssl.enum_certificates The Windows part of load_default_certs() has one major flaw: it can only load certificates that are already in Windows's cert store. However Windows comes only with a small set of default certs and downloads more certs on demand. In order to trigger a download Python or OpenSSL would have to use the Windows API to verify root certificates. It's better than you think. Vista+ has a weekly prefetching procedure that should assure that virtually all root certificates are available: http://support.microsoft.com/kb/931125/en-us BTW, it's patented: http://www.google.de/patents/US6816900 Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
Zitat von Tony Kelman : A maintainer has volunteered. Others will help. Can any core developers please begin reviewing some of his patches? Unfortunately, every attempt to review these patches has failed for me, every time. In the last iteration of an attempt to add mingw64 support, I had asked contributors to also provide instructions on how to use these patches, and haven't received any instructions that actually worked. I'm hesitant to add code that I cannot verify as actually working. I guess anybody else reviewing these patches ran into similar problems (I know some other core developers have tried reviewing them as well, others have stated here that they are unable to review the patches). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] VAX NaN evaluations
Quoting John Klos : The nice Python folks who were at SCALE in Los Angeles last year gave me a Python t-shirt for showing Python working on m68k and for suggesting that I'd get it working on VAX. With libffi support for VAX from Miod Vallat, this is now possible. I'm sorry to hear that - you might have been wasting your time (but then, perhaps not). We decided a while ago that the regular Python releases will not support VAX/VMS any longer. So any code you write has zero chance of being integrated into Python (the same holds for m68k code, BTW). That said, you are free to maintain your own branch of Python. In that branch, you can make whatever changes you consider necessary, but you will be on your own. Kind regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is pygettext.py deprecated?
Quoting "A.M. Kuchling" : GNU xgettext didn't understand any languages other than C at one time, so pygettext.py was written to handle Python code but not C code. That's not the only reason. Another two reasons are that a) it works on Windows (for xgettext, you'll have to install Cygwin, which some consider a bit heavy - if all you need is xgettext) b) it comes with Python (interesting on Unix systems that don't come with a pre-built xgettext; less relevant now as most Unix systems are Linux these days) I see no real harm done by keeping (and possibly fixing) pygettext. I also see little harm in removing it, although I guess that some people might still rely on it. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 0404 and VS 2010
Quoting Barry Warsaw : On Nov 20, 2013, at 09:52 PM, Christian Tismer wrote: Many customers are forced to stick with Python 2.X because of other products, but they require a Python 2.X version which can be compiled using Visual Studio 2010 or better. This is considered an improvement and not a bug fix, where I disagree. I'm not so sure about that. Python 2.7 can still get patches to help extend its useful life by allowing it to be built with newer compiler suites. I believe this has already been done for various Linux compilers. I see no non-technical reason why Python 2.7 can't be taught how to build with VS 2010 or newer. Unfortunately, there are severe *technical* reasons that make this a challenge. If they wouldn't exist, we would have done this long ago. The problem is in the ABI, and the question which msvcrt python27.dll links with. With official support for VS 2010, there would be two ABI-incompatible releases of Python on Windows; entirely different from supporting a new version of gcc (say), as newer gccs will typically be binary-compatible with older releases (I admit I don't understand the ABI compatibility on OSX). So adding VS 2010 build files to the 2.7 tree would be fine with me, but I doubt users are helped with that - they want a binary release that uses them. Releasing this as "Python 2.7", with a "python27.dll" would lead right to DLL hell. I could imagine having a separate release that is called "Python 2.7 for Visual Studio 2010" (and then another one called "Python 2.7 for Visual Studio 2013"). They could give different names to the Python DLL, e.g. py27cr100.dll and py27cr110.dll. Whether this would be a good idea or not, I don't know. It would create separate ecosystems for different releases of Python 2.7 for different CRTs. Package authors would have to create multiple binary releases of the same modules for Windows, and upload them to PyPI. pip would have to learn to download the right one, depending on what build of Python 2.7 is running. I'm skeptical that using "Python 2.8" as the release name would help in the long run. Assuming this is done for VS 2010, then "Python 2.9" will have to be released for binary compatibility with VS 2013, and "Python 2.10" for VS 2016, assuming Python 2.7 is then still around by that time. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 0404 and VS 2010
Quoting Nick Coghlan : Another alternative I'd prefer to an ABI version bump: backporting the "C runtime independence" aspects of the stable ABI to Python 2.7. That sounds doable. If we provided a "python2.dll", would could make the header files using the "restricted API" by default if Python is compiled with VS 2010. Extension builders could then regularly compile their extensions with VS 2010, or VS 2013, despite Python itself being linked with the VS 2008 CRT. Anybody releasing such binaries would either have to target Python 2.7.7, or distribute "python2.dll" along with the binary. It should actually be possible to provide a "python27vs2010.msi" upgrade package that only deploys the python2.dll and the updated header files, on top of an existing Python 2.7 installation. I had originally planned to support the "stable ABI" in Python 2, but the PEP lagged, and I couldn't finish it in time for the 2.7 release. If Chris could contribute to make this happen, it would be much appreciated. Regards, Martin P.S. Thinking about this, there are some issues. The "restricted API" hides the object layout of all objects, in particular of type objects. Adding the PEP 384 API (PyType_FromSpec) might be a bit heavy for 2.7. So it might by better to provide a "py27compat.dll" instead which does not hide the structures (as they won't change during the remaining life of 2.7), but only hides any APIs and macros that directly expose CRT functionality. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 0404 and VS 2010
Quoting Greg Ewing : Concerning the version number, I thought the intention of PEP 404 was simply to say that the PSF would not be releasing anything called Python 2.8, not to forbid anyone *else* from doing so. Or am I wrong about that? That's correct. If I'm right, there's nothing stopping Christian from releasing Stackless Python 2.8 with whatever improvements he wants. "Nothing stopping" is exactly right. People still don't need to like it. Barry wishes there was something stopping him, and be it a lawyer invoking trademark law. If "2.8" was just a version number of Stackless Python not related to Python version (like PyPy's version number currently being 2.2), protest would be much less. People fear that releasing Stackless Python 2.8 would create the impression that even CPython has continued development, and it might require significant PR efforts to educate people. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 0404 and VS 2010
Quoting Christian Tismer : Can I rely on PEP 404 that the "Python 2.8" namespace never will clash with CPython? This question still hasn't been answered (AFAICT). So let me try to answer it, and apologies upfront for being picky. First, I don't understand the question: What is the "Python 2.8" *namespace*? I try to rephrase the question: # May I rely on PEP 404 that CPython will never make a Python 2.8 release? To that, the clear answer is YES. A natural follow-up question then is # May I use the name "Python 2.8" for my own products? To that, the answer is "ONLY IF YOU GET PERMISSION". This is really off-topic for python-dev, but you would have to ask the PSF trademark committee for permission to use that name as a product name. IIUC, the current policy is that you may use "Python" if it a) is related to the Python programming language, and b) is somehow qualified (e.g. "Stackless Python"). To use just "Python", you need permission (and I suspect that this might not be granted). HTH, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 0404 and VS 2010
Quoting Christian Heimes : What about the CAPI functions like PyFile_FromFile() and PyFile_AsFile() that take a FILE* as argument? They are unable in the stable ABI, and would be unavailable in py27compat.dll. Modules using them would have to be rewritten to not use them anymore, or continue to be compiled with VS 2008. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] py.ini documentation improvement
Quoting Terry Reedy : 'Two .ini files will be searched by the launcher' sort of implies to me that the files exist. On my Win7 machine, echo %LOCALAPPDATA% returns C:\Users\Terry\AppData\Local. If I go to Users/Terry with Explorer, there is no AppData. (Same with admin user.) Are you sure about this? What happens if you type "%LOCALAPPDATA%" (without quotes) into the explorer's address bar? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bug? http.client assumes iso-8859-1 encoding of HTTP headers
Quoting Chris Angelico : I'm pretty sure that server is in violation of the spec, so all bets are off as to what any other server will do. If you know you're dealing with this one server, you can probably hack around this, but I don't think it belongs in core code. Unless, of course, I'm completely wrong about the spec, or if there's a de facto spec that lots of servers follow, in which case maybe it would be worth doing. It would be possible to support this better by using "ascii" with "surrogateescape" when receiving the redirect, and using the same for all URLs coming into http.client. This would implement a best-effort strategy at preserving the bogus URL, and still maintain the notion that URLs are text (with the other path being to also allow bytes as URLs, and always parsing Location as bytes). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A misredirected ticket link in hg.python.org/cpython
Quoting Vajrasky Kok : Go to any commit link in hg.python.org/cpython, for example http://hg.python.org/cpython/rev/d11ca14c9a61. You have this commit message: Issue #6815: os.path.expandvars() now supports non-ASCII Unicode environment variables names and values. [#6815] The [#6815] link is going to http://bugs.python.org/6815. If you follow that link, it redirects to http://legacy.python.org/sf/ and you get message: You did not provide a report number. The link should be http://bugs.python.org/issue6815. The bug really is on www.python.org, which (now) redirects /sf/6815 to http://legacy.python.org/sf/, i.e. it drops the issue number there. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final?
Quoting Nick Coghlan : If you don't want to do an rc3 despite the cherry picked changes since rc2, then you need to make it easy for people to test the changes directly from the release branch. An opaque intermittently updated tarball is not acceptable when none of our infrastructure is designed to work that way. I was OK with just the tarball when I assumed you would an rc3 if non-trivial defects were found in rc2. If that's not the case, then we *need* a public mirror of your release clone. Another rc or not - I expect to see a 3.4.1 release *really* shortly after the 3.4.0 release. So except for issues where Python does not work at all, any glitches that remain in the code can be fixed in the bug fix release. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP URLs
Quoting Paul Moore : On 13 March 2014 13:28, Barry Warsaw wrote: http://legacy.python.org/dev/peps/pep-0274/ Are these "legacy.python.org" URLs the canonical locations for PEPs these days? The main page redirects there, but I'd have hoped there would be a www.python.org location... http://www.python.org/dev/peps/pep-0274/ continues to work (as I thought you said yourself), so there *is* a www.python.org location. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements
Quoting Victor Stinner : The drawback is that applications would be benefit immediatly from this work, they should be modified to use the new module. But usually, developers who care of security are able to do these modifications. I think asking developers to make significant modifications to their code is besides the point of the PEP. However, if they are willing to make changes, I'd still recommend that they port their code to Python 3, as that is the better long-term investment. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Appeal for reviews
Quoting "Stephen J. Turnbull" : Your modesty is not in question. :-) An explicit statement that "These are ready" allows an experienced developer to give you feedback not only about whether the patches are in fact ready, but whether your judgment about patches is ready for commit privileges -- in Python it's desirable that code be "Pythonic" as well as correct and clean. For a regular patch, I think this is asking to much. The default assumption should be that the patch is finished, and not work-in-progress. If it was, people typically indicate so. But I then regularly ask that the tracker is not abused as a version control system to develop a change; it's easy enough to use a Mercurial clone for that. For gaining commit access, it's really more important that the patch is factually finished, than that it's author believes it to. If people get it right the first time often enough, they get commit access. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7.7. on Windows
Quoting "Stephen J. Turnbull" : Mike Miller writes: > However, this bug has been shitcanned for a decade. This is the > last chance to fix this bug in a branch that's going to be > supported until 2020! Probably. I'm not convinced. But that doesn't really matter. Your bigger concern is the deafening silence from the senior committers, which suggests you're going to need to come up with a stronger argument. Maybe there's a relevant CVE? *My* silence is easy to explain: I don't maintain Python 2.7 any longer. So my view doesn't really matter (although I do support Steve's position). In any case, I think Mike is following a lost cause. If the maintainer in charge (i.e. Steve) doesn't want to make a specific change, the only way to force him would be from "higher up", which could be the release manager, the BDFL, or a PEP. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
Zitat von Steven D'Aprano : * Having a build-time option to restrict all strings to ASCII-only. (I think what they mean by that is that strings will be like Python 2 strings, ASCII-plus-arbitrary-bytes, not actually ASCII.) An ASCII-plus-arbitrary-bytes type called "str" would prevent claiming "Python 3.4 compatibility" for sure. Restricting strings to ASCII (as Chris apparently actually suggested) would allow to claim compatibility with a stretch: existing Python code might not run on such an implementation. However, since a lot of existing Python code wouldn't run on MicroPython, anyway, one might claim to implement a Python 3.4 subset. * Implementing Unicode internally as UTF-8, and giving up O(1) indexing operations. Would either of these trade-offs be acceptable while still claiming "Python 3.4 compatibility"? My own feeling is that O(1) string indexing operations are a quality of implementation issue, not a deal breaker to call it a Python. I can't see any requirement in the docs that str[n] must take O(1) time, but perhaps I have missed something. I agree. It's an open question whether such an implementation would be practical, both in terms of existing Python code, and in terms of existing C extension modules that people might want to port to MicroPython. There are more things to consider for the internal implementation, in particular how the string length is implemented. Several alternatives exist: 1. store the UTF-8 length (i.e. memory size) 2. store the number of code points (i.e. Python len()) 3. store both 4. store neither, but use null termination instead Variant 3 is most run-time efficient, but could easily use 8 bytes just for the length, which could outweigh the storage of the actual data. Variants 1 and 2 lose on some operations (1 loses on computing len(), 2 loses on string concatenation). 3 would add the restriction of not allowing U+ in a string (which would be reasonable IMO), and make all length computations inefficient. However, it wouldn't be worse than standard C. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The end of 2.7
Quoting Benjamin Peterson : This means we need to talk about how many more 2.7 releases there are going to be. At the release of 2.7.0, I thought we promised 5 years of bugfix maintenance, but my memory may be fuddled. I'd like to promote the idea to abandon 2.7 bug fix releases earlier than that, e.g. along with the release of 3.4. My recollection is that "we" didn't actually promise any specific time frame; I recall that Guido said that Python 2.7 would be supported "indefinitely", which is not "infinitely" [1]. The Whats New says [2] """It’s very likely the 2.7 release will have a longer period of maintenance compared to earlier 2.x versions.""" which explicitly refuses to set a date. Of course, individual committers may have promised a more specific policy publicly in the past. Since Christian asked: I'll likely continue to make binary releases for Windows as along as Benjamin declares releases to be bug fix releases. However, it will become increasingly difficult for users to actually use these releases to build extension modules since Microsoft decided to take VS 2008 Express offline (VS 2008 remains available to MSDN subscribers; getting it from a store might also be difficult in 2014). I wonder whether the burden of maintaining three branches for bug fixes (2.7, 3.3, default) and three more for security fixes (2.6, 3.1, 3.2) is really sustainable for committers. I wouldn't want to back off wrt. security fixes, and 2.6 will soon fall out of the promised 5 years (after the initial release). However, stopping to accept bug fixes for 2.7 would IMO significantly reduce the load for committers - it would certainly continue to get security fixes, and (for the time being) "indefinitely" so. Wrt. to the 3.x migration rate: I think this is a self-fulfilling prophecy. Migration rate will certainly increase once we announce an end of 2.7, and then again when the end is actually reached. I'm doubtful with respect to a community-managed ongoing 2.7 bug fix release (i.e. I doubt that it will happen); the same was discussed for a next 2.x feature release, and it hasn't happened. OTOH, it is very likely that people will publish their own patches to 2.7 throughout the net, just as the Linux distributions already do. It may even happen that some volunteer offers to publish a combined repository for such patches, with various members of the community having write access to such a repository (but no formal releases coming out of that). Regards, Martin [1] http://mail.python.org/pipermail/python-dev/2009-November/093651.html [2] http://docs.python.org/dev/whatsnew/2.7.html#the-future-for-python-2-x ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The end of 2.7
Martin, you guys are shooting yourself in a foot. Almost noone uses python 3 in production, even at pycon, which is the more progressive crowd. There is a giant group of people using python that are not as vocal. While I bet some are using Python 3, Python 2 is incredibly popular for the "long tail" of libraries and applications. How much is 2.7 a burden? There are no major changes and it's pretty cool to consider it "done". Indeed - hence I think it is just fine to stop applying bug fixes to it, as well. People for whom it works fine today apparently don't run into any significant bugs. They can happily continue to use it as-is for ten or more years. It will not go away just when we reduce changes to security fixes. It will remain available for download, the documentation will keep being online, people can continue to ask questions about it on python-list, and continue to get answers. Stopping to apply bug fixes does not really *end* Python 2.7. It's only that people who *do* run into bugs don't have the option anymore that we will eventually publish a fixed release. Their options reduce to - port to 3.x (particularly interesting if Python 3.x *already* fixed it) - find a work-around - maintain a bug fix locally - do something else entirely (like abandoning Python) People deserve to know our plans, so we really need to agree on them and then announce them (see PEP 404). However, people (IMO) have no right to expect us to maintain Python 2.7 until they migrate to 3.x. If we would do that, they will never migrate. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The end of 2.7
Quoting Lennart Regebro : On Sun, Apr 7, 2013 at 7:11 AM, wrote: Wrt. to the 3.x migration rate: I think this is a self-fulfilling prophecy. Migration rate will certainly increase once we announce an end of 2.7, and then again when the end is actually reached. Well... People are in general *stuck* on Python 2. They are not staying because they want to. So I'm not so sure migration rate will increase because an end is announced or reached. I assume you say that because people rely on libraries that haven't been ported (correct me if there are other reasons to be stuck). With an announced end-of-life, I'm certain that migration rate will increase, because people will now urge their suppliers, pointing to the announcement. With Benjamin's proposed schedule, they would still have two years for their suppliers to act. Even under my proposed schedule, there would be plenty of time. Also, this is all free software (at least most of it). Nobody can *really* be stuck on a not-ported dependency, as they could always port it themselves, and even fork if the developer refuses to integrate the port (and you know that this actually happens). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem building Python 2.7.5 with separate sysroot
Quoting Paul Smith : My case may be unusual but even in a more formal cross-compilation environment it's not good to add /usr/include/..., or base such a decision on the behavior of the _build_ system. The offical procedure to cover "unusual" cases is to edit Modules/Setup. If you are not happy with the way in which modules are build, you can override all flags on a per-module basis there. It seems to me (keeping with the theme of this mailing list) that the add_multiarch_paths() function in setup.py is not right. Well, ..., yes. For the last two decades, the build process of Python was *always* wrong, and it always ever will be, in the sense that it doesn't support all cases that people come up with. The only way to deal with this, unfortunately, is to patch the build process for each new use case discovered. As Ned says: if you come up with a patch, don't hesitate to post it to the tracker. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Building a Faster Python
Quoting Steve Dower : As a Windows user, it makes me wonder if compiling with the latest version of the Microsoft compiler would improve things similarly? I'd expect to see some improvement, based solely on the bugs fixed recently by the optimizer team. No idea how much, but I know that Martin builds with PGO, and that's also been improved recently. I would not expect to see any improvement, unfortunately. I used to build with PGO, but the recent VS releases miscompiled code, which was very hard to track down. So PGO has lost in my view for at least 3 VS releases :-( (Though updating project files to that is almost certainly a bigger project than the gcc update.) No idea what the gcc update involved, but this is only accurate if it took less than two minutes :-) I've upgraded the CPython projects before and everything worked fine. Still, we couldn't release 2.7 with any other compiler because of the CRT dependency. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Our failure at handling GSoC students
Zitat von Antoine Pitrou : One cruel example is the set of PEP 3121 / PEP 384 refactorings done by Robin Schreiber: I personally dont consider it failed, yet. I still plan to integrate them, hopefully for 3.4. Robin has produced many patches that seem to reach the stated goal (refactor C extension modules to take advantage of the latest PEPs about module initialization and extension types definition). Unfortunately, tackling both goals at the same time produces big patches with a lot of churn; and it is also not obvious the PEP 384 refactoring is useful for the stdlib (while the PEP 3121 refactoring definitely is). Choice of supporting PEP 384 was deliberate. It will change all types into heap types, which is useful for multiple-interpreter support and GC. What didn't produce an alarm during Robin's work is that GSoC work is done in private. It wasn't really done in private. Robin posted to python-dev, anybody who would have been interested could have joined discussions. It is also likely that the mentor gets overworked after the GSoC period is over, is unable to finalize the patch and push it, and other core devs have a hard time catching up on the work and don't know what the shortcomings are. It's indeed unfortunate that RL interfered with my Python contributions. I apologize for that. However, anybody who wanted to catch up could have contacted Robin or myself. As overworked as we all are, nobody did. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Our failure at handling GSoC students
Zitat von Eli Bendersky : I would like to point out something that stands out in this list of issues: such a method of producing dozens of patches simultaneously is extremely unwise, unless there's a crucial piece of history I'm missing. It is much more prudent to start with one or two exemplary modules, and if those fully pass code review, send out patches for others. The reason is obvious - code review may turn up problems or requests for change. Going backwards to modify 57 patches is not something anyone would want to do. Robin did exactly that: submit a few patches first, receive feedback, submit more patches. At the end of the project,he submitted his entire work. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Our failure at handling GSoC students
Zitat von Lennart Regebro : On Tue, Aug 6, 2013 at 9:26 PM, Antoine Pitrou wrote: What didn't produce an alarm during Robin's work is that GSoC work is done in private. Why is it done in private? It wasn't really done in private, not more than any other contribution. A PEP was accepted before the project even started. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Make str/bytes hash algorithm pluggable?
Quoting Victor Stinner : I still fail to understand the real impact of a hash DoS compared to other kinds of DoS. I think the key question is: how many attacking nodes do you need to control to effectively make some system deny service. A threat is bigger if you can do it in 10 requests/s from a single host, instead of needing 10,000 hosts, each making 1000 request/s. With the hash DoS, the threat is that if you manage to fill some dictionary with colliding keys, then each lookup will take a very long time, and you might arrange to put many lookups into a single HTTP request. So a single HTTP request might get very costly CPU-wise. Whether this is a serious threat or not depends on what other threats the system being attacked is vulnerable to. Maybe there is something even simpler, or maybe the hash attack is the only hope of bringing the system to its knees. IMO, the hash attack is particularly tricky since it is very easy to argue and very difficult to demonstrate. So it can result in fear and uncertainty very easily, causing people to overreact just so that they won't be accused of inactivity. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reduce memory footprint of Python
Quoting Benjamin Peterson : If you know that your application uses a lot of memory, it is interesting to sometimes (when the application is idle) try to release some bytes to not use all the system memory. On embedded devices, memory is expensive and very limited. Each byte is important :-) How many embedded systems are running Python? And of those, how many use the linecache module? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reduce memory footprint of Python
Quoting Victor Stinner : Slowly, I'm trying to see if it would be possible to reduce the memory footprint of Python using the tracemalloc module. [...] Should I open a separated issue for each idea to track them in the bug tracker, or a global issue? There is a third alternative which I would recommend: not open tracker issues at all - unless you can also offer a patch. The things you find are not bugs per se, not even "issues". It is fine and applaudable that you look into this, but other people may have other priorities (like reimplementing the hash function of string objects). So if you remember that there is a potential for optimization, that may be enough for the moment. Or share it on python-dev (as you do below); people may be intrigued to look into this further, or ignore it. It's easy to ignore a posting to python-dev, but more difficult to ignore an issue on the tracker (*something* should be done about it, e.g. close with no action). First, I noticed that linecache can allocate more than 2 MB. What do you think of adding a registry of "clear cache" functions? For exemple, re.purge() and linecache.clearcache(). gc.collect() clears free lists. I don't know if gc.collect() should be related to this new registy (clear all caches) or not. I'm -1 on this idea. There are some "canonical" events that could trigger clearance of caches, namely - out-of-memory situations - OS signals indicating memory pressure While these sound interesting in theory, they fail in practice. For example, they are very difficult to test. The dictionary of interned Unicode strings can be large: up to 1.5 MB (with +30,000 strings). Just the dictionary, excluding size of strings. Is the size normal or not? Using tracemalloc, this dictionary is usually to largest memory block. I'd check the contents of the dictionary. How many strings are in there; how many of these are identifiers; how many have more than one outside reference; how many are immortal? If there is a lot of strings that are not identifiers, some code possibly abuses interning, and should use its own dictionary instead. For the refcount-1 mortal identifiers, I'd trace back where they are stored, and check if many of them originate from the same module. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reduce memory footprint of Python
Quoting Antoine Pitrou : The linecache module is used implicitly by the traceback and warnings module, so perhaps quite a bit more than one would imagine :-) I think limiting the linecache cache size would be good enough. So what specific limit would you suggest? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r51525 - in python/trunk:Lib/test/test_float.py Objects/floatobject.c
Zitat von Tim Peters <[EMAIL PROTECTED]>: > Huh. It's been a (mildly controversial, but intentional all the same) > feature that Python tries to raise raise OverflowError on overflowing > libm operations. Doesn't work all that well, since there's no > consistency across platforms about when libm sets errno, or to what > (when it does). The other question is whether pow() is guaranteed to return inf when it sets errno to ERANGE; I believe there is no such guarantee. So I think the change should be reverted. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can LOAD_GLOBAL be optimized to a simple arraylookup?
Zitat von "K.S.Sreeram" <[EMAIL PROTECTED]>: > How about this approach: > (disclaimer: this is just a rough sketch!) This is actually the problem. There are many fine details which can affect performance and correctness. So about there are only two sensible ideas to treat such ideas: either implement them, or ignore them. If you think your approach could work, please try to implement it. It's not that anybody is objecting the goal; there is just debate about the feasibility of the implementation. So if you can come up with an implementation, we are in a much better position to praise or criticise the approach: it then becomes possible to study whether the implementation is really compatible with the current Python, and whether it really does improve performance. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] MSI: Remove dependency from win32com.client module (issue4080047)
What's the rationale of this change? It certainly doesn't remove the dependency from win32com.client (i.e. the code continues to depend on win32com). http://codereview.appspot.com/4080047/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] MSI: Remove dependency from win32com.client module (issue4080047)
On 2011/02/01 07:22:57, techtonik wrote: It removes the dependency from msi.py making it easier to do the rest in subsequent patches. What rest specifically? Why are the msilib changes needed for that? http://codereview.appspot.com/4080047/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] MSI: Remove dependency from win32com.client module (issue4080047)
On 2011/02/02 07:14:17, techtonik wrote: On 2011/02/01 19:50:23, Martin v. Löwis wrote: > On 2011/02/01 07:22:57, techtonik wrote: > > It removes the dependency from msi.py making it easier to do the rest in > > subsequent patches. > > What rest specifically? Why are the msilib changes needed for that? The rest is to use ctypes, so that no external packages will be required to build Python installer. Ah. That shouldn't be done. If anything is to be done, the builtin msilib can be considered, instead. Given the choice of using either ctypes or an external package, I prefer the external package. http://codereview.appspot.com/4080047/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] SF tracker closed now
We are starting with the tracker conversion. Please stop using the bug trackers (either SF or roundup) until further notice. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] for loop with if filter
I started thinking about itertools when I saw this then I realised that your question was about changing the syntax to produce fewer lines of code rather than writing more effiicient code.. seemed like a case where you could use ifilter. //Martin are talking about cvhanging the syntax rQuoting Gustavo Carneiro <[EMAIL PROTECTED]>: > I am finding myself often doing for loops over a subset of a list, like: > > for r in results: > if r.numNodes != numNodes: > continue > # do something with r > > It would be nice if the plain for loop was as flexible as list > comprehensions and allowed an optional if clause, like this: > > for r in results if r.numNodes == numNodes: > # do something with r > > Has this idea come up before? Does anyone else like this idea? > > -- > Gustavo J. A. M. Carneiro > INESC Porto, Telecommunications and Multimedia Unit > "The universe is always one step beyond logic." -- Frank Herbert > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] bigmemtests for really big memory too slow
I benchmarked some of the bigmemtests when run with -M 80G. They run really slow, because they try to use all available memory, and then take a lot of time processing it. Here are some runtimes: test_capitalize (test.test_bigmem.StrTest) ... ok (420.490846s) test_center (test.test_bigmem.StrTest) ... ok (149.431523s) test_compare (test.test_bigmem.StrTest) ... ok (200.181986s) test_concat (test.test_bigmem.StrTest) ... ok (154.282903s) test_contains (test.test_bigmem.StrTest) ... ok (173.960073s) test_count (test.test_bigmem.StrTest) ... ok (186.799731s) test_encode (test.test_bigmem.StrTest) ... ok (53.752823s) test_encode_ascii (test.test_bigmem.StrTest) ... ok (8.421414s) test_encode_raw_unicode_escape (test.test_bigmem.StrTest) ... ok (3.752774s) test_encode_utf32 (test.test_bigmem.StrTest) ... ok (9.732829s) test_encode_utf7 (test.test_bigmem.StrTest) ... ok (4.998805s) test_endswith (test.test_bigmem.StrTest) ... ok (208.022452s) test_expandtabs (test.test_bigmem.StrTest) ... ok (614.490436s) test_find (test.test_bigmem.StrTest) ... ok (230.722848s) test_format (test.test_bigmem.StrTest) ... ok (407.471929s) test_hash (test.test_bigmem.StrTest) ... ok (325.906271s) In the test suite, we have the bigmemtest and precisionbigmemtest decorators. I think bigmemtest cases should all be changed to precisionbigmemtest, giving sizes of just above 2**31. With that change, the runtime for test_capitalize would go down to 42s. What do you think? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Heads up: Apple llvm gcc 4.2 miscompiles PEP 393
The gcc that Apple ships with the Lion SDK (not sure what Xcode
version that is)
miscompiles Python now. I've reported this to Apple as bug 10143715;
not sure whether
there is a public link to this bug report.
In essence, the code
typedef struct {
long length;
long hash;
int state;
int *wstr;
} PyASCIIObject;
typedef struct {
PyASCIIObject _base;
long utf8_length;
char *utf8;
long wstr_length;
} PyCompactUnicodeObject;
void *_PyUnicode_compact_data(void *unicode) {
return PyASCIIObject*)unicode)->state & 0x20) ?
((void*)((PyASCIIObject*)(unicode) + 1)) :
((void*)((PyCompactUnicodeObject*)(unicode) + 1)));
}
miscompiles (with -O2 -fomit-frame-pointer) to
__PyUnicode_compact_data:
Leh_func_begin1:
leaq32(%rdi), %rax
ret
The compiler version is
gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)
This unconditionally assumes that sizeof(PyASCIIObject) needs to be
added to unicode, independent of whether the state bit is set or not.
I'm not aware of a work-around in the code. My work-around is to use gcc-4.0,
which is still available on my system from an earlier Xcode installation
(in /Developer-3.2.6)
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Unicode exception indexing
There is a backwards compatibility issue with PEP 393 and Unicode exceptions: the start and end indices: are they Py_UNICODE indices, or code point indices? On the one hand, these indices are used in formatting error messages such as "codec can't encode character \u%04x in position %d", suggesting they are regular indices into the string (counting code points). On the other hand, they are used by error handlers to lookup the character, and existing error handlers (including the ones we have now) use PyUnicode_AsUnicode to find the character. This suggests that the indices should be Py_UNICODE indices, for compatibility (and they currently do work in this way). The indices can only be different if the string is an UCS-4 string, and Py_UNICODE is a two-byte type (i.e. on Windows). So what should it be? As a compromise, it would be possible to convert between these indices, by counting the non-BMP characters that precede the index if the indices might differ. That would be expensive to compute, but provide backwards compatibility to the C API. It's less clear what backwards compatibility to Python code would require - most likely, people would use the indices for slicing operations (rather than performing an UTF-16 conversion and performing indexing on that). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
Zitat von Victor Stinner : The current implementation of dict uses a linked-list. [...] Tell me if I am wrong. At least with this statement, you are wrong: the current implementation does *not* use a linked-list. Instead, it uses open addressing. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hash collision security issue (now public)
(Well, technically, you could use trees or some other O log n data structure as a fallback once you have too many collisions, for some value of "too many". Seems a bit wasteful for the purpose, though.) I don't think that would be wasteful. You wouldn't just use the tree for the case of too many collisions, but for any collision. You might special-case the case of a single key, i.e. start using the tree only if there is a collision. The issue is not the effort, but the need to support ordering if you want to use trees. So you might restrict this to dicts that have only str keys (which in practice should never have any collision, unless it's a deliberate setup). I'd use the tagged-pointer trick to determine whether a key is an object pointer (tag 0) or an AVL tree (tag 1). So in the common case of interned strings, the comparison for pointer equality (which is the normal case if the keys are interned) will succeed quickly; if pointer comparison fails, check the tag bit. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python as a Metro-style App
Zitat von Benjamin Peterson :
2012/1/7 "Martin v. Löwis" :
I just tried porting Python as a Metro (Windows 8) App, and failed.
Is this required for Python to run on Windows 8?
No. Existing applications ("desktop applications") will continue to work
unmodified. Metro-style apps are primarily intended for smart phones and
tablet PCs, and will be distributed through the Windows app store. The
current VS prerelease supports both Intel and ARM processors for Apps.
A related question is whether Python will compile unmodified with Visual
Studio 11. Although I had some difficulties with that also so far, I expect
that this will ultimately work (although not unmodified - the project files
need to be updated, as will the packaging process).
A then-related question is whether Python 3.3 should be compiled with Visual
Studio 11. I'd still be in favor of that, provided Microsoft manages
to release
that soon enough.
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python as a Metro-style App
Zitat von Eli Bendersky : A then-related question is whether Python 3.3 should be compiled with Visual Studio 11. I'd still be in favor of that, provided Microsoft manages to release that soon enough. Martin, I assume you mean the Express version of Visual Studio 11 here, right? *Here*, I mean "Visual Studio 11, any edition". I don't think the edition matters for determining what version the project files have - any edition will be able to read the project files, Express or not. If you are specifically asking whether I would make the release of the express edition a prerequisite to releasing Python: no, I wouldn't. I would expect that Microsoft releases the express edition along with or soon after the commercial editions, and the commercial edition is sufficient for running the Python release process. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python as a Metro-style App
When you say MoveFile is absent, is MoveFileEx supported instead? Or is moving files just totally impossible? I can't check the SDK headers right now, but according to the online documentation, MoveFileExW is indeed available. I'm not sure whether you are allowed to pass arbitrary file names in an App, though. Depending on the extent of removed/disabled functionality, it might not be very interesting to have a Metro port at all. I'm not so sure. Even if the low-level Win32 API was not available, you might still be able to do useful things with the higher-level APIs, such as Windows.Storage (in case of file access). If you use, say, Windows.Storage.ApplicationData.RoamingSettings in your app, you should not actually worry what the file is named on disk (or whether there is a spinning disk in the system at all, which probably isn't). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python as a Metro-style App
Perhaps this is better for another topic, but is anyone using the PGO stuff? I know we have PGInstrument and PGUpdate build configurations but I've never seen them mentioned anywhere. I'm using them in the 32-bit builds. I don't use them for the 64-bit builds, as the build machine was a 32-bit system (but perhaps I start with PGO for Win64 for 3.3). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python as a Metro-style App
Zitat von Antoine Pitrou : > When you say MoveFile is absent, is MoveFileEx supported instead? WinRT strongly prefers asynchronous methods for all lengthy operations. The most likely call to use for moving files is StorageFile.MoveAsync. http://msdn.microsoft.com/en-us/library/windows/apps/br227219.aspx How does it translate to C? Not sure whether you are asking literally for *C*: please remember that my original report said that C is apparently not currently supported for Apps. In any case, for native C++ code, do StorageFile ^the_file = something(); the_file->MoveAsync(destinationFolder, "newfile.txt"); This may look like managed C++ to you, but it really compiles into native code. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python as a Metro-style App
Let me see if I can try this. Hopefully I still have my VM w/ this all setup and I can see if I can get it building this way. I can always ping some people on the C++ team and ask them for help if I run into issues. I'll give it a shot tomorrow and get back to you. Hi Dino, I reported that as a bug. If you need that for reference, see https://connect.microsoft.com/VisualStudio/feedback/details/717395/c1083-when-compiling-c-code-in-a-metro-app Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the fix for the hash collision vulnerability
What an implementation looks like: http://pastebin.com/9ydETTag some stuff to be filled in, but this is all that is really required. I think this statement (and the patch) is wrong. You also need to change the byte string hashing, at least for 2.x. This I consider the biggest flaw in that approach - other people may have written string-like objects which continue to compare equal to a string but now hash different. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the fix for the hash collision vulnerability
I think this statement (and the patch) is wrong. You also need to change the byte string hashing, at least for 2.x. This I consider the biggest flaw in that approach - other people may have written string-like objects which continue to compare equal to a string but now hash different. They're unlikely to have rewritten the hash algorithm by hand - especially given the caveats wrt. differences between Python integers and C integers. See the CHAR_HASH macro in http://hg.python.org/cpython/file/e78f00dbd7ae/Modules/expat/xmlparse.c It's not *that* unlikely that more copies of that algorithm exist. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the fix for the hash collision vulnerability
It doesn't change anything, you will still get collisions. That depends right? If the collision is because they all have the same hash(), yes. It might be different if it is because the secondary hashing (or whatever it's called :-) causes collisions. But Python deals with the latter case just fine already. The open hashing approach relies on the dict resizing "enough" to prevent collisions after the dictionary has grown. Unless somebody can demonstrate a counter example, I believe this discussion is a red herring. Plus: if an attacker could craft keys that deliberately cause collisions because of the dictionary size, they could likely also craft keys in the same number that collide on actual hash values, bringing us back to the original problem. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hashing proposal: change only string-only dicts
Zitat von Victor Stinner : Each string would get two hashes: the "public" hash, which is constant across runs and bugfix releases, and the dict-hash, which is only used by the dictionary implementation, and only if all keys to the dict are strings. The distinction between secret (private, secure) and "public" hash (deterministic) is not clear to me. It's not about privacy or security. It's about compatibility. The dict-hash is only used in the dict implementation, and never exposed, leaving the tp_hash unmodified. Example: collections.UserDict implements __hash__() using hash(self.data). Are you sure? I only see that used for UserString, not UserDict. collections.abc.Set computes its hash using hash(x) of each item. Same question. The hash of the Set should most certainly use the element's tp_hash. That *is* the hash of the objects, and it may collide for strings just fine due to the vulnerability. If we need to use the secret hash, it should be exposed in Python. It's not secret, just specific. I don't mind it being exposed. However, that would be a new feature, which cannot be added in a security fix or bug fix release. Which function/method would be used? I suppose that we cannot add anything to stable releases like 2.7. Right. Nor do I see any need to expose it. It fixes the vulnerability just fine without being exposed. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the fix for the hash collision vulnerability
If randomized hash cannot be turned on by default, an alternative is to switch them off by default, and add an option (command line option, environment variable, etc.) to enable it. That won't really fix the problem. If people install a new release because it fixes a vulnerability, it better does so. In 2003, Python was not seen as vulnerable. Maybe because the hash function is different than Perl hash function, or because nobody tried to generate collisions. Today it is clear that Python is vulnerable (64 bits version is also affected), and it's really fast to generate collisions using the right algorithm. There is the common vulnerability to the threat of confusing threats with vulnerabilities [1]. Python was vulnerable all the time, and nobody claimed otherwise. It's just that nobody saw it as a threat. I still don't see it as a practical threat, as there are many ways that people use in practice to protect against this threat already. But I understand that others feel threatened now. Why is it so long to fix the vulnerability in Python, whereas it was fixed quickly in Ruby? (they chose to use a randomized hash) Because the risk of breakage for Python is much higher than it is for Ruby. Regards, Martin [1] http://jps.anl.gov/Volume4_iss2/Paper3-RGJohnston.pdf ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Switching to Visual Studio 2010
Is this considered a new feature that has to be in by the first beta? I'm hoping to have it completed much sooner than that so we can get mileage on it, but is there a cutoff for changing the compiler? At some point, I'll start doing this myself if it hasn't been done by then, and I would certainly want the build process adjusted (with all buildbots updated) before beta 1. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [issue13703] Hash collision security issue
I'm curious why AVL tree rather than RB tree, simpler implementation? Somewhat arbitrary. AVL trees have a better performance than RB trees (1.44 log2(N) vs 2 log2(N) in the worst case). Wrt. implementation, I looked around for a trustworthy, reusable, free (as in speech), C-only implementation of both AVL and RB trees. The C++ std::map is out of question as it is C++, and many other free implementations are out of question as they are GPLed and LGPLed. Writing an implementation from scratch for a bugfix release is also out of the question. So I found Ian Piumarta's AVL tree 1.0 from 2006. I trust Ian Piumarta to get it right (plus I reviewed the code a little). There are some API glitches (such as assuming a single comparison function, whereas it would better be rewritten to directly invoke rich comparison, or such as node removal not returning the node that was removed). It gets most API decisions right, in particular wrt. memory management. The license is in the style of the MIT license. If somebody could propose an alternative implementation (e.g. one with an even more liberal license, or with a smaller per-node memory usage), I'd be open to change it. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [issue13703] Hash collision security issue
However, note that my comment on Glenn's question regarding a stdlib addition of a tree type still applies I agree with all that. Having a tree-based mapping type in the standard library is a different issue entirely. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [issue13703] Hash collision security issue
Another issue occurs to me: when a hash with colliding keys (one that has been attacked, and has trees) has a non-string key added, isn't the flattening process likely to have extremely poor performance? Correct. "Don't do that, then" I don't consider it mandatory to fix all issues with hash collision. In fact, none of the strategies fixes all issues with hash collisions; even the hash-randomization solutions only deal with string keys, and don't consider collisions on non-string keys. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [issue13703] Hash collision security issue
How so? None of the patches did, but I think it was said several times that other types (int, tuple, float) could also be converted to use randomized hashes. What's more, there isn't any technical difficulty in doing so. The challenge again is about incompatibility: the more types you apply this to, the higher the risk of breaking third-party code. Plus you still risk that the hash seed might leak out of the application, opening it up again to the original attack. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] plugging the hash attack
1. Simple hash randomization is the way to go. We think this has the best chance of actually fixing the problem while being fairly straightforward such that we're comfortable putting it in a stable release. 2. It will be off by default in stable releases and enabled by an envar at runtime. This will prevent code breakage from dictionary order changing as well as people depending on the hash stability. I think this is a good compromise given the widely varying assessments of the issue. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hashing proposal: 64-bit hash
Zitat von Serhiy Storchaka : 27.01.12 23:08, Frank Sievertsen написав(ла): As already mentioned, the vulnerability of 64-bit Python rather theoretical and not practical. The size of the hash makes the attack is extremely unlikely. Unfortunately this assumption is not correct. It works very good with 64bit-hashing. It's much harder to create (efficiently) 64-bit hash-collisions. But I managed to do so and created strings with a length of 16 (6-bit)-characters (a-z, A-Z, 0-9, _, .). Even 14 characters would have been enough. You need less than twice as many characters for the same effect as in the 32bit-world. The point is not the length of the string, but the size of string space for inspection. To search for a string with a specified 64-bit hash to iterate over 2 ** 64 strings. I think you entirely missed the point of Frank's message. Despite your analysis that it shall not be possible, Frank has *actually* computed colliding strings, most likely also for a specified hash value. Of course, to calculate the hash function to use secure, not allowing "cut corners" and reduce computation time. This issue wouldn't be that relevant if there wasn't a documented algorithm to significantly reduce the number of tries you need to make to produce a string with a desired hash value. My own implementation would need 2**33 tries in the worst case (for a 64-bit hash value); thanks to the birthday paradox, it's actually a significant chance that the algorithm finds collisions even faster. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A new dictionary implementation
Hey, I like this! It's a subtle encouragement for developers to initialize all their instance variables in their __init__ or __new__ method, with a (modest) performance improvement for a carrot. (Though I have to admit I have no idea how you do it. Wouldn't the set of dict keys be different while __init__ is in the middle of setting the instance variables?) The "type's attribute set" will be a superset of the instance's, for a shared key set. Initializing the first instance grows the key set, which is put into the type. Subsequent instances start out with the key set as a candidate, and have all values set to NULL in the dict values set. As long as you are only setting attributes that are in the shared key set, the values just get set. When it encounters a key not in the shared key set, the dict dissociates itself from the shared key set. Another question: a common pattern is to use (immutable) class variables as default values for instance variables, and only set the instance variables once they need to be different. Does such a class benefit from your improvement? It depends. IIUC, if the first instance happens to get this attribute set, it ends up in the shared key set, and subsequent instances may have a NULL value for the key. I'm unsure how *exactly* the key set gets frozen. You cannot allow resizing the key set once it is shared, as you would have to find all instances with the same key set and resize their values. It would be possible (IIUC) to add more keys to the shared key set if that doesn't cause a resize, but I'm not sure whether the patch does that. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] peps: Update with bugfix releases.
I understand that but, to me, it makes no sense to send out truly broken releases. Besides, the hash collision attack is not exactly new either. Another few weeks can't make that much of a difference. Why would the release be truly broken? It surely can't be worse than the current releases (which apparently aren't truly broken, else there would have been no point in releasing them back then). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is this safe enough? Re: [Python-checkins] cpython: _Py_Identifier are always ASCII strings
I would prefer to see _Py_Identifier renamed to _Py_ASCII_Identifier, or at least a comment stating that Identifiers will (per PEP 3131) always be ASCII -- preferably with an assert to back that up. Please ... no. This is a *convenience* interface, whose sole purpose is to make something more convenient. Adding naming clutter destroys this objective. I'd rather restore support for allowing UTF-8 source here (I don't think that requiring ASCII really improves much), than rename the macro. The ASCII requirement is actually more in the C compiler than in Python. Since not all of the C compilers that we compile Python with support non-ASCII identifiers, failure to comply to the ASCII requirement will trigger a C compilation failure. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a maximum element count to parse_qs?
It's an API change, so it is a) in violation with current practice for bug fix releases, and We are already violating a lot of things in order to fix this issue. Not really. There isn't any significant API change in the proposed patch (the ones that are there are safe to ignore in applications). There is, of course, a major behavior change, but that is deliberately opt-in. b) of limited use for existing installations which won't use the API. Obviously it won't fix vulnerabilities due to some other API. If you propose other APIs we can also fix them. No, you are missing my point. I assume you proposed (even though you didn't say so explicitly) that parse_qs gets an opt-in API change to limit the number of parameters. If that is added, it will have no effect on any existing applications, as they will all currently not pass that parameter. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a maximum element count to parse_qs?
No, you are missing my point. I assume you proposed (even though you didn't say so explicitly) that parse_qs gets an opt-in API change to limit the number of parameters. If that is added, it will have no effect on any existing applications, as they will all currently not pass that parameter. No, I said it would include a default value of (say) 1000 parameters. That default value would be applied to anyone doesn't use the new API. (the reason I'm proposing a new API is to allow people to change or disable the limit, in case they really want to pass a large number of parameters) I see. -1 on that proposal, then: there are certainly applications that will break with that. I don't find 1000 POST parameters a lot, and I'm sure that people use that in a programmatic fashion (e.g. to mass-upload stuff). If you really think that kind of change is necessary, develop a separate patch that people who are worried can apply. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 397 - Python launcher for Windows
Zitat von Mark Hammond : I'm wondering what thoughts are on PEP 397, the Python launcher for Windows. I've been using the implementation for a number of months now and I find it incredibly useful. I wonder what the rationale for the PEP (as opposed to the rationale for the launcher) is - why do you need to have a PEP for it? As written, it specifies some "guidelines" that some software package of yours might adhere to. You don't need a PEP for that, just write the software and adhere to the guidelines, possibly putting them into the documentation. A PEP needs to have controversial issues, or else there would not have been a point in writing it in the first place. Also, it needs to concern CPython, or the Python language, else it does not need to be a *P*EP. To be a proper PEP, you need to include these things: - what is the action that you want to see taken? - what is the Python version (or versions) that you want to see the action taken for? - what alternative actions have been proposed, and what are (in your opinion, and the opinion of readers) pros and cons of each action? Assuming you are proposing some future action for CPython, I'm opposed to the notion that the implementation of the launcher is the specification. The specification needs to be in the PEP. It may leave room, in which case the remaining details need to be specified in the documentation. A critical question (IMO) is the question how the launcher gets onto systems. Will people have to download and install it themselves, or will it come as part of some Python distribution? If it comes with the Python distribution, how get multiple copies of the launcher coordinated? Also: what's the name of the launcher? How can I actually use it? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 397 - Python launcher for Windows
The launcher was slightly controversial when the pep was initially written 12 months ago. So what were the objections? Assuming you are proposing some future action for CPython, I'm opposed to the notion that the implementation of the launcher is the specification. The specification needs to be in the PEP. It may leave room, in which case the remaining details need to be specified in the documentation. I'm really not sure what you are trying to say here. Let me try again: I dislike the phrase "written in C, which defines the detailed implementation". That means that in order to find out what the launcher does, you have to read its source code. I also dislike the phrase "but instead to offer guidelines the launcher should adhere to"; the PEP should not just be guidelines, but a clear, prescriptive specification. I admit that I had difficulties to find the places in the PEP where it specifies things, as opposed to explaining things. It seems that all of the sections - An overview of the launcher. - Guidelines for a Python launcher. - Shebang line parsing - Virtual commands in shebang lines: - Customized Commands: - Python Version Qualifiers - Command-line handling - Process Launching are specification, so it may help to group them as subsections of a top-level heading "Specification". OTOH, "Process Launching" has 4 paragraphs of discussion, then two sentences of specification, then 1,5 sentences of discussion. I wish it was easier to find out what the PEP actually says. That the PEP should remove all references to an implementation specification, or that the PEP simply should be withdrawn? Having references to the implementation is fine; saying that you have to read the code to understand what it does, and that the code takes precedence over the PEP is not. If it comes with the Python distribution, how get multiple copies of the launcher coordinated? This may not be specified as well as it could, but: "Future versions of the launcher should remain backwards compatible with older versions, so later versions of Python can install an updated version of the launcher without impacting how the previously installed version of the launcher is used." That's not really my concern. I didn't originally find the place where it said that the launcher goes into the Windows directory. Now that I see it: how do you prevent deinstallation of py.exe when some version of Python is uninstalled, but other versions remain? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] folding cElementTree behind ElementTree in 3.3
Basically, if something is just documented as being callable without subclassing or instance checks being mentioned as supported in the docs, it can be implemented as either a type or an ordinary function, or pretty much any other kind of callable without being deemed an API change So what would be your evaluation of http://docs.python.org/library/xml.etree.elementtree.html#xml.etree.ElementTree.Element in that respect? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP czar for PEP 3144?
Does anyone object to me naming myself PEP czar for PEP 3144? “Tsar is a title used to designate certain European Slavic monarchs or supreme rulers.” Is this our official word? "supreme ruler" sounds good to me. I could go for "inquisitor" instead of "czar" as well... Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hash randomization in 3.3
I'm with Antoine here - being able to force a particular seed still matters for testing purposes. However, the documentation of the option may need to be updated for 3.3 to emphasise that it should only be used to reproduce sporadic failures. Using it to work around applications that can't cope with randomised hashes would be rather ill-advised. In the tracker, someone proposed that the option is necessary to synchronize the seed across processes in a cluster. I'm sure people will use it for that if they can. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] folding cElementTree behind ElementTree in 3.3
Thanks for describing the use case. By the way, when working with ET I also wanted to track the memory usage of the package a couple of times, which made me lament that there's no useful memory profiler in the stdlib. A memory profiler can be a ten-line Python function which, however, does need to be tuned to the application. So I'm not sure it can be provided by the stdlib in a reasonable fashion beyond what's already there, but it may not be necessary to have it in the stdlib, either. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows build - fixing compile warnings before VS2010
Zitat von Brian Curtin : While some effort has gone on to get the 32-bit build to compile without warnings (thanks for that!), 64-bit still has numerous warnings. Before I push forward on more of the VS2010 port, I'd like to have a clean 2008 build all around so we can more easily track what may have changed. Does that *really* have to be a prerequisite for porting to VS 2010? If yes, then my hopes that we can move to VS 2010 before 3.3 are falling... While I have your attention, I'd like to throw two other things out there to follow up the above effort: 1. Is anyone opposed to moving up to Level 4 warnings? Not sure what this means. What kind of warnings would this get us? MS says "This option should be used only to provide "lint" level warnings and is not recommended as your usual warning level setting." Usually, following MS recommendations is a good thing to do on Windows. But then, the documentation goes on saying "For a new project, it may be best to use /W4 in all compilations. This will ensure the fewest possible hard-to-find code defects." ...take a deep breath... 2. Is anyone opposed to enabling warnings as errors? The immediate consequence would be that the Windows buildbots break when somebody makes a checkin on Unix, and they cannot easily figure out how to rewrite the code to make the compiler happy. So I guess I'm -1. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] The ultimate question of life, the universe, and everything
What is the hash of "ePjNTUhitHkL"? Regards, Martin P.S. It took me roughly 86h to compute 150 strings colliding for the 64-bit hash function. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status regarding Old vs. Advanced String Formating
Zitat von Tshepang Lekhonkhobe :
On Fri, Feb 24, 2012 at 23:39, "Martin v. Löwis" wrote:
It is a burden for some people to learn and remember the exact details
of both systems and exactly how they differ. Having both in the stdlib
hurts readability for such people. I would prefer that the stdlib only
used {} formatting or if not that, that it only used the simple,
hard-to-forget forms of % formatting (%s, %d, %f, without elaboration).
If that issue was getting serious, I would prefer if the .format method
was deprecated, and only % formatting was kept.
Why is that? Isn't .format regarded superior?
I find the .format syntax too complicated and difficult to learn. It has
so many bells and whistles, making it more than just a *mini* language.
So for my own code, I always prefer % formatting for simplicity.
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 - Unicode Literals for Python 3
Zitat von Armin Ronacher : Hi, On 2/27/12 10:17 AM, "Martin v. Löwis" wrote: There are a few other unproven performance claims in the PEP. Can you kindly provide the benchmarks you have been using? In particular, I'm interested in the claim " In many cases 2to3 runs one or two orders of magnitude slower than the testsuite for the library or application it's testing." The benchmarks used are linked in the PEP. Maybe I'm missing something, but there doesn't seem to be a benchmark that measures the 2to3 performance, supporting the claim that it runs "two orders of magnitude" slower (which I'd interpret as a factor of 100). If the claim actually cannot be supported, please remove it from the PEP. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 - Unicode Literals for Python 3
On 2/27/12 9:58 PM, R. David Murray wrote: But the PEP doesn't address the unicode_literals plus str() approach. That is, the rationale currently makes a false claim. Which would be exactly what that u() does not do? Armin, I propose that you correct the *factual* deficits of the PEP (i.e. remove all claims that cannot be supported by facts, or are otherwise incorrect or misleading). Many readers here would be more open to accepting the PEP if it was factual rather than polemic. The PEP author is supposed to collect all arguments, even the ones he doesn't agree with, and refute them. In this specific issue, the PEP states "the unicode_literals import the native string type is no longer available and has to be incorrectly labeled as bytestring" This is incorrect: even though the native string type indeed is no longer available, it is *not* consequential that it has to be labeled as byte string. Instead, you can use the str() function. It may be that you don't like that solution for some reason. If so, please mention the approach in the PEP, along with your reason for not liking it. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 - Unicode Literals for Python 3
A couple of people have said that 'native string' is spelt 'str', but I'm not sure that's the right answer. For example, 2.x's cString.StringIO expects native strings, not Unicode: Your counter-example is non-ASCII characters/bytes. I doubt that this is a valid use case; in a "native" string, these shouldn't occur (i.e. native strings should always be ASCII), since the semantics of non-ASCII changes drastically between 2.x and 3.x. So whoever defines some API to take "native" strings can't have defined a valid use of non-ASCII in that interface. I'm not saying this is the right thing to do for all cases - just that str() may not be, either. This should be elaborated in the PEP. Indeed it should. If there is a known application of non-ASCII native strings, I surely would like to know what that is. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 - Unicode Literals for Python 3
In the end, that's not particularly relevant, because you don't have to run the test suite entirely; when working on small changes, you usually re-run the impacted parts of the test suite until everything goes fine; on the other hand, 2to3 *has* to run on the entire code base. Not at all. If you are working on the code, 2to3 only needs to run on the parts of the code that you changed, since the unmodified parts will not need to be re-transformed using 2to3. So, really, it's a couple of seconds to run a single bunch of tests vs. several minutes to run 2to3 on the code base. Not in my experience. The incremental run-time of 2to3 after a single change is in the order of fractions of a second. And it's not just the test suite: every concrete experiment with the library you're porting has a serial dependency on running 2to3. Therefore, your build process should support incremental changes. Fortunately, distribute does support this approach. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Spreading the Python 3 religion (was Re: PEP 414 - Unicode Literals for Python 3)
If PEP 414 helps some projects migrate to Python 3, great. But I really hope we as a community don't perpetuate the myth that you cannot port to Python 3 without this, and I hope that we spend as much effort on educating other Python developers on how to port to Python 3 *right now* supporting Python 2.6, 2.7, and 3.2. One thing that the PEP will certainly achieve is to spread the myth that you cannot port to Python 3 if you also want to support Python 2.5. That's because people will accept the "single source" approach as the one right way, and will accept that this only works well with Python 2.6. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting PEP 414
Why is readding u'' a feature and not a bug? There is a really simple litmus test for whether something is a bug: does it deviate from the specification? In this case, the specification is the grammar, and the implementation certainly doesn't deviate from it. So it can't be a bug. Regards, Martin P.S. Before anybody over-interprets this criterion: there is certain "implicit behavior" assumed in Python that may not actually be documented, such as "the interpreter will not core dump", and "the source code will compile with any standard C compiler". Deviation from these implicit assumption is also a bug. However, they don't apply here. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
Zitat von Lennart Regebro : Just my 2 cents on the PEP rewrite: u'' support is not just if you want to write code that doesn't use 2to3. Even when you use 2to3 it is useful to be able to flag strings s binary, unicode or "native". How so? In the Python 3 code, the u"" prefix would not appear, even if appears in the original source, as 2to3 eliminates it. So you surely need the u"" prefix to distinguish binary, unicode, or native strings in your source - but with 2to3, the PEP 414 change is unnecessary. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP-393/PEP-3118: unicode format specifiers
The main reason why I raised the issue is this: If Python-3.3 is shipped with 'u' -> UCS4 in the array module and *then* someone figures out that the above format codes are a great idea, we'd be stuck with yet another format code incompatibility. Ah. I think the array module should maintain compatibility with Python 3.2, i.e. "u" should continue to denote Py_UNICODE, i.e. 7fa098f6dc6a should be reverted. It may be that the 'u' code is not particularly useful, but AFAICT, it never was useful. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
When IDLE crashes, running it from a cmd window is the only way to get a traceback to help diagnose the problem. Certainly. In this case, there is no PATH issue, though: you have to CD into the Python installation, anyway, to start IDLE - and there you have python.exe in the current directory. Plus, you can still launch Lib\idlelib\idle.py without prefixing it with python.exe. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] GSoC 2012: Python Core Participation?
I'm wondering whether Python Core should participate in GSoC 2012 or not, as core contributors have shown little interest in acting as mentors in the past. If you are a core committer and volunteer as GSoC mentor for 2012, please let me know by Friday (March 23rd). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 393 decode() oddity
Anyone can test.
$ ./python -m timeit -s 'enc = "latin1"; import codecs; d =
codecs.getdecoder(enc); x = ("\u0020" * 10).encode(enc)' 'd(x)'
1 loops, best of 3: 59.4 usec per loop
$ ./python -m timeit -s 'enc = "latin1"; import codecs; d =
codecs.getdecoder(enc); x = ("\u0080" * 10).encode(enc)' 'd(x)'
1 loops, best of 3: 28.4 usec per loop
The results are fairly stable (±0.1 µsec) from run to run. It looks
funny thing.
This is not surprising. When decoding Latin-1, it needs to determine
whether the string is pure ASCII or not. If it is not, it must be all
Latin-1 (it can't be non-Latin-1).
For a pure ASCII string, it needs to scan over the entire string,
trying to find a non-ASCII character. If there is none, it has to inspect
the entire string.
In your example, as the first character is is already above 127, search
for the maximum character can stop, so it needs to scan the string only
once.
Try '\u0020' * 99+'\u0080', which is a non-ASCII string but still
takes the same time as the pure ASCII string.
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrating the PEP 397 launcher
Zitat von Brian Curtin : After talking with Martin and several others during the language summit and elsewhere around PyCon, PEP 397 should be accepted. I don't remember who, but some suggested it should just be a regular old feature instead of going through the PEP process. So...does this even need to continue the PEP process? I don't think PEP 397 can be accepted as it stands; it has too many open issues. However, I am in favor of accepting the proposed feature. Now that we do have the PEP, I think that should be done properly. I thought you offered to rewrite it. Formally, I could accept the PEP being withdrawn, and the feature integrated anyway, but I still consider that bad style. So: I can offer to rewrite the PEP to give a full specification of the feature. It might be that I still need some help to provide end-user prose in the PEP if people would want to see that as well. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.2): Issue #14409: IDLE doesn't not execute commands from shell with default
Is my suggested rephrasing correct? I don't know, as I'm not familiar with either the original problem or what was done to fix it. I don't know, either, and Andrew may not able to answer the question as he may not see the fine difference between what he wrote and what you wrote (your phrasing is grammatically fairly advanced). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue13210 : Support Visual Studio 2010
Does anyone object if I submit my patches sxs.patch and errnomodule.patch? These allow python to work correctly when built with vs2010. Please see my review: "allow to work correctly" is not a good explanation of what it does, and why it does that. As it is highly counter-intuitive, it needs at least justification. I'm trying to investigate whether it is actually correct (and that it works for CCP is not sufficient proof that it is correct). Also, please clarify what branches you would apply this to. 3.3 is certainly fine; for any other branches, I'd like to point out that this is not a bug fix. There is also the PCBuild10.patch, but that can wait. Indeed, it shouldn't be added at all. Instead, the PCbuild tree should become a VS2010 (or VS2012) tree ultimately. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using Cython for developing a module to be used from postgreSQL/pl/python
Can you recommend a similar place for asking the same queston about ctypes ? That is using ctypes for calling back to "outer" c application which embeds the python interpreter. ISTM that Postgres lists should be the best place for this kind of question. Alternatively, try python-list or db-sig. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing surplus fields from the frame object and not adding any new ones.
The point I really wanted to make is that many of the fields in the frame object belong elsewhere and adding new fields to the frame object is generally a bad idea. I disagree with that statement, and don't think you have offered sufficient proof of it. The structure may look irregular to you, but maybe you just need to get used to it. Factually, I don't think that *many* of the fields belong elsewhere. The majority of the fields clearly belongs where it is, and there is nothing wrong with adding new fields if there is a need for it. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Upgrading tcl/tk deps
Zitat von Brian Curtin : Can someone let me in on the process to upgrade tcl and tk on svn.python.org? For the VS2010 port it looks like I need to upgrade since the 8.5.9 versions do not work. They use link options that choke on 2010. Taking 8.5.11, which is the current release, seems to work out alright so far. It seems as easy as downloading the tarball and checking that in. Am I missing any official process here? Yes. There is a set of changes that you need to preserve. Tk *never* works with any recent VC compilers, so even if you use a new version, you still likely have to adjust the sources and the build process. Also, make sure Tix works. So there are two options: a) adjust the existing sources to work with the new compiler. To do so, modify tk-8.5.9.x (or whatever we currently use), then tag your modifications as tk-8.5.9. (would be .1 AFAICT), then update Tools/buildbot and PCbuild/readme.txt to refer to these. b) import new sources into tk-8.X.Y.x, then go through the changes in tk-8.5.9.x, and port over what is still needed. Again, tag your imported tree so that the Python tree refers to the tag, allowing for modifications to Tk should they be necessary. Switching to the most recent Tk release is a good idea, anyway, so b) is preferred. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 9141, finalizers and gc module
What I want to know is, why is this limitation in place? Here are two possibilities: 1) "The order of calling finalizers in a cycle is undefined so it is not a solvable problem". But this would allow a single object, with only internal cycles to be collected. Currently this is not the case. It's similar to this, but not exactly: "A finalizer in a cycle mail try to refer back to an object that was already cleared, and fail because of that; this may cause arbitrary failures changing from run to run". It's true that a cycle involving only a single object with __del__ could be safely collected. This special case was not implemented. 2) "During collection, the interpreter is in a fragile state (linked lists of gc objects with refcount bookkeeping in place) and no unknown code can be allowed to run". This is the reason I personally think is the true reason. No, that's not the case at all. As Antoine explains in the issue, there are plenty of ways in which Python code can be run when breaking a cycle. Not only weakrefs, but also objects released as a consequence of tp_clear which weren't *in* the cycle (but hung from it). So, I ask you: What is allowed during tp_clear()? Is this a hard rule? What is the reason? We are all consenting adults. Everything is allowed - you just have to live with the consequences. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 9141, finalizers and gc module
Well, we specifically decided that objects with __del__ methods that are part of a cycle cannot be run. The same reasoning was applied to generators, if they are in a certain state. What makes iobase so special that its 'close' method can be run even if it is part of a cycle? It's a hack, and I find it well-documented in iobase.c. It explains what tricks it has to go through to still invoke methods from tp_del. Invoking methods in tp_clear I find fairly harmless, in comparison. My only concern is that errors are silently ignored. However, I don't think this matters in practice, since io objects typically are not part of cycles, anyway. Why not allow it for all objects, then? It's *allowed* for all objects. Why do you think it is not? It must be opt-in, though. In the IO case, there are certain drawbacks; not being able to report errors is the most prominent one. Any other object implementation will have to evaluate whether to follow the iobase approach, or implement a regular __del__. I personally consider the resurrection in tp_del a much more serious problem, though, as this goes explicitly against the design of the release procedure. For iobase, it's ok since it can evolve along with the rest of the code base. Any third-party author would have to accept that such approach can break from one Python release to the next. I wonder why Python couldn't promise to always invoke tp_clear on GC objects; ISTM that this would remove the need for resurrection in tp_del. At the very least, I think this behaviour (this exception for iobase) merits being explicitly documented. I find all of this well-documented in iobase.c. If you think anything else needs to be said, please submit patches. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cython for cPickle?
So I disagree that we could easily reach the point where it's "close to the speed of what we have." And if we were to attempt this, it would be a multiple months undertaking. I would rather see that time spent on improving pickle than on yet another reimplementation. Of course, this being free software, anybody can spend time on whatever they please, and this should not make anybody feel sad. You just don't get merits if you work on stuff that nobody cares about. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What do PyAPI_FUNC & PyAPI_DATA mean?
Quoting Kristján Valur Jónsson : You know that I'm speaking of Windows, right? Yes, but this may only be valid for CCP; for CPython, we certainly have to consider Unix as well. IMHO, we shouldn't put the PyAPI* stuff on functions unless they are actual API functions. I don't know how the export tables for ELF .so objects is generated, but it surely can't export _everything_. It certainly does. Any global symbol in an ELF shared library gets exported. There are recent (10 years or so) ways to restrict this, by declaring symbols as hidden in the object file, but exporting everything is the default that Python uses. Anyway, marking stuff as part of the API makes sense, and marking functions as being part of the API makes no sense and is wasteful when they are not. There are cases where it's necessary: when an extension module uses a function that is not in the API. We might even have something similar for the stable api. I don't understand. Everything in the stable api is part of the API, and thus needs to be exported from the Python DLL. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What do PyAPI_FUNC & PyAPI_DATA mean?
Quoting Kristján Valur Jónsson : Aha, so that is the rationale. Because the export table on unix is so generous, we force ourselves to be generous on windows too? Yes. If the code compiles and links on Unix, it shall also compile and link on Windows. I did some unix programming back in the day. IRIX, actually (a Sys V derivative). I'm pretty sure we had to explicitly specify our .so exports. But I might be mistaken. Maybe on IRIX, probably in a way that predates ELF. In the old days, on Linux, you had to globally request address space from Linus Torvalds for shared libraries. These days are long gone. ELF shared libraries are designed to give the same experience (roughly) as static libraries, wrt. source compatibility. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Implement PEP 412: Key-sharing dictionaries (closes #13903)
Benchmarks should measure memory usage too, of course. Sadly that is not possible in standard cPython. It's actually very easy in standard CPython, using sys.getsizeof. Btw, this is of great interest to me at the moment, our Shanghai engineers are screaming at the memory waste incurred by dictionaries. A 10 item dictionary consumes 1/2k on 32 bits, did you know this? I did. In Python 3.3, this now goes down to 248 bytes (32 bits). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is it safe to assume that Python 2.7 is always built with unicode support?
I'm looking into issue 1065986 [1], and in order to submit a patch I need to know whether I have to take into account the eventuality that cpyhon 2.7 be built without unicode support. It's intended (at least, it is *my* intention) that Python 2.7 can be built without Unicode support, and it's a bug if that is not possible anymore. Certain embedded configurations might want that. That doesn't mean that the bug needs to be fixed; this can be deferred until somebody actually requests that bug being fixed, or better, until somebody contributes a patch to do so. However, it *does* mean that we shouldn't further break the feature, at least not knowingly. OTOH, it's clear that certain functionality cannot work if Unicode is disabled, so it may be acceptable if pydoc breaks in such a configuration. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Open PEPs and large-scale changes for 3.3
Do you know when a more detailed schedule for VS 2012 will be available (and confirmation regarding XP support)? Unfortunately, Microsoft doesn't publish any release dates. It's ready when it's ready :-( I just search again, and it appears that some roadmap has leaked: http://www.zdnet.com/blog/microsoft/microsoft-roadmap-leaks-for-office-15-ie-10-and-more-key-products/12417 That says that a release is scheduled for "late 2012", which would put it after the Python 3.3 release (contrary to rumors I heard elsewhere). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] CRLF line endings
Zitat von Chris Angelico : On Thu, May 3, 2012 at 10:50 PM, Vinay Sajip wrote: Although these are text files, the CRLF line endings are needed because otherwise, the files won't be presented correctly on Windows, e.g. in Notepad. Not all Windows editors choke on \n line endings; when I'm on Windows and run into one, I open it in Wordpad (or, if I have one, a dedicated programming editor like SciTE or the Open Watcom editor). AFAIK only Notepad (of standard Windows utilities) has trouble. Not sure if that makes a difference or not. I think that .bat files strictly *have* to have CRLF line endings. Not sure about PowerShell, though. In any case, having CRLF for these files sounds good to me. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
