[Python-Dev] Optimize Unicode strings in Python 3.3

Hi, Different people are working on improving performances of Unicode strings in Python 3.3. This Python version is very different from Python 3.2 because of the PEP 393, and it is still unclear to me what is the best way to create a new Unicode string. There are different approachs: * Use the

Re: [Python-Dev] time.clock_info() field names

> To me, "adjusted" and "is_adjusted" both imply that an adjustment > has already been made; "adjustable" only implies that it is possible. The documentation is: "True if the clock can be adjusted (e.g. by a NTP daemon), False otherwise." I prefer "adjustable", because no OS tell us if the clock

Re: [Python-Dev] time.clock_info() field names

> I'm -1 on that. To my mind "adjustable" suggests that the caller can > adjust the clock, while "adjusted" suggests that the clock may be adjusted > by a mechanism outside the caller's hands. That latter is the meaning > in the context of the PEP. Anyway, the implementation and/or the documentati

Re: [Python-Dev] [Python-checkins] cpython: Issue #14716: Change integer overflow check in unicode_writer_prepare()

> However, it's not my fault. I suggested `newlen < (PY_SSIZE_T_MAX - > PY_SSIZE_T_MAX / 5)` and not `newlen <= (PY_SSIZE_T_MAX - PY_SSIZE_T_MAX / > 5)`. In this case, there is no overflow. Oh. I didn't understand why you replaced <= by <, and so I used <=. Anyway, I reverted the change for all r

Re: [Python-Dev] Point of building without threads?

> I guess a long time ago, threading support in operating systems wasn't > very widespread, but these days all our supported platforms have it. > Is it still useful for production purposes to configure > --without-threads? Do people use this option for something else than > curiosity of mind? At w

Re: [Python-Dev] [Python-checkins] cpython: Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs.

> http://hg.python.org/cpython/rev/cdcc816dea85 > changeset:   76971:cdcc816dea85 > user:        Antoine Pitrou > date:        Tue May 15 23:48:04 2012 +0200 > summary: >  Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. > Patch by Serhiy Storchaka. Such optimization should

Re: [Python-Dev] [Python-checkins] cpython: Issue #14744: Use the new _PyUnicodeWriter internal API to speed up str%args

68d76 tag: tip user: Victor Stinner date:Tue May 29 18:53:56 2012 +0200 files: Objects/unicodeobject.c description: Issue #14744: Fix compilation on Windows (part 2) changeset: 77230:6abab1a103a6 user: Victor Stinner date:Tue May 29 18:51:10 2012 +0

Re: [Python-Dev] Optimize Unicode strings in Python 3.3

Hi, >  * Use a Py_UCS4 buffer and then convert to the canonical form (ASCII, > UCS1 or UCS2). Approach taken by io.StringIO. io.StringIO is not only > used to write, but also to read and so a Py_UCS4 buffer is a good > compromise. >  * PyAccu API: optimized version of chunks=[]; for ...: ... > chu

Re: [Python-Dev] Optimize Unicode strings in Python 3.3

>> The "two steps" method is not promising: parsing the format string >> twice is slower than other methods. > > The "1.5 steps" method is more promising -- first parse the format string in > an efficient internal representation, and then allocate the output string > and then write characters (or e

Re: [Python-Dev] Issue 2736: datetimes and Unix timestamps

> Anyway, I was pointed to issue 2736, which seems to have got a lot of > discouraged core contributors (Victor, Antoine, David and Ka-Ping, to > name just a few) up against Alexander (the datetime maintainer, > AFAIK). It seems like a fairly straightforward case of practicality > over purity: Alex

[Python-Dev] Issue #11022: locale.getpreferredencoding() must not set temporary LC_CTYPE

Hi, I would like to know if it is too late (or not) to change the behaviour of open() for text files (TextIOWrapper). Currently, it calls locale.getpreferredencoding() to get the locale encoding by default. It is convinient and I like this behaviour... except that it changes temporary the LC_CTYPE

Re: [Python-Dev] Issue #11022: locale.getpreferredencoding() must not set temporary LC_CTYPE

> Fine with me. Ok, done with changeset 2587328c7c9c. > So in theory, your change should have no effect, unless somebody has > modified some environment variables. Changing TextIOWrapper to call locale.getpreferredlocale(False) instead of getpreferredlocale() has these two effects: 1) without t

Re: [Python-Dev] time.clock_info() field names

2012/5/4 Victor Stinner : > Anyway, the implementation and/or the documentation is buggy and > should be fixed (especially the Windows case). Done, I renamed "adjusted" to "adjustable", fixed its value on Windows (time.time) and Linux (time.monotonic), and updated the

Re: [Python-Dev] PEP 362 Third Revision

Sorry if I'm asking dummy questions, I didn't follow the discussion. > * format(...) -> str >    Formats the Signature object to a string.  Optional arguments allow >    for custom render functions for parameter names, >    annotations and default values, along with custom separators. Hum, what a

Re: [Python-Dev] PEP 362 Third Revision

>> I don't understand what the purpose of is_implemented is, or how it is >> supposed to be computed. > > It's computed based on locally available functionality.  Its purpose is to > allow LBYL when using functionality that may not be available on all > platforms.  See issue 14626 for a specific us

Re: [Python-Dev] PEP 362: 4th edition

> 1. Should we keep 'Parameter.implemented' or not.  *Please vote* It looks like only few functions of the os module will use it. Even for the os module, I'm not convinced that it is the appropriate solution to indicate if a feature is supported or not. I would prefer something like os.path.suppor

Re: [Python-Dev] PEP 362: 4th edition

> Right now we have no way to automatically generate signatures for built-in > functions.  So, as of current trunk, any such signatures would have to be > built by hand. > > If we could somehow produce signature information in C, what then?  Two > possible approaches suggest themselves: > > Pre-gen

Re: [Python-Dev] Status of packaging in 3.3

What is the status of the third party module on PyPI (distutils2)? Does it contain all fixes done in the packaging module? Does it have exactly the same API? Does it support Python 2.5 to 3.3, or maybe also 2.4? How is the distutils2 module installed? Installed manually? Using pip or setuptools? I

Re: [Python-Dev] Do we need __length_hint__ at all? (Was PEP 0424: A method for exposing a length hint)

> *If* resizing list is so slow, then why not make it faster? A simple solution to speed up such problem is to change the overallocation factor, but it may waste memory. Python tries to be fast and to not waste too much memory. > Why is it a significant optimisation? > How much slower is it? > Wh

[Python-Dev] Use function names instead of functions for os.supports_dir_fd?

Hi, Python 3.3 introduced os.supports_dir_fd to check if some os functions do accept a file descriptor instead of a path. The problem is that os.supports_dir_fd is a list of functions, not a list of function names. If os functions are monkey patched, you cannot test anymore if a function supports

[Python-Dev] A new JIT compiler for a faster CPython?

lowing for optional static type declarations." Based on `Pyrex <http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/>`_ == See also == * `Volunteer developed free-threaded cross platform virtual machines? <http://www.boredomandlaziness.org/2012/07/volunteer-supporte

Re: [Python-Dev] A new JIT compiler for a faster CPython?

> Personally, I like the idea of having a JIT compiler more or less as an > extension module at hand. Sort-of like a co-processor, just in software. > Lets you run your code either interpreter or JITed, just as you need. Me too, so something like psyco. LLVM is written in C++ and may have license

Re: [Python-Dev] A new JIT compiler for a faster CPython?

> It's the JIT compiler of Unladen Swallow that "failed"; in > my understanding because LLVM is crap (i.e. it is slow, memory-consuming, > and buggy) - as a low-level virtual machine; it may be ok as a compiler > backend (but I still think it is buggy there as well). What is the status of LLVM now

Re: [Python-Dev] A new JIT compiler for a faster CPython?

> As your original message shows, there has already been > enough duplication of effort in this area. I didn't find yet a project reusing ceval.c: most projects implement their own eval loop and don't use CPython at all. My idea is not to write something new, but just try to optimize the existing

Re: [Python-Dev] Use function names instead of functions for os.supports_dir_fd?

>> Monkey patching is a common practice in Python. test_os.py replaces >> os.exec*() functions temporary for example. > > Perhaps for testing, but I don't think monkey-patching is common in > production code. Perhaps you are thinking of Ruby :) The gevent library does monkey-patch os.fork (and tim

Re: [Python-Dev] PEP updating

Oh, thanks Martin! Victor Le 29 juil. 2012 13:20, Martin v. Löwis a écrit : > > I added a cron job on www.python.org that checks every 5 minutes for > a modified PEP. So automatic updating should work again. > > Regards, > Martin > ___ > Python-Dev mail

Re: [Python-Dev] cpython: Fix findnocoding.p and pysource.py scripts

2012/8/1 Serhiy Storchaka : >> changeset: 78375:67d36e8ddcfc >>Fix findnocoding.p and pysource.py scripts >> > >> -line1 = infile.readline() >> -line2 = infile.readline() >> +with infile: >> +line1 = infile.readline() >> +line2 = infile.readline() >> >> -if get

Re: [Python-Dev] [Python-checkins] cpython (merge 3.2 -> default): Make TextIOWrapper's documentation clearer by copying the newline argument's

2012/8/4 Chris Jerdonek : > On Fri, Aug 3, 2012 at 3:59 PM, antoine.pitrou > wrote: >> http://hg.python.org/cpython/rev/f17a1410ebe5 >> changeset: 78401:f17a1410ebe5 >> summary: >> Make TextIOWrapper's documentation clearer by copying the newline >> argument's description from open(). > > Now

Re: [Python-Dev] What's New in Python 3.3

Does Python 3.3 support cross-compilation? There are two new option for configure: --host and --build, but it's not mentioned in What's New in Python 3.3. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/py

[Python-Dev] AST optimizer implemented in Python

Hi, I started to implement an AST optimizer in Python. It's easy to create a new AST tree, so I'm surprised that I didn't find any existing project. https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py To test its peephole optimizations (by checking manually its final bytecode), I wr

Re: [Python-Dev] AST optimizer implemented in Python

>> Other idea to improve this optimizer: >> - move invariant out of loops. Example: "x=[]; for i in range(10): >> x.append(i)" => "x=[]; x_append=x.append; for i in range(10): >> x_append(i)". Require to infer the type of variables. > > But this is risky. It's theoretically possible for x.append t

Re: [Python-Dev] AST optimizer implemented in Python

> I started to implement an AST optimizer in Python. It's easy to create > a new AST tree, so I'm surprised that I didn't find any existing > project. I done more research. I found the AST optimizer of PyPy, which implements basic optimizations: https://bitbucket.org/pypy/pypy/src/default/pypy/int

Re: [Python-Dev] AST optimizer implemented in Python

2012/8/14 Kristján Valur Jónsson : >> I moved the script to a new dedicated project on Bitbucket: >> https://bitbucket.org/haypo/astoptimizer >> >> Join the project if you want to help me to build a better optimizer! >> >> It now works on Python 2.5-3.3. > > I had the idea (perhaps not an original

Re: [Python-Dev] 3.3 str timings

2012/8/18 Terry Reedy : > The issue came up in python-list about string operations being slower in > 3.3. (The categorical claim is false as some things are actually faster.) Yes, some operations are slower, but others are faster :-) There was an important effort to limit the overhead of the PEP 3

[Python-Dev] Sphinx issue in What's New in Python 3.3 doc

Hi, In the first example of the "PEP 409: Suppressing exception context" section, I read "from None...". http://docs.python.org/dev/whatsnew/3.3.html#pep-409-suppressing-exception-context It's confusing because I don't remember what was the last choice for the PEP: None or ... :-) The reST "code

[Python-Dev] Unicode support of the curses module in Python 3.3

Hi, I changed many functions of the curses module in Python 3.3 to improve its Unicode support: - new functions: curses.unget_wch() and window.get_wch() - new attribute: window.encoding - the default encoding is now the locale encoding instead of UTF-8 - use the C functions *_wch() and *wstr()

Re: [Python-Dev] Unicode support of the curses module in Python 3.3

>> For example, if the Python curses module is not linked to libncursesw, >> get_wch() is not available and addch("é") raises an OverflowError if >> the locale encoding is UTF-8 (because "é".encode("utf-8") is longer >> than 1 byte). > > OverflowError? That is very surprising. I wouldn't guess that

Re: [Python-Dev] Tru64 support

2012/9/4 Andrew Svetlov : > I don't see Tru64 in list of available buildbots. Nope, there is no Tru64 buildbot. > Do we need to care about this platform? And how to make sure what > existing code works fine for that? I don't know this OS. According to Wikipedia, it's a closed source OS which onl

[Python-Dev] Release of astoptimizer 0.3

Hi, Here are some progress on my astoptimizer project. If you are interested by the optimizer, run it on your own project or help me to implement more optimizations. http://pypi.python.org/pypi/astoptimizer https://bitbucket.org/haypo/astoptimizer --- The last version (0.3) works on Python 2.6-

Re: [Python-Dev] Release of astoptimizer 0.3

>> * Call builtin functions if arguments are constants. Examples: >> >> - len("abc") => 3 >> - ord("A") => 65 > > Does it preserve python semantics? What if you change the len builtin? This optimization is disabled by default (in the version 0.3), because builtin functions may be shadowed. Exa

Re: [Python-Dev] Release of astoptimizer 0.3

2012/9/11 Nick Coghlan : > This is fine in an external project, but should never be added to the > standard library. The barrier to semantic changes that break > monkeypatching should be high. The version 0.3 has a known bug: "len=chr; print(len('A'))" is optimized, whereas it should not. It is no

Re: [Python-Dev] Release of astoptimizer 0.3

>> * Call builtin functions if arguments are constants. Examples: >> >>- len("abc") => 3 >>- ord("A") => 65 > > Does this mean transformation print("A") => None and output at compile time? Only a subset of builtin functions are called at compile time, and not with any constant value. See a

Re: [Python-Dev] Release of astoptimizer 0.3

>>> - frozenset("ab") | frozenset("bc") => frozenset("abc") > > That's a name lookup, too. Yes, and it is only optimized if the "builtin_funcs" feature is enabled explictly. > Except that evaluating something like '"abc" * constant' can eat up all > memory, imagine this code: > > KILL_MEMOR

Re: [Python-Dev] Release of astoptimizer 0.3

> i am a longtime Reader of this list and this is the first time i a dare to > speak up. > Apology in advance for any noise, silly comments and not posting to > python-ideas ;). Welcome! > Well how about implementing guards like in pypy? Guards would allow to generate specialized functions witho

Re: [Python-Dev] Release of astoptimizer 0.3

2012/9/11 Guido van Rossum : > FWIW, I expect that there are few places where len() > is actually used. I found one revelant example in the stdlib. if path.endswith('.dylib'): yield path[:-len('.dylib')] + suffix + '.dylib' else: yield path

Re: [Python-Dev] Release of astoptimizer 0.3

2012/9/12 Serhiy Storchaka : >>> set([x for ...]) => {x for ...} >>> dict([(k, v) for ...]) => {k: v for ...} >>> dict((k, v) for ...) => {k: v for ...} >>> ''.join([s for ...]) => ''.join(s for ...) >>> a.extend([s for ...]) => a.extend(s for ...) >> >> These optimizations look correct. > > Actual

Re: [Python-Dev] Release of astoptimizer 0.3

>> Projects using the same code base for Python 2 and Python 3 contain a >> lot of inefficient code. For example, using the six library, a simple >> Unicode literal strings becomes a function code: u('unicode'). > > But are you able to do enough static analysis to feel comfortable that > this is th

[Python-Dev] What's New in Python 3.3: missing unittest.mock!

Hi, I just noticed that the new unittest.mock module is missing from the What's New in Python 3.3. Can someone add it? (I cannot right now). Thanks. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-

[Python-Dev] Split unicodeobject.c into subfiles?

Hi, I would like to split the huge unicodeobject.c file into smaller files. It's just the longest C file of CPython: 14,849 lines. I don't know exactly how to split it, but first I would like to know if you would agree with the idea. Example: - Objects/unicode/codecs.c - Objects/unicode/mod_fo

Re: [Python-Dev] Split unicodeobject.c into subfiles?

2012/10/4 Benjamin Peterson : > 2012/10/4 Victor Stinner : >> I only see one argument against such refactoring: it will be harder to >> backport/forwardport bugfixes. > > I imagine it could also prevent inlining of hot paths. It depends how the code is compiled. The stringli

Re: [Python-Dev] Split unicodeobject.c into subfiles?

> I am not siding with either side of the change yet, but an additional > argument against is that history may become less convenient to > navigate and track (e.g. hg annotate may lose information depending on > how the split is done). If new files are created using "hg cp unicodeobject.c unicode/

Re: [Python-Dev] Split unicodeobject.c into subfiles?

2012/10/5 Benjamin Peterson : > 2012/10/4 Victor Stinner : >> If new files are created using "hg cp unicodeobject.c >> unicode/newfile.c", the historic is kept. > > Yes, but you can only create one file that way. You can create as many files as you want. Try: --- h

[Python-Dev] Issue #16153: PyUnicode_FromFormatV() must fail if the format string is invalid

Hi, Does anyone know if there is a good reason why PyUnicode_FromFormatV() does not fail on invalid format string, but copies the format string as-if into the result? I would like to change this to raise a ValueError instead. PyUnicode_FromFormatV() is used by PyErr_Format(), so PyErr_Format() ma

Re: [Python-Dev] Split unicodeobject.c into subfiles?

> The amount of code will not be reduced, but now you also need to guess what > file some piece of functionality may be in. How do you search a piece of code? If you search for a function by its name, it does not matter in which file it is defined if you an IDE or vim/emacs with a correct configur

Re: [Python-Dev] Issue #16153: PyUnicode_FromFormatV() must fail if the format string is invalid

2012/10/8 Barry Warsaw : >>Does anyone know if there is a good reason why PyUnicode_FromFormatV() >>does not fail on invalid format string, but copies the format string >>as-if into the result? >> >>I would like to change this to raise a ValueError instead. >>PyUnicode_FromFormatV() is used by PyEr

[Python-Dev] Why not using the hash when comparing strings?

Hi, I would like to know if there a reason for not using the hash of (bytes or unicode) strings when comparing two objects and the hash of the two objects was already been computed. Using the hash would speed up comparaison of long strings when the two strings are different. Something like:

Re: [Python-Dev] Why not using the hash when comparing strings?

2012/10/19 Benjamin Peterson : > It would be interesting to see how common it is for strings which have > their hash computed to be compared. I implemented a quick hack. When running "./python -m test test_os": Python calls PyUnicode_RichCompare() 15206 times with Py_EQ or Py_NE operator. In 41.4%

Re: [Python-Dev] Segmentaion fault with wrongly set PYTHONPATH on Windows

2012/10/22 anatoly techtonik : > Could anybody reopen http://bugs.python.org/issue8766 ? I can't. > Reproducible 100% with Python 3.2 and 3.3 (3.1 didn't test). > >> set PYTHONHOME=C:\ >> python The issue #8766 is about PYTHONPATH environment variable, not PYTHONHOME. Test on Linux with Python 3.4

Re: [Python-Dev] Split unicodeobject.c into subfiles

> Such a restructuring should not result in compilers > no longer being able to optimize code by inlining functions > in one of the most important basic types we have in Python 3. I agree that performances are important. But I'm not convinced than moving functions has a real impact on performances

Re: [Python-Dev] [Python-checkins] cpython: Issue #15478: Fix test_os on Windows (os.chown is missing)

Hi, 2012/10/31 Jeremy Kloth : > On Tue, Oct 30, 2012 at 6:07 PM, victor.stinner > wrote: >> http://hg.python.org/cpython/rev/01cc9fb52887 >> changeset: 80068:01cc9fb52887 >> user:Victor Stinner >> date:Wed Oct 31 01:04:10 2012 +0100 >> sum

Re: [Python-Dev] ctypes is not an acceptable implementation strategy for modules in the standard library?

I'm not sure that ctypes is always available (available on all platforms). Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive

Re: [Python-Dev] cpython: Issue #16218: skip test if filesystem doesn't support required encoding

2012/11/7 Alexandre Vassalotti : > The Unicode code points in the U+DC00-DFFF range (low surrogate area) can't > be encoded in UTF-8. Quoting from RFC 3629: > > The definition of UTF-8 prohibits encoding character numbers between U+D800 > and U+DFFF, which are reserved for use with the UTF-16 encod

[Python-Dev] Register-based VM for CPython

Hi, I'm still looking for the best approach to speedup CPython. I found the following article comparing a stack-based VM to a register-based VM which announces a speed up between 20% and 40% (depending if the instruction dispatch uses a dummy switch or computed goto): http://static.usenix.org/eve

Re: [Python-Dev] cpython: Issue #16455: On FreeBSD and Solaris, if the locale is C, the

Hi, 2012/12/4 Christian Heimes : > Am 04.12.2012 03:23, schrieb victor.stinner: >> http://hg.python.org/cpython/rev/c25635b137cc >> changeset: 80718:c25635b137cc >> parent: 80716:b845901cf702 >> user:Victor Stinner >> date:Tue Dec

Re: [Python-Dev] Do more at compile time; less at runtime

Do know my registervm project? I try to emit better bytecode. See the implementation on http://hg.python.org/sandbox/registervm See for example the documentation: hg.python.org/sandbox/registervm/file/tip/REGISTERVM.txt I implemented other optimisation

Re: [Python-Dev] test___all__ polluting sys.modules?

Oh, I just remember this old issue which is somehow similar: http://bugs.python.org/issue11995 "test_pydoc loads all Python modules" Running test_pydoc in a subprocess would work around this issue. Arfrever wrote a patch, I didn't read it (sorry!). Victor ___

Re: [Python-Dev] PandaBoard, Raspberry Pi coming to Buildbot fleet

2013/1/4 Trent Nelson : > The installation of Ubuntu on the Pandaboard went smoothly. > However, it crashes after about an hour. Console output: > > http://trent.snakebite.net/pandaboard-crash.txt > > Any ARM wizards out there with suggestions? The bug was already reported to

[Python-Dev] Add "e" (close and exec) mode to open()

Hi, I would like add a new flag to open() mode to close the file on exec: "e". This feature exists using different APIs depending on the OS and OS version: O_CLOEXEC, FD_CLOEXEC and O_NOINHERIT. Do you consider that such flag would be interesting? On Linux (glibc), the feature is available as the

Re: [Python-Dev] Add "e" (close and exec) mode to open()

t platforms. Victor 2013/1/8 Benjamin Peterson : > 2013/1/7 Gregory P. Smith : >> >> >> >> On Mon, Jan 7, 2013 at 4:03 PM, Benjamin Peterson >> wrote: >>> >>> 2013/1/7 Victor Stinner : >>> > Hi, >>> > >>> >

Re: [Python-Dev] Add "e" (close and exec) mode to open()

feature using a portable API to hide the "The myriad cloexec APIs". Victor 2013/1/8 Victor Stinner : > Benjamin wrote: >> I'm not sure it's worth cluttering the open() interface with such a >> non-portable option. > Zbyszek wrote: >> If the best-effort fall

Re: [Python-Dev] Add "e" (close and exec) mode to open()

2013/1/8 Victor Stinner : > I don't know platform without this flag. According to the following email, fcntl.FD_CLOEXEC was not available in Python 2.2 on Red Hat 7.3 (in 2003): http://communities.mentor.com/community/cs/archives/qmtest/msg00501.html I don't know if the constant wa

Re: [Python-Dev] Add "e" (close and exec) mode to open()

2013/1/8 Benjamin Peterson : > Okay, fair enough, but I really would like it not to ever raise > NotImplementedError. Then you would end up having different codepaths > for various oses anyway. So what do you suggest? Victor ___ Python-Dev mailing list

Re: [Python-Dev] Add "e" (close and exec) mode to open()

2013/1/8 Benjamin Peterson : > 2013/1/8 Victor Stinner : >> 2013/1/8 Benjamin Peterson : >>> Okay, fair enough, but I really would like it not to ever raise >>> NotImplementedError. Then you would end up having different codepaths >>> for various oses anyway.

Re: [Python-Dev] Point of building without threads?

2013/1/8 Trent Nelson : > On Tue, Jan 08, 2013 at 06:15:45AM -0800, Stefan Krah wrote: >> Trent Nelson wrote: >> > All our NetBSD, OpenBSD and DragonFlyBSD slaves use --without-thread. >> > Without it, they all wedge in some way or another. (That should be >> > fixed*/investigated, bu

[Python-Dev] Set close-on-exec flag by default in SocketServer

Hi, The SocketServer class creates a socket to listen on clients, and a new socket per client (only for stream server like TCPServer, not for UDPServer). Until recently (2011-05-24, issue #5715), the listening socket was not closed after fork for the ForkingMixIn flavor. This caused two issues: i

Re: [Python-Dev] Set close-on-exec flag by default in SocketServer

2013/1/9 Charles-François Natali : >> My question is: would you accept to break backward compatibility (in >> Python 3.4) to fix a potential security vulnerability? > > Although obvious, the security implications are not restricted to > sockets (yes, it's a contrived example): > ... > f = open("/tm

Re: [Python-Dev] Set close-on-exec flag by default in SocketServer

2013/1/10 Victor Stinner : > A better API is maybe to add a "cloexec" argument to open(), ... I realized that setting close-on-exec flag is a non trivial problem. There are many ways to set it depending on the function, on the OS, and on the OS version. There is also the difficult qu

Re: [Python-Dev] fork or exec?

> 2. the solution you're proposing doesn't close the file descriptors > after fork() but after *exec()*. Exact. My PEP cannot solve all issues :-) I will update it to make it more explicit. > Having an official afterfork facility would still help. Yes. IMO it is a different topic: you can also c

Re: [Python-Dev] fork or exec?

2013/1/10 Charles-François Natali : > Disclaimer: I'm not saying we should be changing all FDs to > close-on-exec by default like Ruby did, I'm just saying that there's a > real problem. I changed my mind, the PEP does not propose to change the *default* behaviour (don't set close-on-exec by defau

Re: [Python-Dev] fork or exec?

2013/1/10 Antoine Pitrou : >> I changed my mind, the PEP does not propose to change the *default* >> behaviour (don't set close-on-exec by default). >> >> But the PEP proposes to *add a function* to change the default >> behaviour. Application developers taking care of security can set >> close-on-

Re: [Python-Dev] fork or exec?

2013/1/10 Charles-François Natali : >> Network servers like inetd or apache MPM (prefork) uses a process >> listening on a socket, and then fork to execute a request in a child >> process. I don't know how it works exactly, but I guess that the child >> process need a socket from the parent to send

Re: [Python-Dev] fork or exec?

2013/1/10 Tres Seaver : > Why would we wander away from POSIX semantics here? There are good > reasons not to close open descriptors (the 'pipe()' syscall, for > instance), and there is no POSIXy way to ask for them *not* to be closed. There are different methods to disable the close-on-exec flag

Re: [Python-Dev] fork or exec?

2013/1/11 Ben Leslie : > Python is not UNIX, but I think if you are directly using the POSIX > interfaces they should > work (more or less) the same way the would if you were writing a C program. > (Some of us > still use Python to prototype things that will later be converted to C!). I completed

[Python-Dev] PEP 433: Add cloexec argument to functions creating file descriptors

HTML version: http://www.python.org/dev/peps/pep-0433/ *** PEP: 433 Title: Add cloexec argument to functions creating file descriptors Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-January-2013 Python

Re: [Python-Dev] PEP 433: Add cloexec argument to functions creating file descriptors

> PEP: 433 > Title: Add cloexec argument to functions creating file descriptors > Status: Draft The PEP is still a draft. I'm sending it to python-dev to get a first review. The main question is the choice between the 3 different options: * don't set close-on-exec flag by default * always set

Re: [Python-Dev] PEP 433: Add cloexec argument to functions creating file descriptors

> Alternatives > > > Always set close-on-exec flag > - Oops, this is completly wrong. This section is basically exactly the same than the PEP proposal, except that the default value of cloexec is set to True. The correct title is: "Set the close-on-exec fla

Re: [Python-Dev] PEP 433: Add cloexec argument to functions creating file descriptors

Hi, Thanks for your feedback, I already updated the PEP for most of your remarks. 2013/1/13 Charles-François Natali : > Also, ISTM that Windows also supports this flag. Yes it does, see Appendix: Operating system support > Windows. >> .. note:: >>OpenBSD older 5.2 does not close the file de

Re: [Python-Dev] PEP 433: Add cloexec argument to functions creating file descriptors

2013/1/14 Nick Coghlan : > I'm a fan of the conservative approach, with an environment variable > and command line option to close FDs by default in 3.4 (similar to > PYTHONHASHSEED and -R in the pre-3.3 security releases), and the > cloexec/noinherit behaviour becoming the default (with no way to

Re: [Python-Dev] PEP 433: Add cloexec argument to functions creating file descriptors

2013/1/13 Jeff Allen : > I think io, meaning _io and _pyio really, would be amongst the impacted > modules, and should perhaps be in the examples. (I am currently working on > the Jython implementation of the _io module.) It seems to me that io.open, > and probably all the constructors, such as _io

Re: [Python-Dev] PEP 433: Add cloexec argument to functions creating file descriptors

2013/1/13 Charles-François Natali : >> .. note:: >>OpenBSD older 5.2 does not close the file descriptor with >>close-on-exec flag set if ``fork()`` is used before ``exec()``, but >>it works correctly if ``exec()`` is called without ``fork()``. > > That would be *really* surprising, are

Re: [Python-Dev] Modules/socketmodule.c: avoiding second fcntl() call worth the effort?

Since Linux 2.6.27, it's possible to set SOCK_NONBLOCK directly at the creation of the socket (using socket() and accept4()). So you don't need any extra call. I implemented something similar for SOCK_CLOEXEC flag to implement the PEP 433. See for example: http://hg.python.org/features/pep-433/fil

[Python-Dev] Implementation of the PEP 433

Hi, I implemented the PEP 433 in the following repository: http://hg.python.org/features/pep-433 You can use it if you would like to experiment/test the PEP 433. PEP 433: "Easier suppression of file descriptor inheritance" http://www.python.org/dev/peps/pep-0433/ I added a PYTHONCLOEXEC environ

[Python-Dev] PEP 433: Choose the default value of the new cloexec parameter

Hi, The PEP 433 proposes different options to add a new cloexec parameter: a) cloexec=False by default b) cloexec=True by default c) configurable default value I tried to list in the PEP 433 advantages and drawbacks of each option. If I recorded correctly opinions, the different options have

Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter

2013/1/25 Charles-François Natali : > You can actually count me in the cloexec=False camp, and against the > idea of a configurable default value. Oh ok. > Why cloexec shouldn't be set by default: > - While it's really tempting to fix one of Unix historical worst > decisions, I don't think we can

Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter

> I think the default behaviour needs to be configurable from the > environment and the command line, but I don't believe it should be > configurable from within the interpreter. sys.setdefaultcloexec() is convinient for unit test, but it may also be used by modules. A web framework may want to en

Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter

2013/1/25 Nick Coghlan : > I just realised I could be converted to a +0 if the runtime time > switch could only be used to set the global default as "cloexec=True" > and couldn't be used to switch it back again (for testing purposes, if > you only want to switch it on temporarily, use a subprocess)

Re: [Python-Dev] Implementation of the PEP 433

I created the following issue to track and review the implementation of the PEP 433: http://bugs.python.org/issue17036 Last changes: - don't hold the GIL to ensure "atomicity" of the close-on-exec flag - apply the default value of cloexec on all C functions - sys.setdefaultcloexec() don't have

Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter

> So, if we agree that "cloexec-by-default" is a desirable goal, despite > the inconsistency with POSIX (just as changing integer division was > seen as desirable, despite the inconsistency with C) ... I don't plan to enable cloexec by default in a near future, nor in python 4. Someone else may ch

Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter

2013/1/27 Guido van Rossum : > I had missed this detail. I agree that it should be exposed in the > interpreter. To my mind it is more like PYTHONPATH (which corresponds > roughly to sys.path manipulations) than like -R (which changes > something that should never be changed again, otherwise the sa

Re: [Python-Dev] Anyone building Python --without-doc-strings?

Why don't you compile using python -OO and distribute only .pyo code? Victor 2013/1/27 Kristján Valur Jónsson : > We (CCP) are certainly compiling python without docstrings for our embedded > platforms (that include the PS3) > Anyone using python as en engine to be used by programs and not users

<    27   28   29   30   31   32   33   >