Re: [Python-Dev] [Python-checkins] cpython (3.3): Add -b and -X options to python man page.

2013-06-24 Thread Victor Stinner
2013/6/24 R. David Murray : > There is one. -X faulthandler. I'm sure others would agree about > -O, but that long predates -X. FYI I didn't chose "-X" because it is specific to CPython, but just because it becomes really hard to choose a random letter to add a new option... I prefer long option

Re: [Python-Dev] [Python-checkins] cpython: Issue #9566: recv(), recvfrom(), send(), sendall() and sendto() methods

2013-06-24 Thread Victor Stinner
Oh, I didn't know that."if defined(MS_WIN64) || defined(MS_WINDOWS)" is a common pattern in the Python source code. I simplified the #ifdef in many places: http://hg.python.org/cpython/rev/dfc020b4b123 I also read that MS_WIN32 is always defined on Windows. "#ifdef MS_WIN32" (used in many files, e

[Python-Dev] End of the mystery "@README.txt Mercurial bug"

2013-06-25 Thread Victor Stinner
Hi, One month ago, unit tests were added to IDLE (cool!) with a file called @README.txt. The @ was used to see the name on top in a listing of the directory. Some developers began to get strange Mercurial errors like: "abort: data/Lib/idlelib/idle_test/@README.txt.i at 7573717b9e6f: no match" " 8

Re: [Python-Dev] End of the mystery "@README.txt Mercurial bug"

2013-06-26 Thread Victor Stinner
2013/6/26 Eric V. Smith : > I think that's exactly what's happening. > > From the bug report: > > find $(srcdir) '(' -name '*.fdc' -o -name '*~' \ >-o -name '[@,#]*' -o -name '*.old' \ >-o -name '*.orig' -o -name '*.rej' \ >

[Python-Dev] PyArg_ParseTupe(): parse unsigned integer and check for overflow

2013-06-26 Thread Victor Stinner
Hi, I don't understand why the B, H, I, k formats of PyArg_ParseTupe() do not check for overflow, whereas formats for signed numbers do check for overflow. What is the useful of ignoring overflow? http://docs.python.org/3/c-api/arg.html I would to parse an integer in [0; UINT_MAX] to fix the zlib

Re: [Python-Dev] End of the mystery "@README.txt Mercurial bug"

2013-07-01 Thread Victor Stinner
Hi Georg, 2013/7/1 Georg Brandl : > Am 26.06.2013 16:24, schrieb Victor Stinner: >> 2013/6/26 Eric V. Smith : >>> I think that's exactly what's happening. >>> >>> From the bug report: >>> >>> find $(srcdir) '(&#

Re: [Python-Dev] Expose stack effects to Python?

2013-07-03 Thread Victor Stinner
Hi, For my registervm project (fork of CPython using register-based bytecode, instead of stack-based bytecode), I implemented a Instruction.use_stack() method which just checks if the stack is "used": read the stack, exchange values in the stack (like "ROT" instruction), push or pop a value. Inst

[Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

2013-07-04 Thread Victor Stinner
HTML version: http://www.python.org/dev/peps/pep-0446/ PEP: 446 Title: Add new parameters to configure the inherance of files and for non-blocking sockets Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3

Re: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

2013-07-04 Thread Victor Stinner
2013/7/4 Victor Stinner : > Add a new optional *cloexec* on functions creating file descriptors: > > * ``io.FileIO`` > * ``io.open()`` > * ``open()`` The PEP 433 proposes adding an "e" mode to open in alternatives. I didn't keep this idea because the fopen() functi

Re: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

2013-07-04 Thread Victor Stinner
> PEP: 446 > Title: Add new parameters to configure the inherance of files and for > non-blocking sockets > (...) > Rejected Alternatives > = > > PEP 433 > --- > > The PEP 433 entitled "Easier suppression of file descriptor inheritance" > is a previous attempt proposing vari

Re: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

2013-07-04 Thread Victor Stinner
2013/7/4 Ronald Oussoren : >> The PEP 433 proposes adding an "e" mode to open in alternatives. I >> didn't keep this idea because the fopen() function of the GNU libc >> library has no mode for the O_NONBLOCK flag. IMO it is not interesting >> to mention it in the PEP 466. > > I don't understand yo

Re: [Python-Dev] lament for the demise of unbound methods

2013-07-04 Thread Victor Stinner
2013/7/4 Chris Withers : > That doesn't seem helpful as a sensible way to get back to the class object: > >>> globals()[MyClass.method.__qualname__.split('.')[0]] > globals() can only be used if MyClass is in the same module. Otherwise, you a more complex function: --- import types d

Re: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

2013-07-04 Thread Victor Stinner
2013/7/4 Tres Seaver : > Not commenting on either the form or the substance (pun intended), but > the word you want is "inheritance" -- "inherence" is a valid term[1], but > would a good deal stranger notion to apply to a file descriptor. ;) Oh... I don't know why I wrote "inherance", it was "inhe

Re: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

2013-07-04 Thread Victor Stinner
2013/7/5 Cameron Simpson : > You might want to make clear that the "blocking" parameter refers > only to the file creation calls (eg socket.socket) and not to the > file descriptor itself, and is not to be confused with the UNIX > O_NONBLOCK file descriptor flag (and whatever equivalent flag may >

Re: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

2013-07-05 Thread Victor Stinner
2013/7/5 Cameron Simpson : > | Both set O_NONBLOCK flag (UNIX) > > Oh, how embarassing. You said that the PEP is not cristal clear. Do you have a suggestion to make it more clear? Should I mention that the close-on-exec flag is O_CLOEXEC on UNIX, and HANDLE_FLAG_INHERIT on Windows? (except that H

Re: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

2013-07-06 Thread Victor Stinner
2013/7/6 Charles-François Natali : >> I've read your "Rejected Alternatives" more closely and Ulrich >> Drepper's article, though I think the article also supports adding >> a blocking (default True) parameter to open() and os.open(). If you >> try to change that default on a platform where it does

Re: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

2013-07-06 Thread Victor Stinner
2013/7/6 Cameron Simpson : > Yes. Please forget I mentioned fork(); it is only relevant if you > were offering some facility to undo the addition of cloexec to a > Popen passed file descriptor. Which you are not. Oh... gotcha. I now understood your concern. There is a little "trick" here: at fork

Re: [Python-Dev] Accepting PEP 445

2013-07-06 Thread Victor Stinner
2013/7/6 Antonio Cavallo : > Could that remove the need for the --with-pydebug flag? With the PEP 445, you still have to recompile Python with --with-debug, but you don't have to recompile Python extensions anymore. Debug checks on memory allocators are now implemented as hooks, instead of using C

Re: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

2013-07-07 Thread Victor Stinner
2013/7/7 Charles-François Natali : > 2013/7/7 Cameron Simpson : >> On 06Jul2013 11:23, Charles-François Natali wrote: >> | > I've read your "Rejected Alternatives" more closely and Ulrich >> | > Drepper's article, though I think the article also supports adding >> | > a blocking (default True) par

Re: [Python-Dev] Accepting PEP 445

2013-07-07 Thread Victor Stinner
2013/7/6 Antoine Pitrou : > I'm accepting PEP 445 (A C API to customize memory allocators) by > Victor. There is probably some grammar to correct here and there > (neither Victor nor I are native English speakers), but I don't want > this to hold back acceptance. The PEP is an obvious improvement

[Python-Dev] pyfailmalloc: new debug tool to test how your applications handles MemoryError

2013-07-08 Thread Victor Stinner
Hi, The PEP 445 (Add new APIs to customize Python memory allocators) has been accepted, I commited its implementation. So it's time to have fun with this API. I developed a small Python module (150 lines of C code) to inject memory allocation failures: https://pypi.python.org/pypi/pyfailmalloc T

Re: [Python-Dev] pyfailmalloc: new debug tool to test how your applications handles MemoryError

2013-07-08 Thread Victor Stinner
2013/7/9 Victor Stinner : > I developed a small Python module (150 lines of C code) to inject > memory allocation failures: > https://pypi.python.org/pypi/pyfailmalloc Bitbucket was down, so I was unable to give the link to its source code. The server is back, here is the C co

[Python-Dev] Is it safe to call PyEval_EvalFrameEx() with an exception set?

2013-07-15 Thread Victor Stinner
Hi, I'm working on the issue #18408 (fix issues found by my pyfailmalloc tool). To analyze some bugs, I have to add debug traces in various functions to find which function returned NULL without setting an error, or the opposite: returned a valid object, but with an exception set. I would like to

Re: [Python-Dev] [Python-checkins] cpython (merge default -> 3.3): merge 3.3 (#18470)

2013-07-15 Thread Victor Stinner
2013/7/16 benjamin.peterson : > http://hg.python.org/cpython/rev/8889c9b5dd3a > changeset: 84654:8889c9b5dd3a > branch: 3.3 > parent: 84653:c3a510b22218 > parent: 84652:8a078bf3cf14 > user:Benjamin Peterson > date:Mon Jul 15 19:15:49 2013 -0700 > summary: > merge

Re: [Python-Dev] [Python-checkins] cpython (3.3): check the return value of new_string() (closes #18470)

2013-07-15 Thread Victor Stinner
2013/7/16 benjamin.peterson : > http://hg.python.org/cpython/rev/c3a510b22218 > changeset: 84653:c3a510b22218 > branch: 3.3 > parent: 84651:e22dd5fda5a8 > user:Benjamin Peterson > date:Mon Jul 15 19:15:34 2013 -0700 > summary: > check the return value of new_string()

Re: [Python-Dev] Is it safe to call PyEval_EvalFrameEx() with an exception set?

2013-07-16 Thread Victor Stinner
2013/7/16 Antoine Pitrou : > Le Tue, 16 Jul 2013 02:34:49 +0200, > Victor Stinner a écrit : >> I would like to add assertions in Python/ceval.c to detect such bugs >> earlier. The problem is that some functions rely on the ability to >> call PyEval_EvalFrameEx() with

Re: [Python-Dev] cpython: Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure

2013-07-16 Thread Victor Stinner
2013/7/16 Serhiy Storchaka : >> http://hg.python.org/cpython/rev/533eb9ab895a >> summary: >>Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure >> >> bytes is NULL on _PyBytes_Resize() failure > > Why not Py_DECREF? Because Py_DECREF(NULL) does crash. Victor _

Re: [Python-Dev] cpython: Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure

2013-07-16 Thread Victor Stinner
2013/7/17 Serhiy Storchaka : > Oh, I meaned Py_XDECREF. Ah ok :-) Well, for this specific code, it can probably be replaced with: if (_PyBytes_Resize(&bytes, n) < 0) return NULL; I'm not sure that _PyBytes_Resize() *always* decref bytes and then set bytes to NULL. I was too

Re: [Python-Dev] Is it safe to call PyEval_EvalFrameEx() with an exception set?

2013-07-16 Thread Victor Stinner
2013/7/16 Victor Stinner : > I would like to add assertions in Python/ceval.c to detect such bugs > earlier. The problem is that some functions rely on the ability to > call PyEval_EvalFrameEx() with an exception set. Is it expected? > Should it be avoided? The current exception can

Re: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

2013-07-17 Thread Victor Stinner
2013/7/7 Charles-François Natali : > Well, it complicates the signature and implementation. > If we go the same way, why stop there and not expose O_DSYNC, O_SYNC, > O_DIRECT... I added this counter-argument to the PEP. > If you want precise control over the open() sementics, os.open() is > the w

Re: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

2013-07-17 Thread Victor Stinner
2013/7/7 Cameron Simpson : > On 06Jul2013 14:43, Victor Stinner wrote: > | 2013/7/6 Cameron Simpson : > | > Yes. Please forget I mentioned fork(); it is only relevant if you > | > were offering some facility to undo the addition of cloexec to a > | > Popen passed file d

Re: [Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

2013-07-17 Thread Victor Stinner
2013/7/13 Cameron Simpson : > This sentence: > > The flag is cleared in the child process before executing the > program, the change does not change the flag in the parent process. > > needs a semicolon, not a comma. Fixed. Victor ___ Python-Dev mai

Re: [Python-Dev] [Python-checkins] cpython: Fix #18530. Remove extra stat call from posixpath.ismount

2013-07-22 Thread Victor Stinner
Could you please keep the comment "# A symlink can never be a mount point" ? It is useful. (I didn't know that, I'm not a windows developer.) Victor Le 22 juil. 2013 20:08, "brian.curtin" a écrit : > http://hg.python.org/cpython/rev/240adc564539 > changeset: 84791:240adc564539 > parent: 8

Re: [Python-Dev] [Python-checkins] cpython: Issue #18520: Add a new PyStructSequence_InitType2() function, same than

2013-07-22 Thread Victor Stinner
used when parameters are added. It is not the case here. Victor Le 22 juil. 2013 23:59, "victor.stinner" a écrit : > http://hg.python.org/cpython/rev/fc718c177ee6 > changeset: 84793:fc718c177ee6 > user:Victor Stinner > date:Mon Jul 22 22:24:54 2013 +0

[Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-23 Thread Victor Stinner
Hi, Guido van Rossum and others asked me details on how file descriptors and handles are inherited on Windows, for the PEP 446. http://www.python.org/dev/peps/pep-0446/ I hacked Python 3.4 to add a os.get_cloexec() function (extracted from my implementation of the PEP 433), here are some results.

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-23 Thread Victor Stinner
HANDLE_FLAG_INHERIT flag: they use DuplicateHandle(), whereas SetHandleInformation() could be used (to reuse the existing handle instead of creating a new handle). 2013/7/24 Victor Stinner : > Python functions open(), os.open() and os.dup() create file > descriptors with the HANDLE_FLAG_INHERIT flag set (c

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-24 Thread Victor Stinner
2013/7/24 Richard Oudkerk : >> Wow. Indeed you can -- I just tested this myself. How is this >> accomplished? I guess the CRT has a backchannel to talk to itself when >> it creates a process using spawn*? > > CreateProcess() takes a STARTUPINFO argument with undocumented fields > cbReserved2, lpRes

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-26 Thread Victor Stinner
2013/7/24 Guido van Rossum : > But I'm also ready to propose that all this is such a mess that we > *should* change the default fd/handle inheritance to False, *across > platforms*, and damn the torpedoes -- i.e. accept breaking all > existing 3rd party UNIX code for subprocess creation that bypass

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-26 Thread Victor Stinner
2013/7/26 Antoine Pitrou : >> The main drawback is the additionnal syscalls: on some platforms, 2 >> additional syscalls are need to make a file descriptor non-inheritable >> for each creation of file descriptor. According to my benchmark on the >> implementation of the PEP 433: the overhead of mak

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-26 Thread Victor Stinner
2013/7/26 Antoine Pitrou : > On Fri, 26 Jul 2013 22:17:47 +0200 >> """ >> On Linux, setting the close-on-flag has a low overhead on >> performances. Results of bench_cloexec.py on Linux 3.6: >> >> - close-on-flag not set: 7.8 us >> - O_CLOEXEC: 1% slower (7.9 us) >> - ioctl(): 3% slower (8.0 us) >>

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-27 Thread Victor Stinner
2013/7/27 Guido van Rossum : > Do we even need a new PEP, or should we just do it? Or can we adapt > Victor's PEP 446? I can rewrite the PEP 446 to: * make all file descriptors and handles non-inheritable * remove the cloexec parameter * remove everything about non-blocking sockets (O_NONBLOCK),

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-07-27 Thread Victor Stinner
2013/7/27 Guido van Rossum : > P.S. perhaps more important than a PEP rewrite is a working patch to see how > realistic this is. Could you make the alpha 1 release? I already ran the whole Python test suite with non-inheritable file descriptors when I developed the PEP 433: it just works. So I'm c

[Python-Dev] PEP 446: Open issues/questions

2013-07-27 Thread Victor Stinner
Hi, I have a few more questions on the PEP 446: (A) How should we support support where os.set_inheritable() is not supported? Can we announce that os.set_inheritable() is always available or not? Does such platform exist? (B) Should subprocess make the file descriptors of pass_fds inheritable?

Re: [Python-Dev] PEP 446: Open issues/questions

2013-07-29 Thread Victor Stinner
2013/7/28 Charles-François Natali : > 2013/7/28 Antoine Pitrou : >>> (C) Should we handle standard streams (0: stdin, 1: stdout, 2: stderr) >>> differently? For example, os.dup2(fd, 0) should make the file >>> descriptor 0 (stdin) inheritable or non-inheritable? On Windows, >>> os.set_inheritable(f

Re: [Python-Dev] PEP 446: Open issues/questions

2013-07-29 Thread Victor Stinner
2013/7/28 Antoine Pitrou : >> (B) Should subprocess make the file descriptors of pass_fds >> inheritable? If yes, should it be done before or after the fork? If it >> is done after the fork and before exec, it only affects the child >> process, at least on Linux (the file descriptor is still >> non

Re: [Python-Dev] PEP 446: Open issues/questions

2013-07-30 Thread Victor Stinner
2013/7/28 Charles-François Natali : > Also, it'll be puzzling to have syscalls automatically set the cloexec > flag. I guess a lot of people doing system programming with Python > will get bitten, but that's a discussion we already had months ago... The inheritance of file descriptors (and Windows

Re: [Python-Dev] PEP 446: Open issues/questions

2013-07-30 Thread Victor Stinner
Le 30 juil. 2013 09:11, "Charles-François Natali" a écrit : > > Perhaps this advocates for a global flag, e.g. > > sys.set_default_fd_inheritance(), with False (non-inheritable) being > > the default for sanity and security. > > This looks more and more like PEP 433 :-) I don't like the global mo

Re: [Python-Dev] PEP 446: Open issues/questions

2013-07-30 Thread Victor Stinner
2013/7/30 Richard Oudkerk : > Note that on Windows subprocess has no equivalent of a passfds argument, and > if you redirect the standard streams then you are forced to inherit all > inheritable handles. You can redirect standard streams (stdin, stdout, stderr) using the startup info structure: s

Re: [Python-Dev] PEP 446: Open issues/questions

2013-08-01 Thread Victor Stinner
2013/7/30 Richard Oudkerk : > The documentation for STARTUPINFO says this about STARTF_USESTDHANDLES: > > If this flag is specified when calling one of the process creation > functions, the handles must be inheritable and the function's > bInheritHandles parameter must be set to TRUE. > >

Re: [Python-Dev] PEP 446: Open issues/questions

2013-08-01 Thread Victor Stinner
2013/7/30 Victor Stinner : > I would be nice to have a "pass_handles" on Windows. I'm not sure that it's possible to implement this atomically. It's probably better to leave the application to choose how the inheritance is defined. Example: for handle in handles:

Re: [Python-Dev] PEP 446: Open issues/questions

2013-08-01 Thread Victor Stinner
2013/7/28 Antoine Pitrou : >> (A) How should we support support where os.set_inheritable() is not >> supported? Can we announce that os.set_inheritable() is always >> available or not? Does such platform exist? > > FD_CLOEXEC is POSIX: > http://pubs.opengroup.org/onlinepubs/9699919799/functions/fcn

Re: [Python-Dev] PEP 446: Open issues/questions

2013-08-01 Thread Victor Stinner
2013/7/30 Antoine Pitrou : > Le Tue, 30 Jul 2013 09:09:38 +0200, > Charles-François Natali a écrit : >> This looks more and more like PEP 433 :-) >> >> And honestly, when I think about it, I think that this whole mess is a >> solution looking for a problem. >> If we don't want to inherit file desc

Re: [Python-Dev] PEP 446: Open issues/questions

2013-08-02 Thread Victor Stinner
Le 2 août 2013 08:32, "Charles-François Natali" a écrit : > > 2013/8/2 Victor Stinner : > > 2013/7/28 Antoine Pitrou : > >>> (A) How should we support support where os.set_inheritable() is not > >>> supported? Can we announce that os.set_inheritable

Re: [Python-Dev] PEP 446: Open issues/questions

2013-08-02 Thread Victor Stinner
http://support.microsoft.com/kb/315939/en-us Ah yes, we may implement pass_handles on Windows using a critical section to inherit *handles*. File descriptors cannot be inherited using CreateProcess(), only using spawn(). Or can we rely on the undocumented fields used by spawn()? Victor _

Re: [Python-Dev] PEP 446: Open issues/questions

2013-08-02 Thread Victor Stinner
Is it possible to implement atfork on Windows? A Python lock would be ignored by other C threads. It is unsafe if Python is embedded. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-08-05 Thread Victor Stinner
2013/7/27 Nick Coghlan : >> Do we even need a new PEP, or should we just do it? Or can we adapt >> Victor's PEP 446? > > Adapting the PEP sounds good - while I agree with switching to a sane > default, I think the daemonisation thread suggests there may need to > be a supporting API to help force F

Re: [Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

2013-08-05 Thread Victor Stinner
I checked the python-daemon module: it closes all open file descriptors except 0, 1, 2. It has a files_preserve attribute to keep some FD opens. It redirects stdin, stdout and stderr to /dev/null and keep these file descriptors open. If python-daemon is used to execute a new program, the files_pres

[Python-Dev] (New) PEP 446: Make newly created file descriptors non-inheritable

2013-08-05 Thread Victor Stinner
HTML version: http://www.python.org/dev/peps/pep-0446/ You may also want to read my first attempt: http://www.python.org/dev/peps/pep-0433/ PEP: 446 Title: Make newly created file descriptors non-inheritable Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type

Re: [Python-Dev] (New) PEP 446: Make newly created file descriptors non-inheritable

2013-08-05 Thread Victor Stinner
> On Windows, the ``subprocess`` closes all handles and file descriptors > in the child process by default. If at least one standard stream (stdin, > stdout or stderr) is replaced (ex: redirected into a pipe), all > inheritable handles are inherited in the child process. > > Summary: > > ==

Re: [Python-Dev] (New) PEP 446: Make newly created file descriptors non-inheritable

2013-08-06 Thread Victor Stinner
2013/8/6 Victor Stinner : > Oh, the summary table is wrong for the "subprocess, default" line: all > inheritable handles are inherited if at least one standard stream is > replaced. I updated the PEP: - add a new section "Performances of Closing All File Descriptors"

Re: [Python-Dev] (New) PEP 446: Make newly created file descriptors non-inheritable

2013-08-07 Thread Victor Stinner
> Also the socket library creates sockets with inheritable handles by default. Apparently there isn't a reliable way to make sockets non-inheritable because anti-virus/firewall software can interfere: > > http://stackoverflow.com/questions/12058911/can-tcp-socket-handles-be-set-not-inheritable Re

Re: [Python-Dev] Green buildbot failure.

2013-08-11 Thread Victor Stinner
2013/8/11 David Bolen : >> Was the test terminated because it took too long? > > Yes, it looks like it. > > This test (and one on the XP-4 buildbot in the same time frame) was > terminated by an external watchdog script that kills python_d > processes that have been running for more than 2 hours.

Re: [Python-Dev] (New) PEP 446: Make newly created file descriptors non-inheritable

2013-08-11 Thread Victor Stinner
Hi, I fixed various bugs in the implementation of the (new) PEP 446: http://hg.python.org/features/pep-446 At revision da685bd67524, the full test suite pass on: - Fedora 18 (Linux 3.9), x86_64 - FreeBSD 9.1, x86_64 - Windows 7 SP1, x86_64 - OpenIndiana (close to Solaris 11), x86_64 Some tests

Re: [Python-Dev] (New) PEP 446: Make newly created file descriptors non-inheritable

2013-08-11 Thread Victor Stinner
2013/8/12 Victor Stinner : > I fixed various bugs in the implementation of the (new) PEP 446: > http://hg.python.org/features/pep-446 > > At revision da685bd67524, the full test suite pass on: (...) I also checked the usage of atomic flags. There was a minor bug on Linux, it is now f

Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks

2013-08-12 Thread Victor Stinner
>I'm currently working on implementing Import Hooks (PEP302) with Python > 2.7 to be able to import modules whose code is in ZODB. However, I have > stumbled upon a widely known issue about import deadlock[0][1] (...) In Python 3.3, the import machinery has been rewritten (importlib is used by def

Re: [Python-Dev] Guidance regarding tests for the standard lib

2013-08-13 Thread Victor Stinner
Send the patch somewhere (ex: attach it to an email, or to the bug tracker, as you want), or give the error message, if you want some help. > Ask for a pronouncement on the PEP first, and then fix the test breakage > later? Sometimes, it's possible to pronounce on a PEP without a working impleme

Re: [Python-Dev] Deprecating the formatter module

2013-08-15 Thread Victor Stinner
2013/8/15 Antoine Pitrou : > We don't have any substantial change in store for an eventual "Python > 4", so it's quite a remote hypothesis right now. I prefered the transition between Linux 2 and Linux 3 (no major change, just a "normal" release except the version), rather than the transition betw

[Python-Dev] PEP 446: issue with sockets

2013-08-20 Thread Victor Stinner
Hi, I have a new question for my PEP 446 (inheritance of file descriptors). os.get/set_inheritable(handle) has strange behaviour on Windows, and so I would like to add new os.get/set_handle_inheritable() functions to avoid it. The problem is that a socket would require a different function depend

Re: [Python-Dev] PEP 446: issue with sockets

2013-08-20 Thread Victor Stinner
2013/8/21 Victor Stinner : > Should I add a portable helper to the > socket module (socket.get/set_inheritable)? Add the two following functions to the socket module: def get_inheritable(sock): if os.name == 'nt': return os.get_handle_inheritable(sock.fi

Re: [Python-Dev] PEP 446: issue with sockets

2013-08-20 Thread Victor Stinner
2013/8/21 Guido van Rossum : > Also, are you sure the things returned by socket.fleno() are really Windows > handles? I thought they were some other artificial namespace used just by > sockets. (You know what? I know understand and love the UNIX concept "everything is file"!) I don't know if a so

Re: [Python-Dev] PEP 446: issue with sockets

2013-08-20 Thread Victor Stinner
2013/8/21 Guido van Rossum : > Since this is a new API and only applies to sockets, making them methods > sounds good. (I'd put the 'nt' test outside the method defs though so they > are tested only once per import.) I added get_inheritable() and set_inheritable() methods to socket.socket. The nam

Re: [Python-Dev] PEP 446: issue with sockets

2013-08-21 Thread Victor Stinner
2013/8/21 Richard Oudkerk : > On 21/08/2013 1:19am, Victor Stinner wrote: >> I don't know if a socket handle is similar to file handles or if they >> are specials. At least, GetHandleInformation() and >> SetHandleInformation() functions, used by >> os.get/set_han

Re: [Python-Dev] cpython: Cleanup test_builtin

2013-08-22 Thread Victor Stinner
est input, below fp = open(TESTFN, 'w') self.addCleanup(unlink, TESTFN) ... Victor 2013/8/22 Serhiy Storchaka : > 22.08.13 02:59, victor.stinner написав(ла): > >> http://hg.python.org/cpython/rev/0a1e1b929665 >> changeset: 85308:0a1e1b929665

[Python-Dev] PEP 446 (make FD non inheritable) ready for a final review

2013-08-22 Thread Victor Stinner
Hi, I know that I wrote it more than once, but I consider that my PEP 446 is now ready for a final review: http://www.python.org/dev/peps/pep-0446/ The implementation is also working, complete and ready for a review. http://hg.python.org/features/pep-446 http://bugs.python.org/issue18571 I run

Re: [Python-Dev] PEP 446 (make FD non inheritable) ready for a final review

2013-08-23 Thread Victor Stinner
Hi, I will try to answer to your worries. Tell me if I should complete the PEP with these answers. 2013/8/23 Charles-François Natali : > Why does dup2() create inheritable FD, and not dup()? Ah yes, there were as section explaining it. In a previous version of the PEP (and its implementation), o

Re: [Python-Dev] [Python-checkins] cpython (merge 3.3 -> default): merge for issue #18755

2013-08-23 Thread Victor Stinner
2013/8/23 brett.cannon : > http://hg.python.org/cpython/rev/7d30ecf5c916 > changeset: 85339:7d30ecf5c916 > parent: 85336:391f36ef461a > parent: 85337:ddd610cb65ef > user:Brett Cannon > date:Fri Aug 23 11:52:19 2013 -0400 > summary: > merge for issue #18755 > > files:

Re: [Python-Dev] PEP 446 (make FD non inheritable) ready for a final review

2013-08-26 Thread Victor Stinner
2013/8/26 Guido van Rossum : > I have reviewed the PEP and I think it is good. Thank you so much for > pushing this topic and for your very thorough review of all the feedback, > related issues and so on. It is an exemplary PEP! Thanks :-) I updated the PEP: http://hg.python.org/peps/rev/edd8250f6

Re: [Python-Dev] PEP 446 (make FD non inheritable) ready for a final review

2013-08-27 Thread Victor Stinner
2013/8/27 Antoine Pitrou : >> On UNIX, the subprocess module closes almost all file descriptors in >> the child process. This operation requires MAXFD system calls, where >> MAXFD is the maximum number of file descriptors, even if there are >> only few open file descriptors. This maximum can be rea

Re: [Python-Dev] Accepted: PEP 446 -- Make newly created file descriptors non-inheritable

2013-08-27 Thread Victor Stinner
2013/8/28 Guido van Rossum : > Congratulations Victor, PEP 446 is accepted! Thanks. I just commited the implementation into default (future Python 3.4): http://hg.python.org/cpython/rev/ef889c3d5dc6 http://bugs.python.org/issue18571 I tested it on Linux, FreeBSD 9, OpenIndiana and Windows 7. Let

[Python-Dev] Test the test suite?

2013-08-28 Thread Victor Stinner
Hi, I just noticed that tests using @requires_freebsd_version and @requires_linux_version decorator from test.support are never run since this commit (almost 2 years ago): changeset: 72618:3b1859f80e6d user:Charles-François Natali date:Mon Oct 03 19:40:37 2011 +0200 files:

Re: [Python-Dev] cpython: Issue #18571: Implementation of the PEP 446: file descriptors and file handles

2013-08-28 Thread Victor Stinner
2013/8/28 Antoine Pitrou : > I don't want to sound too demanding, but was this patch actually > reviewed? I can't find a single review comment in > http://bugs.python.org/issue18571 No, it was not. The first patch for the PEP 446 (issue #18571) was available for a review approximatively one month

Re: [Python-Dev] cpython: Issue #18571: Implementation of the PEP 446: file descriptors and file handles

2013-08-28 Thread Victor Stinner
2013/8/28 Antoine Pitrou : > Well, reviewing a 1500-line commit is not very doable. You can use Rietveld if you prefer: http://bugs.python.org/review/18571/#ps9085 The commit is this patch + changes to Misc/NEWS and Doc/whatnews/3.4.rst. Victor ___ Pyt

[Python-Dev] Add a new tracemalloc module to trace memory allocations

2013-08-28 Thread Victor Stinner
Hi, Thanks to the PEP 445, it becomes possible to trace easily memory allocations. I propose to add a new tracemalloc module computing the memory usage per file and per line number. It has also a private method to retrieve the location (filename and line number) of a memory allocation of an object

Re: [Python-Dev] Add a new tracemalloc module to trace memory allocations

2013-08-28 Thread Victor Stinner
2013/8/29 Victor Stinner : > My proposed implementation for Python 3.4 is different: > > * no enable() / disable() function: tracemalloc can only be enabled > before startup by setting PYTHONTRACEMALLOC=1 environment variable > > * traces (size of the memory block, Python fil

Re: [Python-Dev] Add a new tracemalloc module to trace memory allocations

2013-08-29 Thread Victor Stinner
2013/8/29 Brett Cannon : >> I also created a "pyfailmalloc" project based on the PEP 445 to inject >> MemoryError exceptions. (...) > > Would extension module authors find it useful? I don't know, I created two months ago and I didn't made a public annoucement. > If so maybe we need a malloc pack

Re: [Python-Dev] Add function to signal module for getting main thread id

2013-08-30 Thread Victor Stinner
2013/8/30 Andrew Svetlov : > Tulip uses check like > threading.current_thread().name == 'MainThread' You should use the identifier, not the name: threading.current_thread().ident. > This approach has a problem: thread name is writable attribute and can > be changed by user code. The ident at

Re: [Python-Dev] Add a new tracemalloc module to trace memory allocations

2013-08-31 Thread Victor Stinner
Le 31 août 2013 19:09, "Gregory P. Smith" a écrit : > First, I really like this. +1 Current votes: +3 (i also want tracemalloc!). No opposition against such addition. > We should be consistent with faulthandler's options. Why do you not want to support both the env var and enable()/disable() fu

Re: [Python-Dev] SEEK_* constants in io and os

2013-09-02 Thread Victor Stinner
2013/9/2 Eli Bendersky : > Yes, now I see a 500 usec difference timed within the Python script. When > timing the whole execution of Python: (...) Can you please provide the list of imported modules by: python -c 'import sys; print(sys.modules)' For python with default options and for python with

[Python-Dev] RFC: PEP 454: Add a new tracemalloc module

2013-09-03 Thread Victor Stinner
module. http://hg.python.org/features/tracemalloc/file/tip/Doc/library/tracemalloc.rst The documentaion contains examples and a short "tutorial". PEP: 454 Title: Add a new tracemalloc module to trace Python memory allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinn

Re: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module

2013-09-03 Thread Victor Stinner
> ``get_object_trace(obj)`` function: > > Get the trace of a Python object *obj* as a ``trace`` instance. > > Return ``None`` if the tracemalloc module did not save the location > when the object was allocated, for example if the module was > disabled. This function and get_traces(

Re: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module

2013-09-04 Thread Victor Stinner
>> ``trace`` class: >> This class represents debug information of an allocated memory block. >> >> ``size`` attribute: >> Size in bytes of the memory block. >> ``filename`` attribute: >> Name of the Python script where the memory block was allocated, >> ``None`` if unknown. >> ``lin

Re: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module

2013-09-05 Thread Victor Stinner
2013/9/5 Alexander Belopolsky : > Please mention that this API is similar to that of faulthandler and add a > link to faulthandler docs. Done. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev U

Re: [Python-Dev] Add a new tracemalloc module to trace memory allocations

2013-09-05 Thread Victor Stinner
2013/9/1 Nick Coghlan : > +1 from me for both tracemalloc and failmalloc in the same vein as > faulthandler (and using similar API concepts to faulthandler). > > While I like the flat top level naming structure, we should clearly document > these as implementation dependent runtime services. Other

Re: [Python-Dev] Add a new tracemalloc module to trace memory allocations

2013-09-05 Thread Victor Stinner
2013/8/31 Gregory P. Smith : > Think of the possibilities, you could even setup a test runner to > enable/disable before and after each test, test suite or test module to > gather narrow statistics as to what code actually _caused_ the allocations > rather than the ultimate individual file/line doi

Re: [Python-Dev] cpython: Issue #18904: test_socket: add inheritance tests using fcntl and FD_CLOEXEC

2013-09-08 Thread Victor Stinner
2013/9/8 Antoine Pitrou : > On Sun, 8 Sep 2013 11:54:00 +0200 (CEST) > victor.stinner wrote: >> http://hg.python.org/cpython/rev/b7f6f6f59e91 >> changeset: 85619:b7f6f6f59e91 >> user:Victor Stinner >> date:Sun Sep 08 11:53:09 2013 +0200

Re: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module

2013-09-08 Thread Victor Stinner
2013/9/4 Victor Stinner : > http://www.python.org/dev/peps/pep-0454/ > > PEP: 454 > Title: Add a new tracemalloc module to trace Python memory allocations > Version: $Revision$ > Last-Modified: $Date$ > Author: Victor Stinner > Status: Draft > Type: Standards Track

Re: [Python-Dev] RFC: PEP 454: Add a new tracemalloc module

2013-09-08 Thread Victor Stinner
2013/9/8 Janzert : > It seems like most of this could live on PyPi for a while so the API can get > hashed out in use? The pytracemalloc is available on PyPI since 6 months. The only feedback I had was something trying to compile it on Windows (which is complex because of the dependency to glib, I

Re: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0a2

2013-09-09 Thread Victor Stinner
2013/9/9 Larry Hastings : > Python 3.4 includes a range of improvements of the 3.x series, including > hundreds of small improvements and bug fixes. Major new features and > changes in the 3.4 release series so far include: > > * PEP 446, changing file descriptors to not be inherited by default >

Re: [Python-Dev] [python-committers] [RELEASED] Python 3.4.0a2

2013-09-09 Thread Victor Stinner
2013/9/9 Antoine Pitrou : > Le Mon, 9 Sep 2013 14:30:50 +0200, > Victor Stinner a écrit : >> 2013/9/9 Larry Hastings : >> > Python 3.4 includes a range of improvements of the 3.x series, >> > including hundreds of small improvements and bug fixes. Major new >&g

Re: [Python-Dev] Add a "transformdict" to collections

2013-09-10 Thread Victor Stinner
2013/9/10 Antoine Pitrou : > In http://bugs.python.org/issue18986 I proposed adding a new mapping > type to the collections module. > > The original use case is quite common in network programming and > elsewhere (Eric Snow on the tracker mentioned an application with stock > symbols). You want to

<    10   11   12   13   14   15   16   17   18   19   >