Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter
Hello, > I tried to list in the PEP 433 advantages and drawbacks of each option. > > If I recorded correctly opinions, the different options have the > following supporters: > > a) cloexec=False by default > b) cloexec=True by default: Charles-François Natali > c) configurable default value: Antoine Pitrou, Nick Coghlan, Guido van Rossum You can actually count me in the cloexec=False camp, and against the idea of a configurable default value. Here's why: Why cloexec shouldn't be set by default: - While it's really tempting to fix one of Unix historical worst decisions, I don't think we can set file descriptors cloexec by default: this would break some applications (I don't think there would be too many of them, but still), but most notably, this would break POSIX semantics. If Python didn't expose POSIX syscalls and file descriptors, but only high-level file streams/sockets/etc, then we could probably go ahead, but now it's too late. Someone said earlier on python-dev that many people use Python for prototyping, and indeed, when using POSIX API, you expect POSIX semantics. Why the default value shouldn't be tunable: - I think it's useless: if the default cloexec behavior can be altered (either by a command-line flag, an environment variable or a sys module function), then libraries cannot rely on it and have to make file descriptors cloexec on an individual basis, since the default flag can be disabled. So it would basically be useless for the Python standard library, and any third-party library. So the only use case is for application writers that use raw exec() (since subprocess already closes file descriptors > 3, and AFAICT we don't expose a way to create processes "manually" on Windows), but there I think they fall into two categories: those who are aware of the problem of file descriptor inheritance, and who therefore set their FDs cloexec manually, and those who are not familiar with this issue, and who'll never look up a sys.setdefaultcloexec() tunable (and if they do, they might think: "Hey, if that's so nice, why isn't it on by default? Wait, it might break applications? I'll just leave the default then."). - But most importantly, I think such a tunable flag is a really wrong idea because it's a global tunable that alters the underlying operating system semantics. Consider this code: """ r, w = os.pipe() if os.fork() == 0: os.execve(['myprog']) """ With a tunable flag, just by looking at this code, you have no way to know whether the file descriptor will be inherited by the child process. That would be introducing an hidden global variable silently changing the semantics of the underlying operating system, and that's just so wrong. Sure, we do have global tunables: """ sys.setcheckinterval() sys.setrecursionlimit() sys.setswitchinterval() hash_randomization """ But those alter "extralinguistic" behavior, i.e. they don't affect the semantics of the language or underlying operating system in a way that would break or change the behavior of a "conforming" program. Although it's not as bad, just to belabor the point, imagine we introduced a new method: """ sys.enable_integer_division(boolean) Depending on the value of this flag, the division of two integers will either yield a floating point or truncated integer value. """ Global variables are bad, hidden global variables are worse, and hidden global variables altering language/operating system semantics are evil :-) What I'd like to see: - Adding a "cloexec" parameter to file descriptor creating functions/classes is fine, it will make it easier for a library/application writer to create file descriptors cloexec, especially in an atomic way. - We should go over the standard library, and create FDs cloexec if they're not handed over to the caller, either because they're opened/closed before returning, or because the underlying file descriptor is kept private (not fileno() method, although it's relatively rare). That's the approach chosen by glibc, and it makes sense: if another thread forks() while a thread is in the middle of getpwnam(), you don't want to leak an open file descriptor to /etc/passwd (or /etc/shadow). cf ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter
On Fri, Jan 25, 2013 at 6:56 PM, Charles-François Natali wrote: > Hello, > >> I tried to list in the PEP 433 advantages and drawbacks of each option. >> >> If I recorded correctly opinions, the different options have the >> following supporters: >> >> a) cloexec=False by default >> b) cloexec=True by default: Charles-François Natali >> c) configurable default value: Antoine Pitrou, Nick Coghlan, Guido van >> Rossum > > You can actually count me in the cloexec=False camp, and against the > idea of a configurable default value. Here's why: > > Why cloexec shouldn't be set by default: > - While it's really tempting to fix one of Unix historical worst > decisions, I don't think we can set file descriptors cloexec by > default: this would break some applications (I don't think there would > be too many of them, but still), but most notably, this would break > POSIX semantics. If Python didn't expose POSIX syscalls and file > descriptors, but only high-level file streams/sockets/etc, then we > could probably go ahead, but now it's too late. Someone said earlier > on python-dev that many people use Python for prototyping, and indeed, > when using POSIX API, you expect POSIX semantics. > > Why the default value shouldn't be tunable: > - I think it's useless: if the default cloexec behavior can be altered > (either by a command-line flag, an environment variable or a sys > module function), then libraries cannot rely on it and have to make > file descriptors cloexec on an individual basis, since the default > flag can be disabled. So it would basically be useless for the Python > standard library, and any third-party library. So the only use case is > for application writers that use raw exec() (since subprocess already > closes file descriptors > 3, and AFAICT we don't expose a way to > create processes "manually" on Windows), but there I think they fall > into two categories: those who are aware of the problem of file > descriptor inheritance, and who therefore set their FDs cloexec > manually, and those who are not familiar with this issue, and who'll > never look up a sys.setdefaultcloexec() tunable (and if they do, they > might think: "Hey, if that's so nice, why isn't it on by default? > Wait, it might break applications? I'll just leave the default > then."). It's a configurable setting in the same way that -Q makes the behaviour of "/" configurable in Python 2 (so your hypothetical example isn't hypothetical at all - it's a description the -Q option), and -R makes random hashing configurable in 2.7 and 3.2: it means we can change the default behaviour in a future version (perhaps Python 4) while allowing people to easily check if their code operates correctly in that state in the current version. I think the default behaviour needs to be configurable from the environment and the command line, but I don't believe it should be configurable from within the interpreter. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter
2013/1/25 Charles-François Natali : > You can actually count me in the cloexec=False camp, and against the > idea of a configurable default value. Oh ok. > Why cloexec shouldn't be set by default: > - While it's really tempting to fix one of Unix historical worst > decisions, I don't think we can set file descriptors cloexec by > default: this would break some applications (I don't think there would > be too many of them, but still), but most notably, this would break > POSIX semantics. (...) Agreed. > Why the default value shouldn't be tunable: > - I think it's useless: if the default cloexec behavior can be altered > (either by a command-line flag, an environment variable or a sys > module function), then libraries cannot rely on it and have to make > file descriptors cloexec on an individual basis, since the default > flag can be disabled. In my experience, in most cases, the default value of cloexec just doesn't matter at all. If your program relies on the state of the close-on-exec flag: you have to make it explicit, specify cloexec paramater. Example: os.pipe(cloexec=True). If you don't modify your application, it will just not work using -e command line option (or PYTHONCLOEXEC environment variable). But why would you enable cloexec by default if your application is not compatible? > So it would basically be useless for the Python > standard library, and any third-party library. So the only use case is > for application writers that use raw exec() (since subprocess already > closes file descriptors > 3, and AFAICT we don't expose a way to > create processes "manually" on Windows), but there I think they fall > into two categories: those who are aware of the problem of file > descriptor inheritance, and who therefore set their FDs cloexec > manually, and those who are not familiar with this issue, and who'll > never look up a sys.setdefaultcloexec() tunable (and if they do, they > might think: "Hey, if that's so nice, why isn't it on by default? > Wait, it might break applications? I'll just leave the default > then."). The problem is that Python can be embeded in application: the application can start a child process independently of Python. > - But most importantly, I think such a tunable flag is a really wrong > idea because it's a global tunable that alters the underlying > operating system semantics. Consider this code: > """ > r, w = os.pipe() > if os.fork() == 0: > os.execve(['myprog']) > """ If this snippet doesn't work with cloexec enabled by default, you have to write: os.pipe(cloexec=False). > - We should go over the standard library, and create FDs cloexec if > they're not handed over to the caller, either because they're > opened/closed before returning, or because the underlying file > descriptor is kept private (not fileno() method, although it's > relatively rare). That's the approach chosen by glibc, and it makes > sense: if another thread forks() while a thread is in the middle of > getpwnam(), you don't want to leak an open file descriptor to > /etc/passwd (or /etc/shadow). I started to make cloexec explicit *everywhere* in the Python stdlib. I reverted my commit because I think that only a few application start child processes, so doing extra work (additionnal syscalls to set cloexec) would slowdown Python for no gain. See my revert commit to see how many functions need to be modified: http://hg.python.org/features/pep-433/rev/963e450fc24f That's why I really like the idea of being able to configure the default value of the cloexec parameter. By default: no overhead nor backward compatibility issue. Whereas you can set close-on-exec flag *everywhere* if you are concern by all issues of inheriting file descriptors (cases listed in the PEP). In the stdlib, I only specified cloexec parameter where it was required to ensure that it works with any default value. Only a few modules were modified (subprocess, multiprocessing, ...). Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter
> I think the default behaviour needs to be configurable from the > environment and the command line, but I don't believe it should be > configurable from within the interpreter. sys.setdefaultcloexec() is convinient for unit test, but it may also be used by modules. A web framework may want to enable close-on-exec flag by default. The drawback of changing the default value after Python started is that Python may have created file descriptors before, so you cannot guarantee that all existing file descriptors have the flag set. Well, I don't know if sys.setdefaultcloexec() is a good idea or not :-) Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter
Le Fri, 25 Jan 2013 12:28:10 +0100, Victor Stinner a écrit : > > I think the default behaviour needs to be configurable from the > > environment and the command line, but I don't believe it should be > > configurable from within the interpreter. > > sys.setdefaultcloexec() is convinient for unit test, but it may also > be used by modules. A web framework may want to enable close-on-exec > flag by default. > > The drawback of changing the default value after Python started is > that Python may have created file descriptors before, so you cannot > guarantee that all existing file descriptors have the flag set. > > Well, I don't know if sys.setdefaultcloexec() is a good idea or > not :-) Both Charles-François and Nick have good points. sys.setdefaultcloexec() is still useful if you want to force the policy from a Python script's toplevel (it's more practical than trying to fit a command-line option in the shebang line). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter
On Fri, Jan 25, 2013 at 9:36 PM, Antoine Pitrou wrote: > Le Fri, 25 Jan 2013 12:28:10 +0100, > Victor Stinner a écrit : >> > I think the default behaviour needs to be configurable from the >> > environment and the command line, but I don't believe it should be >> > configurable from within the interpreter. >> >> sys.setdefaultcloexec() is convinient for unit test, but it may also >> be used by modules. A web framework may want to enable close-on-exec >> flag by default. >> >> The drawback of changing the default value after Python started is >> that Python may have created file descriptors before, so you cannot >> guarantee that all existing file descriptors have the flag set. >> >> Well, I don't know if sys.setdefaultcloexec() is a good idea or >> not :-) > > Both Charles-François and Nick have good points. > sys.setdefaultcloexec() is still useful if you want to force the > policy from a Python script's toplevel (it's more practical than > trying to fit a command-line option in the shebang line). Right, I'm only -0 on that aspect. It strikes me as somewhat dubious, but it's not obviously wrong the way a runtime equivalent of -Q or -R would be. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter
On Fri, Jan 25, 2013 at 10:07 PM, Nick Coghlan wrote: > On Fri, Jan 25, 2013 at 9:36 PM, Antoine Pitrou wrote: >> Le Fri, 25 Jan 2013 12:28:10 +0100, >> Victor Stinner a écrit : >>> Well, I don't know if sys.setdefaultcloexec() is a good idea or >>> not :-) >> >> Both Charles-François and Nick have good points. >> sys.setdefaultcloexec() is still useful if you want to force the >> policy from a Python script's toplevel (it's more practical than >> trying to fit a command-line option in the shebang line). > > Right, I'm only -0 on that aspect. It strikes me as somewhat dubious, > but it's not obviously wrong the way a runtime equivalent of -Q or -R > would be. I just realised I could be converted to a +0 if the runtime time switch could only be used to set the global default as "cloexec=True" and couldn't be used to switch it back again (for testing purposes, if you only want to switch it on temporarily, use a subprocess). That way, as soon as you saw "sys.setdefaultcloexec()" at the beginning of __main__, you'd know descriptors would only be inherited when cloexec=False was set explicitly. If the default flag can also be turned off globally (rather than being a one-way switch), then you have no idea what libraries might do behind your back. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter
2013/1/25 Nick Coghlan : > I just realised I could be converted to a +0 if the runtime time > switch could only be used to set the global default as "cloexec=True" > and couldn't be used to switch it back again (for testing purposes, if > you only want to switch it on temporarily, use a subprocess). (...) Oh, I like this idea. It does simplify many things :-) (And I agree that subprocess can be used to run a test which requires cloexec to be True by default.) -- I tried to be future-proof. If we decide to enable close-on-exec flag globally by default, how do you disable the flag globally? We may add an inverse command line option and another environment variable (ex: PYTHONNOCLOEXEC), but what about sys.setdefaultcloexec()? In a previous version, my implementation expected an argument for PYTHONCLOEXEC: PYTHONCLOEXEC=0 or PYTHONCLOEXEC=1. I realized that it's not how other options (like PYTHONDONTWRITEBYTECODE) are designed. But do we really want to enable close-on-exec in the future? Charles François has really good arguments against such choice :-) It's maybe better to consider that the default at startup will always be False. So we should only provide different ways to set the default to True. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2013-01-18 - 2013-01-25) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open3837 ( +8) closed 24971 (+29) total 28808 (+37) Open issues with patches: 1673 Issues opened (29) == #16335: Integer overflow in unicode-escape decoder http://bugs.python.org/issue16335 reopened by serhiy.storchaka #16995: Add Base32 support for RFC4648 "Extended Hex" alphabet (patch http://bugs.python.org/issue16995 opened by matthaeus.wander #16996: Reuse shutil.which() in webbrowser module http://bugs.python.org/issue16996 opened by serhiy.storchaka #16997: subtests http://bugs.python.org/issue16997 opened by pitrou #16998: Lost updates with multiprocessing.Value http://bugs.python.org/issue16998 opened by lechten #17002: html.entities ImportError in distutils2/pypi/simple.py on Pyth http://bugs.python.org/issue17002 opened by goibhniu #17003: Unification of read()Â and readline() argument names http://bugs.python.org/issue17003 opened by serhiy.storchaka #17004: Expand zipimport to include other compression methods http://bugs.python.org/issue17004 opened by rhettinger #17005: Add a topological sort algorithm http://bugs.python.org/issue17005 opened by rhettinger #17006: Warn users about hashing secrets? http://bugs.python.org/issue17006 opened by christian.heimes #17009: "Thread Programming With Python" should be removed http://bugs.python.org/issue17009 opened by nedbat #17010: Windows launcher ignores active virtual environment http://bugs.python.org/issue17010 opened by bryangeneolson #17011: ElementPath ignores different namespace mappings for the same http://bugs.python.org/issue17011 opened by scoder #17012: Differences between /usr/bin/which and shutil.which() http://bugs.python.org/issue17012 opened by serhiy.storchaka #17013: Allow waiting on a mock http://bugs.python.org/issue17013 opened by pitrou #17015: mock could be smarter and inspect the spec's signature http://bugs.python.org/issue17015 opened by pitrou #17016: _sre: avoid relying on pointer overflow http://bugs.python.org/issue17016 opened by Nickolai.Zeldovich #17018: Inconsistent behaviour of methods waiting for child process http://bugs.python.org/issue17018 opened by wodny #17020: random.random() generating values >= 1.0 http://bugs.python.org/issue17020 opened by klankschap #17023: Subprocess does not find executable on Windows if it is PATH w http://bugs.python.org/issue17023 opened by pekka.klarck #17024: cElementTree calls end() on parser taget even if start() fails http://bugs.python.org/issue17024 opened by scoder #17025: reduce multiprocessing.Queue contention http://bugs.python.org/issue17025 opened by neologix #17026: pdb frames accessible after the termination occurs on uncaught http://bugs.python.org/issue17026 opened by xdegaye #17027: support for drag and drop of files to tkinter application http://bugs.python.org/issue17027 opened by Sarbjit.singh #17028: launcher http://bugs.python.org/issue17028 opened by theller #17029: h2py.py: search the multiarch include dir if it does exist http://bugs.python.org/issue17029 opened by doko #17031: fix running regen in cross builds http://bugs.python.org/issue17031 opened by doko #17032: Misleading error message: global name 'X' is not defined http://bugs.python.org/issue17032 opened by cool-RR #17033: RPM spec file has old config_binsuffix value http://bugs.python.org/issue17033 opened by taavi-burns Most recent 15 issues with no replies (15) == #17031: fix running regen in cross builds http://bugs.python.org/issue17031 #17028: launcher http://bugs.python.org/issue17028 #17027: support for drag and drop of files to tkinter application http://bugs.python.org/issue17027 #17026: pdb frames accessible after the termination occurs on uncaught http://bugs.python.org/issue17026 #17024: cElementTree calls end() on parser taget even if start() fails http://bugs.python.org/issue17024 #17023: Subprocess does not find executable on Windows if it is PATH w http://bugs.python.org/issue17023 #17018: Inconsistent behaviour of methods waiting for child process http://bugs.python.org/issue17018 #17013: Allow waiting on a mock http://bugs.python.org/issue17013 #17010: Windows launcher ignores active virtual environment http://bugs.python.org/issue17010 #16996: Reuse shutil.which() in webbrowser module http://bugs.python.org/issue16996 #16983: header parsing could apply postel's law to encoded words insid http://bugs.python.org/issue16983 #16965: 2to3 should rewrite execfile() to open in 'rb' mode http://bugs.python.org/issue16965 #16964: Add 'm' format specifier for mon_grouping etc. http://bugs.python.org/issue16964 #16962: _posixsubprocess module uses outdated getdents system call http://bugs.python.org/issue16962 #16961: No re
Re: [Python-Dev] Implementation of the PEP 433
I created the following issue to track and review the implementation of the PEP 433: http://bugs.python.org/issue17036 Last changes: - don't hold the GIL to ensure "atomicity" of the close-on-exec flag - apply the default value of cloexec on all C functions - sys.setdefaultcloexec() don't have any argument anymore (unit tests don't use sys.setdefaultcloexec() anymore) Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 433: Choose the default value of the new cloexec parameter
On Sat, Jan 26, 2013 at 12:54 AM, Victor Stinner wrote: > But do we really want to enable close-on-exec in the future? Charles > François has really good arguments against such choice :-) While many of Python's APIs are heavily inspired by POSIX, it still isn't POSIX. It isn't C either, especially in Python 3 (where C truncating division must be written as "//" and our internal text handling has made the migration to being fully Unicode based) > It's maybe > better to consider that the default at startup will always be False. > So we should only provide different ways to set the default to True. I think Charles François actually hit on a fairly good analogy by comparing this transition to the integer division one. To implement that: 1. In 2.x, the "-Q" option was introduced to allow the behaviour to be switched globally, while ensuring it remained consistent for the life of a given application 2. The "//" syntax was introduced to force the use of truncating integer division 3. "float(n) / d" could be used to force floating point division Significantly, code written using either option 2 or option 3 retains exactly the same semantics in Python 3, even though the default behaviour of "/" has now switched from truncating division to floating point division. Note also that in Python 3, there is *no* mechanism to request truncating division globally - if you want truncating division, you have to explicitly request it every time. So, if we agree that "cloexec-by-default" is a desirable goal, despite the inconsistency with POSIX (just as changing integer division was seen as desirable, despite the inconsistency with C), then a sensible transition plan becomes: 1. Introduce a mechanism to switch the behaviour globally, while ensuring it remains consistent for the life of a given application 2. Introduce the "cloexec=None/True/False" 3-valued parameter as needed to allow people to choose between default/definitely-cloexec/definitely-not-cloexec. 3. At some point in the future (perhaps in 3.5, perhaps in 4.0) switch the default behaviour to cloexec=True and remove the ability to change the behaviour globally. The reason I'd be +0 on a "one-way switch", even at runtime, is that you can just make it a no-op after that behaviour becomes the default. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
