[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: > Note that on Windows there is an O_NOINHERIT flag which > almost corresponds to O_CLOEXEC on Linux. > I don't think there is a need to use the win32 api. Ah yes. Because this issue is closed, I created the issue #16850 which is more specific to open + close-a

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Richard Oudkerk added the comment: Note that on Windows there is an O_NOINHERIT flag which almost corresponds to O_CLOEXEC on Linux. I don't think there is a need to use the win32 api. -- nosy: +sbt ___ Python tracker

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Amaury Forgeot d'Arc added the comment: > So why not to add 'e' character You said it: because it can't be written consistently on all platforms. For example, python does not use CreateFile on Windows, see #12939. -- ___ Python tracker

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Марк Коренберг added the comment: Well, I understand. So why not to add 'e' (and 'N', which is the same meaning) character, which: * use O_CLOEXEC in modern Linux * generate Exception if O_CLOEXEC is not supported (or does not work) on platform. Also, implement O_CLOEXEC for open() in Windo

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Amaury Forgeot d'Arc added the comment: And to be explicit, you can now write: def open_cloexex(filename, mode='r'): return open(filename, mode, opener=lambda path, mod: os.open(path, mod|os.O_CLOEXEC)) -- ___ Python tracker

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Charles-François Natali added the comment: > Why it is closed as duplicate? nothing said about CLOEXEC in issue12797 See http://bugs.python.org/issue12760#msg146686 -- ___ Python tracker _

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Марк Коренберг added the comment: Why it is closed as duplicate? nothing said about CLOEXEC in issue12797 -- ___ Python tracker ___ _

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Changes by STINNER Victor : -- resolution: fixed -> duplicate ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: h

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Changes by Charles-François Natali : -- status: open -> closed superseder: -> io.FileIO and io.open should support openat ___ Python tracker ___

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: > > Why not to use CreateFile() on Windows platform? > Good idea! Please open a separate issue for it. Done, issue #12939. -- ___ Python tracker _

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: See also issue #12760 (Add create mode to open). -- ___ Python tracker ___ ___ Python-bugs-list mai

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Amaury Forgeot d'Arc added the comment: > Why not to use CreateFile() on Windows platform? Good idea! Please open a separate issue for it. -- ___ Python tracker ___

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Марк Коренберг added the comment: Some times ago, Python has used fopen() for open() implementation. Now, it uses OS-kernel native function to open files. AFAIK, open() in Windows is a wrapper around CreateFile, created to support some POSIX programs in Windows. Why not to use CreateFile() on

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: Windows provides a _get_osfhandle() function. There is not the opposite function? :-) Anyway, O_CLOEXEC is not available on all platforms. Even on FreeBSD and Linux, it depends on the OS/kernel version. -- ___ Pyt

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Amaury Forgeot d'Arc added the comment: > O_CLOEXEC is not linux-only. Windows has the same flag. > In file-opening functions there is lpSecurityAttributes argument How do you suggest to use it? Even on Windows, python calls open(). And lpSecurityAttributes is an argument of CreateFile (which

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Alexey Smirnov added the comment: FreeBSD 8+ also supports O_CLOEXEC in open(). -- ___ Python tracker ___ ___ Python-bugs-list mailin

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Марк Коренберг added the comment: O_CLOEXEC is not linux-only. Windows has the same flag. In file-opening functions there is lpSecurityAttributes argument. And there is bInheritHandle member of corresponding structure. http://msdn.microsoft.com/en-us/library/aa379560(v=VS.85).aspx : bInheritH

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Charles-François Natali added the comment: Checking the kernel version did the trick, the test now run fines on the buildbots. Thanks Victor. Re-closing. -- status: open -> closed ___ Python tracker _

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Roundup Robot added the comment: New changeset 9a16fa0c9548 by Victor Stinner in branch 'default': Issue #12105: test_posix skips test_oscloexec() on Linux < 2.6.23 http://hg.python.org/cpython/rev/9a16fa0c9548 -- ___ Python tracker

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Charles-François Natali added the comment: @Victor Since linux_version() won't be added to the platform module, could you add it to test.support so that it can be used in the O_CLOEXEC test? -- ___ Python tracker

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Charles-François Natali added the comment: > This is a kernel bug, not a bug in the GNU libc (ask Ulrich if you are not > sure ;-)). Kernels prior to 2.6.23 didn't know about the O_CLOEXEC flag: to catch this kind of problem, every syscall would have to check every bit when it's passed a com

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: > test_socket: ...has a nice linux_version() which should be moved to > the platform module I created the issue #12158 for that. -- ___ Python tracker __

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: > The real issue is that the libc defines O_CLOEXEC, but kernels prior > to 2.6.23 don't support it: instead of returning EINVAL, the socket > syscall silently ignores the flag (don't know why I made the comment > about this flag being defined to 0...). This i

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Charles-François Natali added the comment: The real issue is that the libc defines O_CLOEXEC, but kernels prior to 2.6.23 don't support it: instead of returning EINVAL, the socket syscall silently ignores the flag (don't know why I made the comment about this flag being defined to 0...). IMHO

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: > My commit bff9265d677d will tell use the value of O_CLOEXEC on the > "Linux-2.6.22-vs2.2.0.7-gentoo-..." buildbot. Here you have: "AssertionError: 0 is not true : CLOEXEC flag not set (O_CLOEXEC=0x8)" It's the same value on my Debian Sid. So we cannot te

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: Another idea is to write a best-effort function to open a file with CLOEXEC flag: * use O_CLOEXEC if available * use fcntl(fd, F_SETFD, flags | FD_CLOEXEC) if O_CLOEXEC is missing or was silently ignored by the kernel (by open) Attached open_cloexec.py is a

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: Story of O_CLOEXEC in the GNU libc, by Ulrich Drepper: "Secure File Descriptor Handling" http://udrepper.livejournal.com/20407.html -- > I could either add some voodoo configure checks to ensure > that O_CLOEXEC is indeed supported Hum, if I patch tempfile._

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Antoine Pitrou added the comment: > I like this solution, but I don't know how to test that the kernel > doesn't support O_CLOEXEC. My commit bff9265d677d will tell use the > value of O_CLOEXEC on the > "Linux-2.6.22-vs2.2.0.7-gentoo-i686-Intel-R-_Xeon-TM-_CPU_2.80GHz-with-gentoo-2.0.1 > little

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: > or just skip the test on Linux kernels older than 2.6.23 I like this solution, but I don't know how to test that the kernel doesn't support O_CLOEXEC. My commit bff9265d677d will tell use the value of O_CLOEXEC on the "Linux-2.6.22-vs2.2.0.7-gentoo-i686-In

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Roundup Robot added the comment: New changeset bff9265d677d by Victor Stinner in branch 'default': Issue #12105: test_posix, add the value of O_CLOEXEC in the error message http://hg.python.org/cpython/rev/bff9265d677d -- ___ Python tracker

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Charles-François Natali added the comment: > And apparently some buildbot doesn't like it Linux-2.6.22-vs2.2.0.7-gentoo-i686-Intel-R-_Xeon-TM-_CPU_2.80GHz-with-gentoo-2.0.1 little-endian O_CLOEXEC support was added to Linux 2.6.23: this means that the libc defines it while the kernel doesn't

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Antoine Pitrou added the comment: > Python doesn't suppose atomic open+CLOEXEC anymore, I consider this as a > regression from Python 2 (which support open("re") with the GNU libc). It has never been documented (nor supported) so, no, I wouldn't consider it a regression. > But... it tooks some

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: > One moment -- adding a new value to the os module looks like a new > feature to me. Is there any convincing reason why this needs to go to > 3.2? (And it most definitely shouldn't go to 3.1.) Python doesn't suppose atomic open+CLOEXEC anymore, I consider th

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Antoine Pitrou added the comment: And apparently some buildbot doesn't like it: http://www.python.org/dev/buildbot/all/builders/x86%20Gentoo%20Non-Debug%203.x/builds/57/ == FAIL: test_oscloexec (test.test_posix.PosixTester) ---

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Charles-François Natali added the comment: I've committed the patch to 3.3. Since the documentation aspect is traced in Issue #12103, I'm closing this issue. Марк, thanks for reporting this! -- resolution: accepted -> fixed stage: commit review -> committed/rejected status: open -> clo

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Roundup Robot added the comment: New changeset f1c544245eab by Charles-François Natali in branch 'default': Issue #12105: Add O_CLOEXEC to the os module. http://hg.python.org/cpython/rev/f1c544245eab -- nosy: +python-dev ___ Python tracker

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Antoine Pitrou added the comment: No reason. I think this is definitely 3.3 material. -- nosy: +pitrou versions: -Python 3.1, Python 3.2, Python 3.4 ___ Python tracker ___

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Georg Brandl added the comment: One moment -- adding a new value to the os module looks like a new feature to me. Is there any convincing reason why this needs to go to 3.2? (And it most definitely shouldn't go to 3.1.) -- ___ Python tracker

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: Oh, by the way: it would also be nice to add os.O_CLOEXEC to Python 2.7. For example, tempfile._mkstemp_inner() uses: fd = _os.open(file, flags, 0600) _set_cloexec(fd) # fcntl(fd, F_SETFD, flags | FD_CLOEXEC) which is not atomic. --

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: >> open() documentation may explain the os.fdopen(os.open()) "trick" >> to use low-level options like O_SYNC or O_CLOEXEC. > Why not, but I leave it to someone more comfortable with > documentation than me :-) Issue #12103 should be fine for this idea. -

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: @charles-francois.natali: Your patch is ok, you can commit it into 3.1, 3.2, 3.3. But you may wait after 3.2.1. @Georg Brandl: Should we wait after Python 3.2.1 for this issue? -- nosy: +georg.brandl ___ Python tra

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Changes by Petri Lehtinen : -- nosy: +petri.lehtinen ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mai

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Charles-François Natali added the comment: > To exclude races (in concurrent threads), this two ops should be done under > lock (GIL?) That won't work, because open(), like other slow syscalls, is called without the GIL held. Furthermore, it wouldn't be atomic anyway (imagine a fork is done

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: > Why not to implement 'e' letter in py3k ? > > In systems where O_CLOEXEC is not supported in open(), flag should be set > non-atomically using fcntl. Having an atomic or non-atomic behaviour depending on the OS is not a good idea. -- _

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Марк Коренберг added the comment: Why not to implement 'e' letter in py3k ? In systems where O_CLOEXEC is not supported in open(), flag should be set non-atomically using fcntl. To exclude races (in concurrent threads), this two ops should be done under lock (GIL?) -- _

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Charles-François Natali added the comment: > Using spawn_python() to check that os.O_CLOEXEC flag is correctly set seems > overkill. Why not just testing fcntl.fcntl(f.fileno(), fcntl.F_GETFL) & > FD_CLOEXEC)? Because I couldn't find a place where the CLOEXEC flag was fully tested (I mean, che

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Amaury Forgeot d'Arc added the comment: +1 for the patch! Note that fdopen is now a simple call to open(), so theses lines are equivalent: python2.7: open(filename, 're') python3.3: open(os.open(filename, os.O_RDONLY|os.O_CLOEXEC)) -- resolution: -> accepted stage: -> commit revi

[issue12105] open() does not able to set flags, such as O_CLOEXEC

STINNER Victor added the comment: Using spawn_python() to check that os.O_CLOEXEC flag is correctly set seems overkill. Why not just testing fcntl.fcntl(f.fileno(), fcntl.F_GETFL) & FD_CLOEXEC)? I don't think that there are OSes with O_CLOEXEC but without fcntl(F_GETFL). > Note that I'm not

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Charles-François Natali added the comment: Here's a patch adding O_CLOEXEC to the os module, with test. This patch makes it possible to open and set a FD CLOEXEC atomically. O_CLOEXEC is part of POSIX.1-2008, supported by the Linux kernel since 2.6.23 and has been committed recently to FreeBSD

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Changes by Alexey Smirnov : -- nosy: +alexey-smirnov ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mai

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Charles-François Natali added the comment: > In python3, one can still use fcntl(f.fileno(), FD_SET, FD_CLOEXEC) Note that it's not atomic. -- nosy: +neologix ___ Python tracker __

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Amaury Forgeot d'Arc added the comment: These "needed" flags are actually nonstandard extensions of the libc fopen() function on some platforms. In python3, one can still use fcntl(f.fileno(), FD_SET, FD_CLOEXEC), but a "flags" parameter would be more convenient. -- nosy: +amaury.for

[issue12105] open() does not able to set flags, such as O_CLOEXEC

Changes by Марк Коренберг : -- components: +Library (Lib) ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:

[issue12105] open() does not able to set flags, such as O_CLOEXEC

New submission from Марк Коренберг : In Python3 I'm not able to use open('xxx', 're') as it does not call fopen() anymore. What about extra flags like 'e'=O_CLOEXEC? P.S. see issue12103 for list of needed flags P.P.S does 2to3 tool properly detect such cases? -- messages: 136259 nosy: