[issue8052] subprocess close_fds behavior should only close open fds

STINNER Victor added the comment: "Also, it is not said in manual if getdents() may be interrupted by signal. Assuming current code, error is not checked, so some (or all) descriptors will be skipped in case of error." This issue is closed, please open a new issue to discuss that. --

[issue8052] subprocess close_fds behavior should only close open fds

Марк Коренберг added the comment: Also, it is not said in manual if getdents() may be interrupted by signal. Assuming current code, error is not checked, so some (or all) descriptors will be skipped in case of error. -- ___ Python tracker

[issue8052] subprocess close_fds behavior should only close open fds

STINNER Victor added the comment: "Calling getdents()/readdir64() repeatedly while closing descriptors provides unexpected behaviour." Please open a new issue. -- ___ Python tracker ___

[issue8052] subprocess close_fds behavior should only close open fds

Марк Коренберг added the comment: Calling getdents()/readdir64() repeatedly while closing descriptors provides unexpected behaviour. Reading directory while it modified is not safe by default. For example: http://en.it-usenet.org/thread/18514/15719/. So, we should re-open directory if we rece

[issue8052] subprocess close_fds behavior should only close open fds

Gregory P. Smith added the comment: For FreeBSD, Python 3.2 and 3.3 now check to see if /dev/fd is valid. Be sure and "mount -t fdescfs none /dev/fd" on FreeBSD if you want faster subprocess launching. Run a FreeBSD buildbot? Please do it! For Python 3.1 the fix for #13788 would fix this,

[issue8052] subprocess close_fds behavior should only close open fds

Roundup Robot added the comment: New changeset 754c2eb0a92c by Gregory P. Smith in branch '3.2': Fix FreeBSD, NetBSD and OpenBSD behavior of the issue #8052 fix. http://hg.python.org/cpython/rev/754c2eb0a92c New changeset 7d4658a8de96 by Gregory P. Smith in branch 'default': Fix FreeBSD, NetBSD

[issue8052] subprocess close_fds behavior should only close open fds

Roundup Robot added the comment: New changeset d0acd8169c2a by Gregory P. Smith in branch '3.2': Bugfix for issue #8052 fix on *BSD variants. http://hg.python.org/cpython/rev/d0acd8169c2a New changeset 5be3dadd2eef by Gregory P. Smith in branch '3.2': Another issue #8052 bugfix (related to prev

[issue8052] subprocess close_fds behavior should only close open fds

Roundup Robot added the comment: New changeset 780992c9afea by Gregory P. Smith in branch '3.2': Add a Misc/NEWS entry for issue 8052. http://hg.python.org/cpython/rev/780992c9afea New changeset 1f0a01dc723c by Gregory P. Smith in branch 'default': A Misc/NEWS entry for issue 8052. http://hg.py

[issue8052] subprocess close_fds behavior should only close open fds

Roundup Robot added the comment: New changeset 61aa484a3e54 by Gregory P. Smith in branch '3.2': Fixes issue #8052: The posix subprocess module's close_fds behavior was http://hg.python.org/cpython/rev/61aa484a3e54 New changeset 8879874d66a2 by Gregory P. Smith in branch 'default': Fixes issue

[issue8052] subprocess close_fds behavior should only close open fds

Gregory P. Smith added the comment: attaching a better formatted one for review. -- Added file: http://bugs.python.org/file24256/subprocess-close-open-fds-gps02.diff ___ Python tracker

[issue8052] subprocess close_fds behavior should only close open fds

Gregory P. Smith added the comment: attaching a patch that implements this safely. -- keywords: +patch Added file: http://bugs.python.org/file24255/subprocess-close-open-fds-gps01.diff ___ Python tracker _

[issue8052] subprocess close_fds behavior should only close open fds

Ross Lagerwall added the comment: FreeBSD has a /dev/fd as well as a procfs (deprecated AFAIK). However, both may not be mounted so a patch would *need* to at least fallback to the current functionality. -- ___ Python tracker

[issue8052] subprocess close_fds behavior should only close open fds

Ferringb added the comment: >The only question is: do other Unix also have /proc//fd? e.g. >FreeBSD, OpenBSD. That's especially important because FreeBSD can have >a huge RLIMIT_NOFILE by default. Unless the OS gives some way to optimize the process (whether inferring from procfs, or making us

[issue8052] subprocess close_fds behavior should only close open fds

Charles-François Natali added the comment: > Either way, here's a question: does anyone actually know of a unix that does > procfs, and has a daft opendir implementation as described below? Aka, are > we actually worrying about something relevant, or just hypotheticals? I think it's more the

[issue8052] subprocess close_fds behavior should only close open fds

Ferringb added the comment: In #13788, I've uploaded a patch modifying closerange along the same lines as this discussion; someone w/ appropriate rights should set dependencies as needed. Either way, here's a question: does anyone actually know of a unix that does procfs, and has a daft open

[issue8052] subprocess close_fds behavior should only close open fds

STINNER Victor added the comment: test_subprocess.test_leaking_fds_on_error takes more than 5 minutes on x86 FreeBSD 7.2 build slave! This tests creates 1,024 subprocesses, and subprocess has to close 655,000 file descriptors just to create on child process (whereas there are only something l

[issue8052] subprocess close_fds behavior should only close open fds

Charles-Francois Natali added the comment: > I wonder whether the Java people are simply unaware of the potential problem? > Or perhaps they have checked the Linux and Solaris implementations of > readdir() > and confirmed that it is in fact safe on those platforms. Even if this is the > case,

[issue8052] subprocess close_fds behavior should only close open fds

STINNER Victor added the comment: > I like the approach suggested by Gregory in issue11284 (msg129912) - if we are > sure the program is not multithreaded, get the list of fds before forking, and > use that; otherwise, do it the slow way. A signal handler can also opens new files :-) -

[issue8052] subprocess close_fds behavior should only close open fds

Nadeem Vawda added the comment: > For what its worth, an strace of Java's Process class appears to "cheat" by > opening /proc/self/fd inbetween fork & exec. Looking at the OpenJDK source confirms this: http://hg.openjdk.java.net/jdk6/jdk6/jdk/file/tip/src/solaris/native/java/lang/UNIXProcess_md

[issue8052] subprocess close_fds behavior should only close open fds

Martin v. Löwis added the comment: > Can we use FD_CLOEXEC to archive this goal? I think we should use FD_CLOEXEC in all places where it's reasonable. As others have pointed out, we shouldn't set FD_CLOEXEC for file descriptors where the application hasn't explicitly requested that. -

[issue8052] subprocess close_fds behavior should only close open fds

Ross Lagerwall added the comment: I don't think setting the cloexec flag is a viable solution, especially since fds can be opened in custom c modules. For what its worth, an strace of Java's Process class appears to "cheat" by opening /proc/self/fd inbetween fork & exec. --

[issue8052] subprocess close_fds behavior should only close open fds

Charles-Francois Natali added the comment: Ooops, it's of course not going to break code containing accept + fork or pipe + fork, you obviously also need an execve ;-) But the point is that you can't change the semantics of FDs being inheritable across an execve (think about inetd for example)

[issue8052] subprocess close_fds behavior should only close open fds

Charles-Francois Natali added the comment: If you're suggesting to set FDs CLOEXEC by default, I think it's neither possible nor reasonable: - you have to take into account not only files, but also pipes, sockets, etc - there's no portable way to e.g. open a file and set it CLOEXEC atomically -

[issue8052] subprocess close_fds behavior should only close open fds

STINNER Victor added the comment: Can we use FD_CLOEXEC to archive this goal? Example: open all files with FD_CLOEXEC set and don't close explicitly files on fork. preexec_fn will get access to the files, but the problem is exec(), not preexec_fn. I suppose that it will slow down programs not

[issue8052] subprocess close_fds behavior should only close open fds

Ross Lagerwall added the comment: See #11284 (a duplicate) for more discussion about this issue. -- nosy: +haypo, loewis, neologix, s7v7nislands ___ Python tracker ___ __

[issue8052] subprocess close_fds behavior should only close open fds

Antoine Pitrou added the comment: > Of course, procfs isn't standard in any case; would it be necessary to > have a fallback for systems without it? Or do all *nix systems that we > care about provide it? /proc/self doesn't exist under OpenSolaris. You have to use /proc/. And I suppose there ca

[issue8052] subprocess close_fds behavior should only close open fds

Nadeem Vawda added the comment: According to POSIX [1], if a multi-threaded program calls fork(), the child process may only use async-signal-safe system calls between fork() and exec*(). readdir() is not required to be async-safe [2], so reading /proc/self/fds in the child process is undefin

[issue8052] subprocess close_fds behavior should only close open fds

Ross Lagerwall added the comment: Does this mean that it's better to call the close() syscall 1000 or 1000 times rather than listing the open fds & closing say a handful? On Linux, the listdir function results in an open() syscall, 2 or so getdents() calls and a close() call - could this

[issue8052] subprocess close_fds behavior should only close open fds

Gregory P. Smith added the comment: its still a problem, even the C path has to call close() a million times in that case. thats a huge number of wasted syscalls. fixing this is blocking on a good way to get the list of open fds. I have seen other subprocess code do it using the race condit

[issue8052] subprocess close_fds behavior should only close open fds

Antoine Pitrou added the comment: Is it still a problem now that there's a C path? Furthermore, how do you plan to get a list of open fds? -- nosy: +pitrou ___ Python tracker ___

[issue8052] subprocess close_fds behavior should only close open fds

Changes by Mark Lawrence : -- stage: -> needs patch versions: +Python 3.1 ___ Python tracker ___ ___ Python-bugs-list mailing list Uns