[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

STINNER Victor added the comment: Thanks Ed Maste, Conrad Meyer, Kyle Evans and Kubilay Kocak! I ran a benchmark: vstinner@freebsd$ env/bin/python -m pyperf command -v -- ./python -c 'import subprocess, sys; args=[sys.executable, "-sS", "-c", "pass"]; subprocess.run(args)' The optimization

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

STINNER Victor added the comment: New changeset e6f8abd500751a834b6fff4f107ecbd29f2184fe by Victor Stinner in branch 'master': bpo-38061: subprocess uses closefrom() on FreeBSD (GH-19697) https://github.com/python/cpython/commit/e6f8abd500751a834b6fff4f107ecbd29f2184fe -- _

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

STINNER Victor added the comment: New changeset 162c567d164b5742c0d1f3f8bd8c8bab9c117cd0 by Victor Stinner in branch 'master': bpo-38061: os.closerange() uses closefrom() on FreeBSD (GH-19696) https://github.com/python/cpython/commit/162c567d164b5742c0d1f3f8bd8c8bab9c117cd0 --

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

STINNER Victor added the comment: Copy of the very interesting https://svnweb.freebsd.org/ports?view=revision&revision=518640 commit message by koobs: Log Message: lang/python{27,35,36,37,38}: Add closefrom(2) support A single close(fd) syscall is cheap, but when MAXFDS (maximum file

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

STINNER Victor added the comment: Links to the FreeBSD downstream patches: * https://reviews.freebsd.org/rP518640 * https://svnweb.freebsd.org/ports?view=revision&revision=518640 * https://svnweb.freebsd.org/ports/head/lang/python38/files/ * https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=24

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

Change by STINNER Victor : -- pull_requests: +19017 pull_request: https://github.com/python/cpython/pull/19697 ___ Python tracker ___ __

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

Change by STINNER Victor : -- keywords: +patch pull_requests: +19016 stage: -> patch review pull_request: https://github.com/python/cpython/pull/19696 ___ Python tracker ___ _

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

STINNER Victor added the comment: FreeBSD change has been merged: "lang/python{27,35,36,37,38}: Add closefrom(2) support" https://svnweb.freebsd.org/ports?view=revision&revision=518640 -- ___ Python tracker

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

Kubilay Kocak added the comment: On "close_except", close_range and posix_spawn(close_fds=True), I'll talk to my interested FreeBSD colleagues about potential inroads in that regard. In the meantime, we're good on closefrom(2) anywhere it can be used in Python -- ___

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

STINNER Victor added the comment: > That's very unlikely to happen. I believe it was added as an opt-in feature > for some linux compatibility situations. The correct solution is to use > closefrom(2), as it is the optimal current solution, and in our case, safe to > use. If you have like 5

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

Kubilay Kocak added the comment: > Would it be possible to modify FreeBSD to enable it by default? Or is there a > reason to not enable it by default? That's very unlikely to happen. I believe it was added as an opt-in feature for some linux compatibility situations. The correct solution is

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

STINNER Victor added the comment: > @Victor I mounted fdescfs on the buildbot workers to make the tests run > faster. You're correct that it's not enabled by default. Would it be possible to modify FreeBSD to enable it by default? Or is there a reason to not enable it by default? > We're tr

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

Kubilay Kocak added the comment: @Victor I mounted fdescfs on the buildbot workers to make the tests run faster. You're correct that it's not enabled by default. If we could look at the initial support being as simple as possible, we can look to backport this to 2.7 as well --

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

Kubilay Kocak added the comment: We're tracking this in our downstream bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221700 There's a patch there you may base something upstream on Open question: closefrom(2) test in ./configure && ifdef __FreeBSD__ ? -- _

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

STINNER Victor added the comment: > FreeBSD has a similar concept using /dev/fd "file-descriptor file system". > (...) I'm not sure how it is supposed to work. Sadly, on my FreeBSD VM, it seems like /dev/fd/ is not mounted with fdescfs by default, but as a regular directory with 3 hardcoded

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

STINNER Victor added the comment: Oh, _posixsubprocess uses /dev/fd/ on FreeBSD, but only if it's mounted file system (expected to be fdescfs filesystem): #if defined(__FreeBSD__) || (defined(__APPLE__) && defined(__MACH__)) # define FD_DIR "/dev/fd" #else # define FD_DIR "/proc/self/fd" #end

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

STINNER Victor added the comment: > On Linux, _posixsubprocess lists the content of /proc/self/fd/ to only close > open file descriptor, after fork() and before exec(). This code is specific to Linux because it uses the SYS_getdents64 syscall. FreeBSD has a similar concept using /dev/fd "fil

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

STINNER Victor added the comment: See also bpo-13788: "os.closerange optimization". -- ___ Python tracker ___ ___ Python-bugs-list

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

Change by STINNER Victor : -- nosy: +koobs ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

STINNER Victor added the comment: A workaround is to use close_fds=False which is only safe if all code used in Python implements the PEP 446. This PEP mentions the FreeBSD issue, mentioning bpo-11284 with MAXFD=655,000: "The operation can be slow if MAXFD is large. For example, on a FreeBSD

[issue38061] FreeBSD: Optimize subprocess.Popen(close_fds=True) using closefrom()

New submission from STINNER Victor : The default value of subprocess.Popen "close_fds" parameter changed to True in Python 3. Compared to Python 2, close_fds=True can make Popen 10x slower: see bpo-37790. A single close(fd) syscall is cheap. But when MAXFDS (maximum file descriptor number) i