[issue42041] venv subprocess call to python resolves to wrong interpreter
New submission from Mark Keller : Hey, Wanted to report this weird behavior I saw recently. Let me first explain how to reproduce and then talk about where I think the issue comes from. I attached a tar.gz file with 2 Python files in it. Here's how to reproduce: 0. Make sure that pytest is unavailable in your command line, or please edit to another library that you don't have available from another Python install (make sure to update the import statement in `script1.py`). 1. Untar the 2 scripts into current working directory. Note: I reproduced on a Windows VM (Version 10.0.18363 Build 18363). 2. Make a Python 3.7, or 3.8 environment venv. Note: I used Python 3.8.2. 3. Activate venv and install pytest in it. 4. Run `python run_scripts.py` and observe the error. Note: The same thing works on Python 3.6 as expected. My team observed this issue when virtualenv released this Monday. It changed how it works with Windows and Python 3.7+, before it was not using the Windows redirect script generated by venv and just worked like how older versions on Windows do. But what we are seeing is that if you start a subprocess call to `python` then that gets resolved to the system wide Python binary without the venv site-packages being in PYTHONPATH. So the subprocess will fail when importing pytest in the subprocess. If this is not clear please don't hesitate to ask me for clarification, or refer to https://github.com/pypa/virtualenv/issues/1981 for more debugging info. Another interesting thing that I noticed is that if you make a subprocess call to `pytest` instead then that gets resolved correctly. -- components: Windows files: repro.tar.gz messages: 378665 nosy: keller00, paul.moore, steve.dower, tim.golden, vinay.sajip, zach.ware priority: normal severity: normal status: open title: venv subprocess call to python resolves to wrong interpreter versions: Python 3.7, Python 3.8 Added file: https://bugs.python.org/file49520/repro.tar.gz ___ Python tracker <https://bugs.python.org/issue42041> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42041] venv subprocess call to python resolves to wrong interpreter
Mark Keller added the comment: Thank you so much for the prompt responses. I would say that this should be documented on the venv page (https://docs.python.org/3/library/venv.html) as I find the subprocess page too dense of information already, but I do agree that venv page is most likely not where developers would look first. I'm a little embarrassed to say this, but I didn't even know that sys.executable existed. So maybe the better way to document this would be to mention sys.executable on the subprocess page, just as a sure way find where the current interpreter lives and then go into detail why this behavior difference is on the venv page. Maybe have them linked to each other? What do you think about this? -- components: -Documentation versions: +Python 3.7, Python 3.8 -Python 3.10, Python 3.9 ___ Python tracker <https://bugs.python.org/issue42041> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42041] venv subprocess call to python resolves to wrong interpreter
Mark Keller added the comment: This sounds good to me Paul, thank you again! -- ___ Python tracker <https://bugs.python.org/issue42041> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42041] venv subprocess call to python resolves to wrong interpreter
Mark Keller added the comment: Paul explained precisely of what I was going through when I reached out to you with this: > then the "running executable" is "path/to/venv/scripts/python.exe", and so > the path containing the running executable is "path/to/venv/scripts", so a > search for "python" will locate "path/to/venv/scripts/python.exe", because > that's how Windows path search rules work. The problem is that that exe is a > redirector, and the script is *actually* being run by the system Python. However, after reading through your messages and thinking about this for a few days I think it'd be sufficient to point out sys.executable's existence early on in the subprocess docs and then say something like how python is not reliable. Similarly to what Paul said here: https://bugs.python.org/issue42041#msg378727 I used python in my subprocess call because I thought it would always resolve to the the current interpreter. Clearly I don't work a whole lot with Windows... Eryk - I really appreciate your explanation and I think that level of detail should be documented. I'd read it because I'm interested, but that is probably too much detail for someone who is trying to get something done quickly. I think both the quick explanation (and pointing out sys.executable's existence) and the detailed why would have it's place in the docs. Maybe the details could be documented for educational purposes and not as a guarantee of implementation? -- ___ Python tracker <https://bugs.python.org/issue42041> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42041] venv subprocess call to python resolves to wrong interpreter
Mark Keller added the comment: +1 to what Steve said -- ___ Python tracker <https://bugs.python.org/issue42041> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
