[Python-Dev] Patches to distutils for 3.6
Hi All, There are a couple of related patches that were submitted to distutils a few months ago that I think would be great to have in before the feature freeze in 3.6b1 A bug fix in CCompiler.has_function http://bugs.python.org/issue25544 Adding a has_flag method to CCompiler http://bugs.python.org/issue26689 useful to check if a compiler has certain flags available (such as -std=c++11) Is there any chance of having this merged before the release? Thanks, Sylvain ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Adding PEP 515's '_' formatting
PEP 515 adds underscores to numeric literals. As part of that, it adds optional underscores to numeric formatters (similar to PEP 378 for ','). See https://www.python.org/dev/peps/pep-0515/#further-changes I had assigned myself http://bugs.python.org/issue27080 to implement this. Unfortunately, I'm not going to have time to implement this before 3.6 Beta 1. This isn't a hugely complex project, and it doesn't require knowledge of python internals. So someone with decent C skills should be able to accomplish it in less than a day. I'm hoping someone can pick it up. I can offer some pointers if needed. Eric. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patches to distutils for 3.6
On Fri, Aug 19, 2016 at 11:07 AM, Sylvain Corlay wrote: > Hi All, > > There are a couple of related patches that were submitted to distutils a few > months ago that I think would be great to have in before the feature freeze > in 3.6b1 > > A bug fix in CCompiler.has_function > http://bugs.python.org/issue25544 Hi Sylvain, This doesn't look like a new feature to me so it can be committed after feature freeze. However, the patch needs to be tested in Windows. > Adding a has_flag method to CCompiler > http://bugs.python.org/issue26689 > > useful to check if a compiler has certain flags available (such as > -std=c++11) This needs a test case and documentation update to https://docs.python.org/3.5/distutils/apiref.html#distutils.ccompiler.CCompiler NamedTemporaryFile might not work as expected in Windows (we have some open issues about NamedTemporaryFile, but I'm not a Windows user.) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patches to distutils for 3.6
Hi Berker, Thanks for your reply. If `NamedTemporaryFile` has a bug on windows, should it impact the implementation of this? Sylvain > > Adding a has_flag method to CCompiler > > http://bugs.python.org/issue26689 > > > > useful to check if a compiler has certain flags available (such as > > -std=c++11) > > This needs a test case and documentation update to > https://docs.python.org/3.5/distutils/apiref.html#distutils. > ccompiler.CCompiler > NamedTemporaryFile might not work as expected in Windows (we have some > open issues about NamedTemporaryFile, but I'm not a Windows user.) > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding PEP 515's '_' formatting
On Fri, Aug 19, 2016 at 6:35 PM, Eric V. Smith wrote: > PEP 515 adds underscores to numeric literals. As part of that, it adds > optional underscores to numeric formatters (similar to PEP 378 for ','). See > https://www.python.org/dev/peps/pep-0515/#further-changes > > I had assigned myself http://bugs.python.org/issue27080 to implement this. > Unfortunately, I'm not going to have time to implement this before 3.6 Beta > 1. This isn't a hugely complex project, and it doesn't require knowledge of > python internals. So someone with decent C skills should be able to > accomplish it in less than a day. I'm hoping someone can pick it up. I can > offer some pointers if needed. I'm looking into it. Looks like it's part of the default-locale code - and fortunately doesn't involve messing with actual locale data. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Failures in test_site.py - how to debug?
On my main dev system (Debian Stretch), I've had a single
long-standing test failure - test_site.py,
StartupImportTests.test_startup_imports. It's annoying (partly because
it's such a noisy failure), and doesn't appear to be happening on the
buildbots, nor presumably on core devs' computers, so it's probably
not a Python bug. How can I go about figuring out what's actually
going on here?
The test effectively runs this (but with -v, which makes it noisy):
$ ./python -I -c 'import sys; print(set(sys.modules))'
{'_frozen_importlib_external', 'keyword', '_stat', 'weakref',
'sysconfig', 'genericpath', '_sysconfigdata_m', '_io', '_operator',
'_bootlocale', 'encodings.utf_8', 'itertools', '_locale', 'codecs',
'_weakrefset', 'operator', '_frozen_importlib', 'types',
'_sitebuiltins', 'os', 'posixpath', '_functools', '__main__',
'_weakref', '_warnings', '_codecs', 'builtins', 'posix', 'functools',
'io', 'encodings.aliases', 'collections.abc', 'encodings', '_thread',
'collections', 'errno', '_collections', '_collections_abc', '_heapq',
'_imp', 'site', 'sys', 'reprlib', '_signal', 'stat', 'zipimport',
'abc', 'encodings.latin_1', 'os.path', 'marshal', 'heapq',
'mpl_toolkits'}
It then runs into a failure when it asserts that this set has no
intersection with the set of collection_mods. Apparently 'import sys'
on my system (even with -I for isolation mode) loads up a bunch of
modules that it shouldn't be loading. How do I track down what and
why, and figure out whether it's a config problem unique to my system
or not?
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Fri, Aug 19, 2016, at 10:13, Chris Angelico wrote: > On my main dev system (Debian Stretch), I've had a single > long-standing test failure - test_site.py, > StartupImportTests.test_startup_imports. It's annoying (partly because > it's such a noisy failure), and doesn't appear to be happening on the > buildbots, nor presumably on core devs' computers, so it's probably > not a Python bug. How can I go about figuring out what's actually > going on here? > > The test effectively runs this (but with -v, which makes it noisy): the -v output might be helpful in determining what is causing these modules to be imported. It would at least show what order they're imported in. FWIW, the list of modules that yours has that are not present in my 3.5.2: - _bootlocale _collections _functools _heapq _locale _operator - collections collections.abc functools heapq itertools keyword - mpl_toolkits operator reprlib types weakref Modules that mine has imported but yours does not: - _sysconfigdata sitecustomize > It then runs into a failure when it asserts that this set has no > intersection with the set of collection_mods. Apparently 'import sys' > on my system (even with -I for isolation mode) loads up a bunch of > modules that it shouldn't be loading. I very much doubt that "import sys" is responsible here. > How do I track down what and > why, and figure out whether it's a config problem unique to my system > or not? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Sat, Aug 20, 2016 at 12:31 AM, Random832 wrote:
> the -v output might be helpful in determining what is causing these
> modules to be imported. It would at least show what order they're
> imported in.
Here it is in all its spammy glory.
rosuav@sikorsky:~/cpython$ ./python -I -v -c 'import sys;
print(set(sys.modules))'
import _frozen_importlib # frozen
import _imp # builtin
import sys # builtin
import '_warnings' #
import '_thread' #
import '_weakref' #
import '_frozen_importlib_external' #
import '_io' #
import 'marshal' #
import 'posix' #
import _thread # previously loaded ('_thread')
import '_thread' #
import _weakref # previously loaded ('_weakref')
import '_weakref' #
# installing zipimport hook
import 'zipimport' #
# installed zipimport hook
# /home/rosuav/cpython/Lib/encodings/__pycache__/__init__.cpython-36.pyc
matches /home/rosuav/cpython/Lib/encodings/__init__.py
# code object from
'/home/rosuav/cpython/Lib/encodings/__pycache__/__init__.cpython-36.pyc'
# /home/rosuav/cpython/Lib/__pycache__/codecs.cpython-36.pyc matches
/home/rosuav/cpython/Lib/codecs.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/codecs.cpython-36.pyc'
import '_codecs' #
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object
at 0x7f6d060ccda0>
# /home/rosuav/cpython/Lib/encodings/__pycache__/aliases.cpython-36.pyc
matches /home/rosuav/cpython/Lib/encodings/aliases.py
# code object from
'/home/rosuav/cpython/Lib/encodings/__pycache__/aliases.cpython-36.pyc'
import 'encodings.aliases' #
<_frozen_importlib_external.SourceFileLoader object at 0x7f6d060e1898>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader
object at 0x7f6d060cc978>
# /home/rosuav/cpython/Lib/encodings/__pycache__/utf_8.cpython-36.pyc
matches /home/rosuav/cpython/Lib/encodings/utf_8.py
# code object from
'/home/rosuav/cpython/Lib/encodings/__pycache__/utf_8.cpython-36.pyc'
import 'encodings.utf_8' #
<_frozen_importlib_external.SourceFileLoader object at 0x7f6d060f0710>
import '_signal' #
# /home/rosuav/cpython/Lib/encodings/__pycache__/latin_1.cpython-36.pyc
matches /home/rosuav/cpython/Lib/encodings/latin_1.py
# code object from
'/home/rosuav/cpython/Lib/encodings/__pycache__/latin_1.cpython-36.pyc'
import 'encodings.latin_1' #
<_frozen_importlib_external.SourceFileLoader object at 0x7f6d060f4278>
# /home/rosuav/cpython/Lib/__pycache__/io.cpython-36.pyc matches
/home/rosuav/cpython/Lib/io.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/io.cpython-36.pyc'
# /home/rosuav/cpython/Lib/__pycache__/abc.cpython-36.pyc matches
/home/rosuav/cpython/Lib/abc.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/abc.cpython-36.pyc'
# /home/rosuav/cpython/Lib/__pycache__/_weakrefset.cpython-36.pyc
matches /home/rosuav/cpython/Lib/_weakrefset.py
# code object from
'/home/rosuav/cpython/Lib/__pycache__/_weakrefset.cpython-36.pyc'
import '_weakrefset' # <_frozen_importlib_external.SourceFileLoader
object at 0x7f6d060f90f0>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at
0x7f6d060f47f0>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at
0x7f6d060f44a8>
# /home/rosuav/cpython/Lib/__pycache__/site.cpython-36.pyc matches
/home/rosuav/cpython/Lib/site.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/site.cpython-36.pyc'
# /home/rosuav/cpython/Lib/__pycache__/os.cpython-36.pyc matches
/home/rosuav/cpython/Lib/os.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/os.cpython-36.pyc'
import 'errno' #
# /home/rosuav/cpython/Lib/__pycache__/stat.cpython-36.pyc matches
/home/rosuav/cpython/Lib/stat.py
# code object from '/home/rosuav/cpython/Lib/__pycache__/stat.cpython-36.pyc'
import '_stat' #
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at
0x7f6d060a1940>
# /home/rosuav/cpython/Lib/__pycache__/posixpath.cpython-36.pyc
matches /home/rosuav/cpython/Lib/posixpath.py
# code object from
'/home/rosuav/cpython/Lib/__pycache__/posixpath.cpython-36.pyc'
# /home/rosuav/cpython/Lib/__pycache__/genericpath.cpython-36.pyc
matches /home/rosuav/cpython/Lib/genericpath.py
# code object from
'/home/rosuav/cpython/Lib/__pycache__/genericpath.cpython-36.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader
object at 0x7f6d060ab2e8>
import 'posixpath' # <_frozen_importlib_external.SourceFileLoader
object at 0x7f6d060a1fd0>
# /home/rosuav/cpython/Lib/__pycache__/_collections_abc.cpython-36.pyc
matches /home/rosuav/cpython/Lib/_collections_abc.py
# code object from
'/home/rosuav/cpython/Lib/__pycache__/_collections_abc.cpython-36.pyc'
import '_collections_abc' #
<_frozen_importlib_external.SourceFileLoader object at 0x7f6d060ab8d0>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at
0x7f6d0608cf98>
# /home/rosuav/cpython/Lib/__pycache__/_sitebuiltins.cpython-36.pyc
matches /home/rosuav/cpython/Lib/_sitebuiltins.py
# code object from
'/home/rosuav/cpython/Lib/__pycache__/_sitebuiltins.cpython-36.pyc'
import '_sitebu
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2016-08-12 - 2016-08-19) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open5594 ( -1) closed 33965 (+56) total 39559 (+55) Open issues with patches: 2438 Issues opened (34) == #26988: Add AutoNumberedEnum to stdlib http://bugs.python.org/issue26988 reopened by rhettinger #27611: test_tix cannot import _default_root after test_idle http://bugs.python.org/issue27611 reopened by serhiy.storchaka #27748: Simplify test_winsound http://bugs.python.org/issue27748 opened by zach.ware #27749: multprocessing errors on Windows: WriteFile() argument 1 must http://bugs.python.org/issue27749 opened by wevsty #27752: CSV DictReader default dialect name 'excel' is misleading, as http://bugs.python.org/issue27752 opened by lockywolf #27755: Retire DynOptionMenu with a ttk Combobox http://bugs.python.org/issue27755 opened by jfoo #27756: Add pyd icon for 3.6 http://bugs.python.org/issue27756 opened by Decorater #27759: selectors incorrectly retain invalid file descriptors http://bugs.python.org/issue27759 opened by Mark.Williams #27761: Private _nth_root function loses accuracy http://bugs.python.org/issue27761 opened by steven.daprano #27763: Add complex case to test_builtin abs() http://bugs.python.org/issue27763 opened by Evelyn Mitchell #27764: [Patch] Complete bits necessary for making fcntl's file lockin http://bugs.python.org/issue27764 opened by EdSchouten #27766: Add ChaCha20 Poly1305 to SSL ciphers http://bugs.python.org/issue27766 opened by christian.heimes #27768: ssl: add public API for IA-32 processor capabilities vector http://bugs.python.org/issue27768 opened by christian.heimes #27769: IDLE: Replace All up, no wrap replaces one up, all down http://bugs.python.org/issue27769 opened by Qwert225 #27772: Refer to actual format string when creating âzero paddingâ http://bugs.python.org/issue27772 opened by bignose #27776: PEP 524: Make os.urandom() blocking on Linux http://bugs.python.org/issue27776 opened by haypo #2: cgi.FieldStorage can't parse simple body with Content-Length a http://bugs.python.org/issue2 opened by rr- #27778: PEP 524: Add os.getrandom() http://bugs.python.org/issue27778 opened by haypo #27779: Sync-up docstrings in C version of the the decimal module http://bugs.python.org/issue27779 opened by rhettinger #27780: memory leaks in pgen build step abort build with address sanit http://bugs.python.org/issue27780 opened by geeknik #27781: Change sys.getfilesystemencoding() on Windows to UTF-8 http://bugs.python.org/issue27781 opened by steve.dower #27782: Multi-phase extension module initialization, inconsistent exce http://bugs.python.org/issue27782 opened by xiang.zhang #27784: Random failure of test_TCPServer() of test.test_socketserver.S http://bugs.python.org/issue27784 opened by haypo #27787: Avoid test_main() in test_httplib; gc.collect() dangling threa http://bugs.python.org/issue27787 opened by martin.panter #27788: platform module's version number doesn't match its docstring http://bugs.python.org/issue27788 opened by brett.cannon #27790: test_distutils spews linker messages on Windows http://bugs.python.org/issue27790 opened by terry.reedy #27791: test_threading: test_threads_join_2() failed with "Fatal Pytho http://bugs.python.org/issue27791 opened by haypo #27792: bool % int has inconsistent return type. http://bugs.python.org/issue27792 opened by mark.dickinson #27794: setattr a read-only property; the AttributeError should show t http://bugs.python.org/issue27794 opened by ztane #27796: Expose DirEntry constructor http://bugs.python.org/issue27796 opened by moloney #27797: ASCII file with UNIX line conventions and enough lines throws http://bugs.python.org/issue27797 opened by mjpieters #27799: Fix base64-codec and bz2-codec incremental decoders http://bugs.python.org/issue27799 opened by martin.panter #27800: Regular expressions with multiple repeat codes http://bugs.python.org/issue27800 opened by martin.panter #27802: Add __eq__ and __ne__ to collections.abc.Sequence. http://bugs.python.org/issue27802 opened by neil.g Most recent 15 issues with no replies (15) == #27799: Fix base64-codec and bz2-codec incremental decoders http://bugs.python.org/issue27799 #27796: Expose DirEntry constructor http://bugs.python.org/issue27796 #27772: Refer to actual format string when creating âzero paddingâ http://bugs.python.org/issue27772 #27764: [Patch] Complete bits necessary for making fcntl's file lockin http://bugs.python.org/issue27764 #27748: Simplify test_winsound http://bugs.python.org/issue27748 #27741: datetime.datetime.strptime functionality description incorrect http://bugs.python.org/issue27741 #27737: email.header.Header.encode() crashes with IndexError on spaces http
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Sat, Aug 20, 2016 at 1:26 AM, Steve Dower wrote:
> Check any .pth files you can find. I suspect mpl_toolkits has some magic in
> it to make the namespace package work on 2.7.
>>> sys.path
['/usr/local/lib/python36.zip', '/home/rosuav/cpython/Lib',
'/home/rosuav/cpython/Lib/plat-x86_64-linux-gnu',
'/home/rosuav/cpython/build/lib.linux-x86_64-3.6',
'/usr/local/lib/python3.6/site-packages']
/usr/local/lib/python36.zip doesn't exist. In
/usr/local/lib/python3.6/site-packages are a bunch of things that I've
installed using pip, but grepping for '/usr' in the -v output comes up
blank. There is, however, one .pth file there:
$ cat /usr/local/lib/python3.6/site-packages/matplotlib-1.5.1-py3.6-nspkg.pth
import sys, types, os;p =
os.path.join(sys._getframe(1).f_locals['sitedir'],
*('mpl_toolkits',));ie =
os.path.exists(os.path.join(p,'__init__.py'));m = not ie and
sys.modules.setdefault('mpl_toolkits',
types.ModuleType('mpl_toolkits'));mp = (m or []) and
m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
Is it possible that that's the cause? And if so, should "python -I"
remove /usr/local/lib/python3.6/site-packages from sys.path?
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On 19Aug2016 0910, Chris Angelico wrote:
On Sat, Aug 20, 2016 at 1:26 AM, Steve Dower wrote:
Check any .pth files you can find. I suspect mpl_toolkits has some magic in
it to make the namespace package work on 2.7.
$ cat /usr/local/lib/python3.6/site-packages/matplotlib-1.5.1-py3.6-nspkg.pth
import sys, types, os;p =
os.path.join(sys._getframe(1).f_locals['sitedir'],
*('mpl_toolkits',));ie =
os.path.exists(os.path.join(p,'__init__.py'));m = not ie and
sys.modules.setdefault('mpl_toolkits',
types.ModuleType('mpl_toolkits'));mp = (m or []) and
m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
Is it possible that that's the cause? And if so, should "python -I"
remove /usr/local/lib/python3.6/site-packages from sys.path?
That explains the extra modules - types.py imports collections and
collections.abc, which would seem to account for the rest.
I'm not sure whether that's causing the problem or not, but temporarily
renaming that pth file so it isn't picked up may help you track down the
test issue.
"python -S" is the only way to avoid .pth files, but that's a pretty
heavy hammer.
Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On 8/19/2016 10:13 AM, Chris Angelico wrote:
On my main dev system (Debian Stretch), I've had a single
long-standing test failure - test_site.py,
StartupImportTests.test_startup_imports. It's annoying (partly because
it's such a noisy failure), and doesn't appear to be happening on the
buildbots, nor presumably on core devs' computers, so it's probably
not a Python bug. How can I go about figuring out what's actually
going on here?
The test effectively runs this (but with -v, which makes it noisy):
$ ./python -I -c 'import sys; print(set(sys.modules))'
{'_frozen_importlib_external', 'keyword', '_stat', 'weakref',
'sysconfig', 'genericpath', '_sysconfigdata_m', '_io', '_operator',
'_bootlocale', 'encodings.utf_8', 'itertools', '_locale', 'codecs',
'_weakrefset', 'operator', '_frozen_importlib', 'types',
'_sitebuiltins', 'os', 'posixpath', '_functools', '__main__',
'_weakref', '_warnings', '_codecs', 'builtins', 'posix', 'functools',
'io', 'encodings.aliases', 'collections.abc', 'encodings', '_thread',
'collections', 'errno', '_collections', '_collections_abc', '_heapq',
'_imp', 'site', 'sys', 'reprlib', '_signal', 'stat', 'zipimport',
'abc', 'encodings.latin_1', 'os.path', 'marshal', 'heapq',
'mpl_toolkits'}
Sorted() gives an easier to compare listing ;-).
On Windows as of 2 days ago (using required " insteadof ') many fewer:
F:\Python\dev>36\pcbuild\win32\python_d.exe -I -c "import sys;
print(sorted(sys.modules))"
['__main__', '_bootlocale', '_codecs', '_collections_abc',
'_frozen_importlib', '_frozen_importlib_external', '_imp', '_io',
'_locale', '_signal', '_sitebuiltins', '_stat', '_thread', '_warnings',
'_weakref', '_weakrefset', 'abc', 'builtins', 'codecs', 'encodings',
'encodings.aliases', 'encodings.cp1252', 'encodings.cp437',
'encodings.latin_1', 'encodings.mbcs', 'encodings.utf_8', 'errno',
'genericpath', 'io', 'marshal', 'nt', 'ntpath', 'os', 'os.path', 'site',
'stat', 'sys', 'sysconfig', 'winreg', 'zipimport']
test_site passes, so _collections_abc is not a problem
--
Terry Jan Reedy
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On 8/19/2016 12:10 PM, Chris Angelico wrote:
On Sat, Aug 20, 2016 at 1:26 AM, Steve Dower wrote:
Check any .pth files you can find. I suspect mpl_toolkits has some magic in
it to make the namespace package work on 2.7.
sys.path
['/usr/local/lib/python36.zip', '/home/rosuav/cpython/Lib',
'/home/rosuav/cpython/Lib/plat-x86_64-linux-gnu',
'/home/rosuav/cpython/build/lib.linux-x86_64-3.6',
'/usr/local/lib/python3.6/site-packages']
/usr/local/lib/python36.zip doesn't exist. In
/usr/local/lib/python3.6/site-packages are a bunch of things that I've
installed using pip, but grepping for '/usr' in the -v output comes up
blank. There is, however, one .pth file there:
$ cat /usr/local/lib/python3.6/site-packages/matplotlib-1.5.1-py3.6-nspkg.pth
import sys, types, os;p =
os.path.join(sys._getframe(1).f_locals['sitedir'],
*('mpl_toolkits',));ie =
os.path.exists(os.path.join(p,'__init__.py'));m = not ie and
sys.modules.setdefault('mpl_toolkits',
types.ModuleType('mpl_toolkits'));mp = (m or []) and
m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
Is it possible that that's the cause?
Change 'pth' to 'pthx' and try it.
And if so, should "python -I"
remove /usr/local/lib/python3.6/site-packages from sys.path?
--
Terry Jan Reedy
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Sat, Aug 20, 2016 at 2:25 AM, Steve Dower wrote:
> On 19Aug2016 0910, Chris Angelico wrote:
>>
>> On Sat, Aug 20, 2016 at 1:26 AM, Steve Dower
>> wrote:
>>>
>>> Check any .pth files you can find. I suspect mpl_toolkits has some magic
>>> in
>>> it to make the namespace package work on 2.7.
>>
>>
>> $ cat
>> /usr/local/lib/python3.6/site-packages/matplotlib-1.5.1-py3.6-nspkg.pth
>> import sys, types, os;p =
>> os.path.join(sys._getframe(1).f_locals['sitedir'],
>> *('mpl_toolkits',));ie =
>> os.path.exists(os.path.join(p,'__init__.py'));m = not ie and
>> sys.modules.setdefault('mpl_toolkits',
>> types.ModuleType('mpl_toolkits'));mp = (m or []) and
>> m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
>>
>> Is it possible that that's the cause? And if so, should "python -I"
>> remove /usr/local/lib/python3.6/site-packages from sys.path?
>
>
> That explains the extra modules - types.py imports collections and
> collections.abc, which would seem to account for the rest.
>
> I'm not sure whether that's causing the problem or not, but temporarily
> renaming that pth file so it isn't picked up may help you track down the
> test issue.
Sure enough, that did fix the problem - that is to say, renaming it
did allow test_site to pass. Thanks!
> "python -S" is the only way to avoid .pth files, but that's a pretty heavy
> hammer.
Hmmm. So the question is, what is this test testing? For
reference, here's the code:
class StartupImportTests(unittest.TestCase):
def test_startup_imports(self):
# This tests checks which modules are loaded by Python when it
# initially starts upon startup.
popen = subprocess.Popen([sys.executable, '-I', '-v', '-c',
'import sys; print(set(sys.modules))'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdout, stderr = popen.communicate()
stdout = stdout.decode('utf-8')
stderr = stderr.decode('utf-8')
modules = eval(stdout)
self.assertIn('site', modules)
# http://bugs.python.org/issue19205
re_mods = {'re', '_sre', 'sre_compile', 'sre_constants', 'sre_parse'}
# _osx_support uses the re module in many placs
if sys.platform != 'darwin':
self.assertFalse(modules.intersection(re_mods), stderr)
# http://bugs.python.org/issue9548
self.assertNotIn('locale', modules, stderr)
if sys.platform != 'darwin':
# http://bugs.python.org/issue19209
self.assertNotIn('copyreg', modules, stderr)
# http://bugs.python.org/issue19218>
collection_mods = {'_collections', 'collections', 'functools',
'heapq', 'itertools', 'keyword', 'operator',
'reprlib', 'types', 'weakref'
}.difference(sys.builtin_module_names)
self.assertFalse(modules.intersection(collection_mods), stderr)
It asserts that 'site' is in the list of loaded modules, so simply
adding '-S' to the subpython parameters doesn't work. But this test,
as written, depends on the behaviour of arbitrary third-party modules
installed with pip, which seems wrong for a standard library test.
AIUI this is trying to catch changes that accidentally cause large
modules to be loaded on startup, correct? Which is all well and good,
but Python shouldn't have test failures due to matplotlib being
installed.
I see two (or three) ways to deal with this.
1) Declare it a bug in matplotlib, and demand that .pth files not
import anything that wouldn't otherwise be imported.
2) Change the test to add '-S' and remove the assertion about 'site'
being present; this means that site.py itself is no longer covered by
the test.
3) Accept the status quo: this test can fail if .pth files are present.
Incidentally, where's it documented that .pth files can be executed?
It's not in the site.py docs [1] or docstring, only in the addpackage
function within that (which I found by browsing the source). Maybe the
solution is to disable that one specific feature when running this
test?
ChrisA
[1] https://docs.python.org/3.6/library/site.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Aug 19, 2016 6:46 PM, Chris Angelico wrote:
>
> On Sat, Aug 20, 2016 at 2:25 AM, Steve Dower wrote:
> > On 19Aug2016 0910, Chris Angelico wrote:
> >>
> >> On Sat, Aug 20, 2016 at 1:26 AM, Steve Dower
> >> wrote:
> >>>
> >>> Check any .pth files you can find. I suspect mpl_toolkits has some magic
> >>> in
> >>> it to make the namespace package work on 2.7.
> >>
> >>
> >> $ cat
> >> /usr/local/lib/python3.6/site-packages/matplotlib-1.5.1-py3.6-nspkg.pth
> >> import sys, types, os;p =
> >> os.path.join(sys._getframe(1).f_locals['sitedir'],
> >> *('mpl_toolkits',));ie =
> >> os.path.exists(os.path.join(p,'__init__.py'));m = not ie and
> >> sys.modules.setdefault('mpl_toolkits',
> >> types.ModuleType('mpl_toolkits'));mp = (m or []) and
> >> m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
> >>
> >> Is it possible that that's the cause? And if so, should "python -I"
> >> remove /usr/local/lib/python3.6/site-packages from sys.path?
> >
> >
> > That explains the extra modules - types.py imports collections and
> > collections.abc, which would seem to account for the rest.
> >
> > I'm not sure whether that's causing the problem or not, but temporarily
> > renaming that pth file so it isn't picked up may help you track down the
> > test issue.
>
> Sure enough, that did fix the problem - that is to say, renaming it
> did allow test_site to pass. Thanks!
>
> > "python -S" is the only way to avoid .pth files, but that's a pretty heavy
> > hammer.
>
> Hmmm. So the question is, what is this test testing? For
> reference, here's the code:
>
> class StartupImportTests(unittest.TestCase):
>
> def test_startup_imports(self):
> # This tests checks which modules are loaded by Python when it
> # initially starts upon startup.
> popen = subprocess.Popen([sys.executable, '-I', '-v', '-c',
> 'import sys; print(set(sys.modules))'],
> stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
> stdout, stderr = popen.communicate()
> stdout = stdout.decode('utf-8')
> stderr = stderr.decode('utf-8')
> modules = eval(stdout)
>
> self.assertIn('site', modules)
>
> # http://bugs.python.org/issue19205
> re_mods = {'re', '_sre', 'sre_compile', 'sre_constants', 'sre_parse'}
> # _osx_support uses the re module in many placs
> if sys.platform != 'darwin':
> self.assertFalse(modules.intersection(re_mods), stderr)
> # http://bugs.python.org/issue9548
> self.assertNotIn('locale', modules, stderr)
> if sys.platform != 'darwin':
> # http://bugs.python.org/issue19209
> self.assertNotIn('copyreg', modules, stderr)
> # http://bugs.python.org/issue19218>
> collection_mods = {'_collections', 'collections', 'functools',
> 'heapq', 'itertools', 'keyword', 'operator',
> 'reprlib', 'types', 'weakref'
> }.difference(sys.builtin_module_names)
> self.assertFalse(modules.intersection(collection_mods), stderr)
>
>
> It asserts that 'site' is in the list of loaded modules, so simply
> adding '-S' to the subpython parameters doesn't work. But this test,
> as written, depends on the behaviour of arbitrary third-party modules
> installed with pip, which seems wrong for a standard library test.
> AIUI this is trying to catch changes that accidentally cause large
> modules to be loaded on startup, correct? Which is all well and good,
> but Python shouldn't have test failures due to matplotlib being
> installed.
>
> I see two (or three) ways to deal with this.
>
> 1) Declare it a bug in matplotlib, and demand that .pth files not
> import anything that wouldn't otherwise be imported.
>
> 2) Change the test to add '-S' and remove the assertion about 'site'
> being present; this means that site.py itself is no longer covered by
> the test.
>
> 3) Accept the status quo: this test can fail if .pth files are present.
>
4) split the test in two: one running with -S to check no collections, (and for *not* site.py). The other without -S, and check for site.py.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] File system path encoding on Windows
Hi python-dev
About a week ago I proposed on python-ideas making some changes to how
Python deals with encodings on Windows, specifically in relation to how
Python interacts with the operating system.
Changes to the console were uncontroversial, and I have posted patches
at http://bugs.python.org/issue1602 and
http://bugs.python.org/issue17620 to enable the full range of Unicode
input to be used at interactive stdin/stdout.
However, changes to sys.getfilesystemencoding(), which determines how
the os module (and most filesystem functions in general) interpret bytes
parameters, were more heatedly discussed. I've summarised the discussion
in this email
I'll declare up front that my preferred change is to treat bytes as
utf-8 in Python 3.6, and I've posted a patch to do that at
http://bugs.python.org/issue27781. Hopefully I haven't been too biased
in my presentation of the alternatives, but this is so you at least know
which way I'm biased.
I'm looking for some agreement on the answers to the questions I pose in
the summary.
There is much more detail about them presented after that, as there are
a number of non-obvious issues at play here. I suspect this will
eventually become a PEP, but it's presented here as a summary of a
discussion and not a PEP.
Cheers,
Steve
Summary
===
Representing file system paths on Windows as bytes may result in data
loss due to the way Windows encodes/decodes strings via its bytes API.
We can mitigate this by only using Window's Unicode API and doing our
own encoding and decoding (i.e. within posixmodule.c's path converter).
Invalid characters could cause encoding exceptions rather than data loss.
We can go further to fix this by declaring the encoding of bytes paths
on Windows must be utf-8, which would also prevent encoding exceptions,
as utf-8 can fully represent all paths on Windows (natively utf-16-le).
Even though using bytes for paths on Windows has been deprecated for
three releases, this is not widely known and it may be too soon to
change the behaviour.
Questions:
* should we always use Window's Unicode APIs instead of switching
between bytes/Unicode based on parameter type?
* should we allow users to pass bytes and interpret them as utf-8 rather
than letting Windows do the decoding?
* should we do it in 3.6, 3.7 or 3.8?
Background
==
File system paths are almost universally represented as text in some
encoding determined by the file system. In Python, we expose these paths
via a number of interfaces, such as the os and io modules. Paths may be
passed either direction across these interfaces, that is, from the
filesystem to the application (for example, os.listdir()), or from the
application to the filesystem (for example, os.unlink()).
When paths are passed between the filesystem and the application, they
are either passed through as a bytes blob or converted to/from str using
sys.getfilesystemencoding(). The result of encoding a string with
sys.getfilesystemencoding() is a blob of bytes in the native format for
the default file system.
On Windows, the native format for the filesystem is utf-16-le. The
recommended platform APIs for accessing the filesystem all accept and
return text encoded in this format. However, prior to Windows NT (and
possibly further back), the native format was a configurable machine
option and a separate set of APIs existed to accept this format. The
option (the "active code page") and these APIs (the "*A functions")
still exist in recent versions of Windows for backwards compatibility,
though new functionality often only has a utf-16-le API (the "*W
functions").
In Python, we recommend using str as the default format because (with
the surrogateescape handling on POSIX), it can correctly round-trip all
characters used in paths. On Windows this is strongly recommended
because the legacy OS support for bytes cannot round-trip all characters
used in paths. Our support for bytes explicitly uses the *A functions
and hence the encoding for the bytes is "whatever the active code page
is". Since the active code page cannot represent all Unicode characters,
the conversion of a path into bytes can lose information without warning
(and we can't get a warning from the OS here - more on this later).
As a demonstration of this:
>>> open('test\uAB00.txt', 'wb').close()
>>> import glob
>>> glob.glob('test*')
['test\uab00.txt']
>>> glob.glob(b'test*')
[b'test?.txt']
The Unicode character in the second call to glob has been replaced by a
'?', which means passing the path back into the filesystem will result
in a FileNotFoundError (though ironically, passing it back into glob()
will find the file again, since '?' is a single-character wildcard). You
can observe the same results in os.listdir() or any function that
matches the return type to the parameter type.
Why is this a problem?
==
While the obvious and correct answer is to just use str
Re: [Python-Dev] File system path encoding on Windows
#1 sounds like a great idea. I suppose surrogatepass solves approximately the same problem of Rust's WTF-8, which is a way to round-trip bad UCS-2? https://simonsapin.github.io/wtf-8/ #2 sounds like it would leave several problems, since mbcs is not the same as a normal text encoding, IIUC it depends on the active code page. So if your active code page is Russian you might not be able to encode Japanese characters into MBCS. Solution #2a Modify Windows so utf-8 is a valid value for the current MBCS code page. On Fri, Aug 19, 2016 at 3:01 PM Steve Dower wrote: > Hi python-dev > > About a week ago I proposed on python-ideas making some changes to how > Python deals with encodings on Windows, specifically in relation to how > Python interacts with the operating system. > > Changes to the console were uncontroversial, and I have posted patches > at http://bugs.python.org/issue1602 and > http://bugs.python.org/issue17620 to enable the full range of Unicode > input to be used at interactive stdin/stdout. > > However, changes to sys.getfilesystemencoding(), which determines how > the os module (and most filesystem functions in general) interpret bytes > parameters, were more heatedly discussed. I've summarised the discussion > in this email > > I'll declare up front that my preferred change is to treat bytes as > utf-8 in Python 3.6, and I've posted a patch to do that at > http://bugs.python.org/issue27781. Hopefully I haven't been too biased > in my presentation of the alternatives, but this is so you at least know > which way I'm biased. > > I'm looking for some agreement on the answers to the questions I pose in > the summary. > > There is much more detail about them presented after that, as there are > a number of non-obvious issues at play here. I suspect this will > eventually become a PEP, but it's presented here as a summary of a > discussion and not a PEP. > > Cheers, > Steve > > Summary > === > > Representing file system paths on Windows as bytes may result in data > loss due to the way Windows encodes/decodes strings via its bytes API. > > We can mitigate this by only using Window's Unicode API and doing our > own encoding and decoding (i.e. within posixmodule.c's path converter). > Invalid characters could cause encoding exceptions rather than data loss. > > We can go further to fix this by declaring the encoding of bytes paths > on Windows must be utf-8, which would also prevent encoding exceptions, > as utf-8 can fully represent all paths on Windows (natively utf-16-le). > > Even though using bytes for paths on Windows has been deprecated for > three releases, this is not widely known and it may be too soon to > change the behaviour. > > Questions: > * should we always use Window's Unicode APIs instead of switching > between bytes/Unicode based on parameter type? > * should we allow users to pass bytes and interpret them as utf-8 rather > than letting Windows do the decoding? > * should we do it in 3.6, 3.7 or 3.8? > > Background > == > > File system paths are almost universally represented as text in some > encoding determined by the file system. In Python, we expose these paths > via a number of interfaces, such as the os and io modules. Paths may be > passed either direction across these interfaces, that is, from the > filesystem to the application (for example, os.listdir()), or from the > application to the filesystem (for example, os.unlink()). > > When paths are passed between the filesystem and the application, they > are either passed through as a bytes blob or converted to/from str using > sys.getfilesystemencoding(). The result of encoding a string with > sys.getfilesystemencoding() is a blob of bytes in the native format for > the default file system. > > On Windows, the native format for the filesystem is utf-16-le. The > recommended platform APIs for accessing the filesystem all accept and > return text encoded in this format. However, prior to Windows NT (and > possibly further back), the native format was a configurable machine > option and a separate set of APIs existed to accept this format. The > option (the "active code page") and these APIs (the "*A functions") > still exist in recent versions of Windows for backwards compatibility, > though new functionality often only has a utf-16-le API (the "*W > functions"). > > In Python, we recommend using str as the default format because (with > the surrogateescape handling on POSIX), it can correctly round-trip all > characters used in paths. On Windows this is strongly recommended > because the legacy OS support for bytes cannot round-trip all characters > used in paths. Our support for bytes explicitly uses the *A functions > and hence the encoding for the bytes is "whatever the active code page > is". Since the active code page cannot represent all Unicode characters, > the conversion of a path into bytes can lose information without warning > (and we can't get a warning from the OS here - more on this later). > > A
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Fri, 19 Aug 2016 at 09:47 Chris Angelico wrote:
> On Sat, Aug 20, 2016 at 2:25 AM, Steve Dower
> wrote:
> > On 19Aug2016 0910, Chris Angelico wrote:
> >>
> >> On Sat, Aug 20, 2016 at 1:26 AM, Steve Dower
> >> wrote:
> >>>
> >>> Check any .pth files you can find. I suspect mpl_toolkits has some
> magic
> >>> in
> >>> it to make the namespace package work on 2.7.
> >>
> >>
> >> $ cat
> >> /usr/local/lib/python3.6/site-packages/matplotlib-1.5.1-py3.6-nspkg.pth
> >> import sys, types, os;p =
> >> os.path.join(sys._getframe(1).f_locals['sitedir'],
> >> *('mpl_toolkits',));ie =
> >> os.path.exists(os.path.join(p,'__init__.py'));m = not ie and
> >> sys.modules.setdefault('mpl_toolkits',
> >> types.ModuleType('mpl_toolkits'));mp = (m or []) and
> >> m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
> >>
> >> Is it possible that that's the cause? And if so, should "python -I"
> >> remove /usr/local/lib/python3.6/site-packages from sys.path?
> >
> >
> > That explains the extra modules - types.py imports collections and
> > collections.abc, which would seem to account for the rest.
> >
> > I'm not sure whether that's causing the problem or not, but temporarily
> > renaming that pth file so it isn't picked up may help you track down the
> > test issue.
>
> Sure enough, that did fix the problem - that is to say, renaming it
> did allow test_site to pass. Thanks!
>
> > "python -S" is the only way to avoid .pth files, but that's a pretty
> heavy
> > hammer.
>
> Hmmm. So the question is, what is this test testing?
It's making sure people who work on the modules that are imported during
startup don't accidentally add another module dependency to the startup
sequence. Since module imports impact startup time we have historically
worked hard to minimize import dependencies for startup (hence things like
certain functions in the os module doing imports within a function even
though we discourage that practice in general).
> For
> reference, here's the code:
>
> class StartupImportTests(unittest.TestCase):
>
> def test_startup_imports(self):
> # This tests checks which modules are loaded by Python when it
> # initially starts upon startup.
> popen = subprocess.Popen([sys.executable, '-I', '-v', '-c',
> 'import sys; print(set(sys.modules))'],
> stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
> stdout, stderr = popen.communicate()
> stdout = stdout.decode('utf-8')
> stderr = stderr.decode('utf-8')
> modules = eval(stdout)
>
> self.assertIn('site', modules)
>
> # http://bugs.python.org/issue19205
> re_mods = {'re', '_sre', 'sre_compile', 'sre_constants',
> 'sre_parse'}
> # _osx_support uses the re module in many placs
> if sys.platform != 'darwin':
> self.assertFalse(modules.intersection(re_mods), stderr)
> # http://bugs.python.org/issue9548
> self.assertNotIn('locale', modules, stderr)
> if sys.platform != 'darwin':
> # http://bugs.python.org/issue19209
> self.assertNotIn('copyreg', modules, stderr)
> # http://bugs.python.org/issue19218>
> collection_mods = {'_collections', 'collections', 'functools',
>'heapq', 'itertools', 'keyword', 'operator',
>'reprlib', 'types', 'weakref'
> }.difference(sys.builtin_module_names)
> self.assertFalse(modules.intersection(collection_mods), stderr)
>
>
> It asserts that 'site' is in the list of loaded modules, so simply
> adding '-S' to the subpython parameters doesn't work. But this test,
> as written, depends on the behaviour of arbitrary third-party modules
> installed with pip, which seems wrong for a standard library test.
> AIUI this is trying to catch changes that accidentally cause large
> modules to be loaded on startup, correct? Which is all well and good,
> but Python shouldn't have test failures due to matplotlib being
> installed.
>
> I see two (or three) ways to deal with this.
>
> 1) Declare it a bug in matplotlib, and demand that .pth files not
> import anything that wouldn't otherwise be imported.
>
> 2) Change the test to add '-S' and remove the assertion about 'site'
> being present; this means that site.py itself is no longer covered by
> the test.
>
> 3) Accept the status quo: this test can fail if .pth files are present.
>
> Incidentally, where's it documented that .pth files can be executed?
It's not in the site.py docs [1] or docstring, only in the addpackage
> function within that (which I found by browsing the source).
Actually it is documented in the site documentation: "Lines starting with
import (followed by space or tab) are executed". It's subtle, but
"executed" literally means execution in this context; the line is passed
into exec():
https://github.com/python/cpython
Re: [Python-Dev] File system path encoding on Windows
On 19Aug2016 1225, Daniel Holth wrote: #1 sounds like a great idea. I suppose surrogatepass solves approximately the same problem of Rust's WTF-8, which is a way to round-trip bad UCS-2? https://simonsapin.github.io/wtf-8/ Yep. #2 sounds like it would leave several problems, since mbcs is not the same as a normal text encoding, IIUC it depends on the active code page. So if your active code page is Russian you might not be able to encode Japanese characters into MBCS. That's correct. In 99% (or more) of cases, mbcs is going to be the same as what we currently have. The difference is that when we encode/decode in CPython we can use a different handler than 'replace' and at least prevent the _silent_ data loss. Solution #2a Modify Windows so utf-8 is a valid value for the current MBCS code page. Presumably a joke, but won't happen because too many applications assume that the active code page is one byte per character, which it isn't, but it's close enough that most of the time you never notice. (Incidentally, this is also the problem with utf-16, since many applications also assume that it's always one wchar_t per character and get away with it. At least with utf-8 you encounter multi-byte sequences often enough that you basically are forced to deal with them.) Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Sat, Aug 20, 2016 at 5:33 AM, Brett Cannon wrote: >> Hmmm. So the question is, what is this test testing? > > > It's making sure people who work on the modules that are imported during > startup don't accidentally add another module dependency to the startup > sequence. Since module imports impact startup time we have historically > worked hard to minimize import dependencies for startup (hence things like > certain functions in the os module doing imports within a function even > though we discourage that practice in general). Fair enough. Which means it probably *should* import site, because that's going to happen on all normal usage. So that still leaves a few options: 1) Demand that .pth files restrict themselves to what's already imported. This means startup is still fast even if you have a bunch of pths. Downside: Third-party code can break Python's rules. Upside: When they do, it can be reported as a bug, the fixing of which will improve startup times when that's installed. 2) Change the test to somehow disable .pth execution while keeping the rest of site.py intact. This gives more consistent test results, but still is mostly applicable to normal usage. 3) Ignore the problem and assume that the Python test suite will be run with no site-packages installed. Status quo, basically. Possibly with some tweaks so running 'make test' ignores site-packages? Is #2 doable? If not, can 'make test' somehow hack out site-packages? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Fri, 19 Aug 2016 at 14:11 Chris Angelico wrote: > On Sat, Aug 20, 2016 at 5:33 AM, Brett Cannon wrote: > >> Hmmm. So the question is, what is this test testing? > > > > > > It's making sure people who work on the modules that are imported during > > startup don't accidentally add another module dependency to the startup > > sequence. Since module imports impact startup time we have historically > > worked hard to minimize import dependencies for startup (hence things > like > > certain functions in the os module doing imports within a function even > > though we discourage that practice in general). > > Fair enough. Which means it probably *should* import site, because > that's going to happen on all normal usage. So that still leaves a few > options: > > 1) Demand that .pth files restrict themselves to what's already > imported. This means startup is still fast even if you have a bunch of > pths. Downside: Third-party code can break Python's rules. Upside: > When they do, it can be reported as a bug, the fixing of which will > improve startup times when that's installed. > > 2) Change the test to somehow disable .pth execution while keeping the > rest of site.py intact. This gives more consistent test results, but > still is mostly applicable to normal usage. > > 3) Ignore the problem and assume that the Python test suite will be > run with no site-packages installed. Status quo, basically. Possibly > with some tweaks so running 'make test' ignores site-packages? > > Is #2 doable? Anything is doable given enough time and effort. ;) As of right now the only way is with -S. > If not, can 'make test' somehow hack out site-packages? > The problem is you're dealing with startup which means it won't come into effect until it's too late. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On 19Aug2016 1410, Chris Angelico wrote: 2) Change the test to somehow disable .pth execution while keeping the rest of site.py intact. This gives more consistent test results, but still is mostly applicable to normal usage. If you start the process with "-S", then run: import site site.addpackage = lambda *a: None site.main() # rest of the test Then that should do it. This will cut out all .pth processing, but shouldn't affect any of the other steps. Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Sat, Aug 20, 2016 at 7:34 AM, Brett Cannon wrote: >> 1) Demand that .pth files restrict themselves to what's already >> imported. This means startup is still fast even if you have a bunch of >> pths. Downside: Third-party code can break Python's rules. Upside: >> When they do, it can be reported as a bug, the fixing of which will >> improve startup times when that's installed. >> >> 2) Change the test to somehow disable .pth execution while keeping the >> rest of site.py intact. This gives more consistent test results, but >> still is mostly applicable to normal usage. >> >> 3) Ignore the problem and assume that the Python test suite will be >> run with no site-packages installed. Status quo, basically. Possibly >> with some tweaks so running 'make test' ignores site-packages? >> >> Is #2 doable? > > > Anything is doable given enough time and effort. ;) As of right now the only > way is with -S. Heh. I guess what I really mean is: Is #2 a good idea, or would it be considered horrendously hacky? This is my itch, so I'm not afraid to be the one to code a solution, but I don't want to make things overall worse to fix one specific use case. >> If not, can 'make test' somehow hack out site-packages? > > > The problem is you're dealing with startup which means it won't come into > effect until it's too late. Yeah, but the 'make test' operation could set an environment variable or something to say "there are no site-packages, kthxbye". Again, there's currently no such facility, and probably that one is a bad idea, as it'd have implications (possibly security implications) for other runnings of Python. Steve's suggestion would keep all the visible hackery in test_site.py, and it's well known that tests often need to do dark and dangerous things. On Sat, Aug 20, 2016 at 7:51 AM, Steve Dower wrote: > > If you start the process with "-S", then run: > > import site > site.addpackage = lambda *a: None > site.main() > # rest of the test > > Then that should do it. This will cut out all .pth processing, but shouldn't > affect any of the other steps. If I make a patch to do that, would it have a good chance of being accepted? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Aug 19, 2016 11:10 PM, Chris Angelico wrote: > > On Sat, Aug 20, 2016 at 5:33 AM, Brett Cannon wrote: > >> Hmmm. So the question is, what is this test testing? > > > > > > It's making sure people who work on the modules that are imported during > > startup don't accidentally add another module dependency to the startup > > sequence. Since module imports impact startup time we have historically > > worked hard to minimize import dependencies for startup (hence things like > > certain functions in the os module doing imports within a function even > > though we discourage that practice in general). > > Fair enough. Which means it probably *should* import site, because > that's going to happen on all normal usage. So that still leaves a few > options: > > 1) Demand that .pth files restrict themselves to what's already > imported. This means startup is still fast even if you have a bunch of > pths. Downside: Third-party code can break Python's rules. Upside: > When they do, it can be reported as a bug, the fixing of which will > improve startup times when that's installed. > > 2) Change the test to somehow disable .pth execution while keeping the > rest of site.py intact. This gives more consistent test results, but > still is mostly applicable to normal usage. > > 3) Ignore the problem and assume that the Python test suite will be > run with no site-packages installed. Status quo, basically. Possibly > with some tweaks so running 'make test' ignores site-packages? > > Is #2 doable? If not, can 'make test' somehow hack out site-packages? > > ChrisA I'd like to re-iterate my suggestion in case it was missed: split the current test in 2 tests: * Running with `-S` which is for checking that by default the collections are not imported. (Which is what is currently tested) * Running without `-S` which is for checking that `site` gets imported. I hope that is clear enough. Do you see any problems with such an approach? Kind regards, Sjoerd Job ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Sat, Aug 20, 2016 at 7:55 AM, Chris Angelico wrote: > If I make a patch to do that, would it have a good chance of being accepted? > Shortcut. I've made the patch and put it on the tracker. http://bugs.python.org/issue27807 Either it's accepted or it's not :) Thank you all for the debugging help. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Sat, Aug 20, 2016 at 8:07 AM, Sjoerd Job Postmus wrote: > I'd like to re-iterate my suggestion in case it was missed: split the > current test in 2 tests: > > * Running with `-S` which is for checking that by default the collections > are not imported. (Which is what is currently tested) > * Running without `-S` which is for checking that `site` gets imported. > > I hope that is clear enough. Do you see any problems with such an approach? AIUI this wouldn't test that site is sufficiently slim. The test is supposed to mimic normal startup, then assert that unnecessary modules haven't been loaded. Splitting it would prove that a -S run is nice and slim, but would allow bloat in site.py to pass unnoticed. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Fri, Aug 19, 2016, at 18:10, Chris Angelico wrote: > On Sat, Aug 20, 2016 at 8:07 AM, Sjoerd Job Postmus > wrote: > > I'd like to re-iterate my suggestion in case it was missed: split the > > current test in 2 tests: > > > > * Running with `-S` which is for checking that by default the collections > > are not imported. (Which is what is currently tested) > > * Running without `-S` which is for checking that `site` gets imported. > > > > I hope that is clear enough. Do you see any problems with such an approach? > > AIUI this wouldn't test that site is sufficiently slim. The test is > supposed to mimic normal startup, then assert that unnecessary modules > haven't been loaded. Splitting it would prove that a -S run is nice > and slim, but would allow bloat in site.py to pass unnoticed. What about -S and putting "import site" explicitly in the test code? Or would that go back to importing everything on people who have packages installed? Really, there should be a "source-isolated" mode, which removes /usr/lib/whatever from the path in the same way that -I removes the user's stuff, but still runs cpython/Lib/site.py , and all tests (and for that matter stuff like frozen module prep) should be run in this mode. I'm not sure I see a downside to having such a mode be the *default* mode for "python detects it is being run from a source tree". ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Sat, Aug 20, 2016 at 12:55 PM, Random832 wrote: > > What about -S and putting "import site" explicitly in the test code? Or > would that go back to importing everything on people who have packages > installed? I think so, yes. > Really, there should be a "source-isolated" mode, which removes > /usr/lib/whatever from the path in the same way that -I removes the > user's stuff, but still runs cpython/Lib/site.py , and all tests (and > for that matter stuff like frozen module prep) should be run in this > mode. > > I'm not sure I see a downside to having such a mode be the *default* > mode for "python detects it is being run from a source tree". That sounds great, but I don't have the skills to implement that and ensure that it's reliable. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
