Re: [Python-Dev] FAT Python (lack of) performance
On Mon, Jan 25, 2016 at 11:58:12PM +0100, Victor Stinner wrote:
> ...
> Oh, they are a lot of things to do! ...
Just wondering, do you also need a set of (abusive) test-cases which
check 100% conformity to the CPython semantics? I'm sure many of us
would be able to whip up some ideas of things that are possible with
Python and are of the kind "but you should not do that! That's bad
programming!" which may or may not break the optimizations (especially
specialized functions).
I'm thinking of things like
def override_length_function_test():
global len
value = len("abc")
len = lambda obj: ord(obj[0]))
value += len("abc")
assert value == 3 + 97, "Outdated `len` used."
And also cases where `len` was changed not in the function itself, but
in another function that was called indirectly (maybe 4 functions down,
one of which was monkey-patched in after the compilation step):
module_a.py
def test_func(callback):
value = len("abc")
callback()
value += len("abc")
assert value == 3 + 97, "Outdated `len` used."
module_b.py
import module_a
def override_length_function_test():
def callback():
module_a.len = lambda obj: ord(obj[0])
assert module_a.test_func(callback) == 3 + 97, "Outdated `len` used."
(I'm sure some of the other readers of this list can be even more
creative in trying to show that making optimizations like this can break
semantics.)
Other fun tricks I'd like to try is overriding the `len` method from a
signal handler, what happens when you monkey-patch a dependent method,
having `__getitem__` and `__getattr__` on some value override `len`.
Basically: trying things that I normally should not try during my
working hours, on account of wanting to still have a job the next day.
Kind regards,
Sjoerd Job
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
I would like to make just 1 comment regarding the question of accepting (or not) bytes as output of `os.fspath`. The whole point of adding `os.fspath` is to make it easier to use Path objects. This is in an effort to gain greater adoption of pathlib in libraries. Now, this is an excellent idea. However, if it were to reject bytes, that would mean that when libraries start to use pathlib, it would suddenly become harder for people that actually need bytes-support to use pathlib. Now, the claim 'if you need bytes, you should not be using pathlib` is a reasonable one. But what if I need bytes *and* a specific library (say, image handling, or a web framework, or ...). It's not up to me if that library uses pathlib or plain old os.path.join. Is using surrogate-escapes enough for this case? I myself am not sure, (and also not affected), but it sounds to me that rejecting bytes is a wrong approach if there is no proper workaround (assuming the use-case of pathlib is somewhere deep in library code). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
> On 12 May 2016, at 21:30, Ethan Furman wrote: > > If you need bytes support for your paths, there's at least one [1] that has > that support. So if I would need bytes support, I should submit a pull request to which replaces usage of the stdlib pathlib with another variant, upon which they will decline the pull request because it introduces another "useless" dependency. Good to know. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
> On 13 May 2016, at 08:07, Ethan Furman wrote: > > On 05/12/2016 01:59 PM, Sjoerd Job Postmus wrote: >>> On 12 May 2016, at 21:30, Ethan Furman wrote: >>> >>> If you need bytes support for your paths, there's at least one [1] that has >>> that support. >> >> So if I would need bytes support, I should submit a pull request to >> > library> which replaces usage of the stdlib pathlib with another variant, > > upon which they > > will decline the pull request because it introduces another "useless" > > dependency. > > My apologies, I thought you were talking about your own code. > > As far as -- it wouldn't be awesome to me if > I couldn't pass in the data types I need. > > -- > ~Ethan~ I just hope you are right. Thank you (and Brett) for clearing things up for me. My apologies if I came off as (a bit) harsh. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Aug 19, 2016 6:46 PM, Chris Angelico wrote:
>
> On Sat, Aug 20, 2016 at 2:25 AM, Steve Dower wrote:
> > On 19Aug2016 0910, Chris Angelico wrote:
> >>
> >> On Sat, Aug 20, 2016 at 1:26 AM, Steve Dower
> >> wrote:
> >>>
> >>> Check any .pth files you can find. I suspect mpl_toolkits has some magic
> >>> in
> >>> it to make the namespace package work on 2.7.
> >>
> >>
> >> $ cat
> >> /usr/local/lib/python3.6/site-packages/matplotlib-1.5.1-py3.6-nspkg.pth
> >> import sys, types, os;p =
> >> os.path.join(sys._getframe(1).f_locals['sitedir'],
> >> *('mpl_toolkits',));ie =
> >> os.path.exists(os.path.join(p,'__init__.py'));m = not ie and
> >> sys.modules.setdefault('mpl_toolkits',
> >> types.ModuleType('mpl_toolkits'));mp = (m or []) and
> >> m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
> >>
> >> Is it possible that that's the cause? And if so, should "python -I"
> >> remove /usr/local/lib/python3.6/site-packages from sys.path?
> >
> >
> > That explains the extra modules - types.py imports collections and
> > collections.abc, which would seem to account for the rest.
> >
> > I'm not sure whether that's causing the problem or not, but temporarily
> > renaming that pth file so it isn't picked up may help you track down the
> > test issue.
>
> Sure enough, that did fix the problem - that is to say, renaming it
> did allow test_site to pass. Thanks!
>
> > "python -S" is the only way to avoid .pth files, but that's a pretty heavy
> > hammer.
>
> Hmmm. So the question is, what is this test testing? For
> reference, here's the code:
>
> class StartupImportTests(unittest.TestCase):
>
> def test_startup_imports(self):
> # This tests checks which modules are loaded by Python when it
> # initially starts upon startup.
> popen = subprocess.Popen([sys.executable, '-I', '-v', '-c',
> 'import sys; print(set(sys.modules))'],
> stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
> stdout, stderr = popen.communicate()
> stdout = stdout.decode('utf-8')
> stderr = stderr.decode('utf-8')
> modules = eval(stdout)
>
> self.assertIn('site', modules)
>
> # http://bugs.python.org/issue19205
> re_mods = {'re', '_sre', 'sre_compile', 'sre_constants', 'sre_parse'}
> # _osx_support uses the re module in many placs
> if sys.platform != 'darwin':
> self.assertFalse(modules.intersection(re_mods), stderr)
> # http://bugs.python.org/issue9548
> self.assertNotIn('locale', modules, stderr)
> if sys.platform != 'darwin':
> # http://bugs.python.org/issue19209
> self.assertNotIn('copyreg', modules, stderr)
> # http://bugs.python.org/issue19218>
> collection_mods = {'_collections', 'collections', 'functools',
> 'heapq', 'itertools', 'keyword', 'operator',
> 'reprlib', 'types', 'weakref'
> }.difference(sys.builtin_module_names)
> self.assertFalse(modules.intersection(collection_mods), stderr)
>
>
> It asserts that 'site' is in the list of loaded modules, so simply
> adding '-S' to the subpython parameters doesn't work. But this test,
> as written, depends on the behaviour of arbitrary third-party modules
> installed with pip, which seems wrong for a standard library test.
> AIUI this is trying to catch changes that accidentally cause large
> modules to be loaded on startup, correct? Which is all well and good,
> but Python shouldn't have test failures due to matplotlib being
> installed.
>
> I see two (or three) ways to deal with this.
>
> 1) Declare it a bug in matplotlib, and demand that .pth files not
> import anything that wouldn't otherwise be imported.
>
> 2) Change the test to add '-S' and remove the assertion about 'site'
> being present; this means that site.py itself is no longer covered by
> the test.
>
> 3) Accept the status quo: this test can fail if .pth files are present.
>
4) split the test in two: one running with -S to check no collections, (and for *not* site.py). The other without -S, and check for site.py.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failures in test_site.py - how to debug?
On Aug 19, 2016 11:10 PM, Chris Angelico wrote: > > On Sat, Aug 20, 2016 at 5:33 AM, Brett Cannon wrote: > >> Hmmm. So the question is, what is this test testing? > > > > > > It's making sure people who work on the modules that are imported during > > startup don't accidentally add another module dependency to the startup > > sequence. Since module imports impact startup time we have historically > > worked hard to minimize import dependencies for startup (hence things like > > certain functions in the os module doing imports within a function even > > though we discourage that practice in general). > > Fair enough. Which means it probably *should* import site, because > that's going to happen on all normal usage. So that still leaves a few > options: > > 1) Demand that .pth files restrict themselves to what's already > imported. This means startup is still fast even if you have a bunch of > pths. Downside: Third-party code can break Python's rules. Upside: > When they do, it can be reported as a bug, the fixing of which will > improve startup times when that's installed. > > 2) Change the test to somehow disable .pth execution while keeping the > rest of site.py intact. This gives more consistent test results, but > still is mostly applicable to normal usage. > > 3) Ignore the problem and assume that the Python test suite will be > run with no site-packages installed. Status quo, basically. Possibly > with some tweaks so running 'make test' ignores site-packages? > > Is #2 doable? If not, can 'make test' somehow hack out site-packages? > > ChrisA I'd like to re-iterate my suggestion in case it was missed: split the current test in 2 tests: * Running with `-S` which is for checking that by default the collections are not imported. (Which is what is currently tested) * Running without `-S` which is for checking that `site` gets imported. I hope that is clear enough. Do you see any problems with such an approach? Kind regards, Sjoerd Job ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
