Re: [Python-Dev] PEP 443 - Single-dispatch generic functions
On 23 May 2013 22:02, Ronan Lamy wrote: > 2013/5/23 Łukasz Langa >> Last one wins. Just like with assigning names in a scope, defining methods >> in a class or overriding them in a subclass. > > This is a serious annoyance, considering that there are several places where > a large library can reasonably define the implementations (i.e. with the > class, with the function, or in some utility module). Note that in contrast > with the case of functions in a module or methods in a class, linting tools > cannot be expected to detect a duplication between functions with different > names defined in different modules. But isn't it much much worse than names in scope, as with assigning names in a scope it is only your scope that is affected : from os.path import join def join(wibble): 'overloads join in this module only' any other module is unaffected, os.path.join still calls os.path.join however with this all scopes globally are affected by the last one wins rule. -default.py--- from pkgutil import simplegeneric @simplegeneric def fun(x): print 'default impl' ---a.py from default import fun @fun.register(int) def impl_a(x): print 'impl_a' def f(): fun(0) # expect this to call impl_a ---b.py-- from default import fun @fun.register(int) def impl_b(x): print 'impl_b' def f(): fun(0) # expect this to call impl_b >>> import a, b >>> a.f() impl_b >>> b.f() impl_b >>> import b, a >>> a.f() impl_a >>> b.f() impl_a >>> exit() That is rather worrying. It is more analagous in the above example to sys.modules['os.path'].join = myjoin I don't have a solution mind though. Sam ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 443 - Single-dispatch generic functions
On Thu, May 23, 2013 at 6:40 PM, Steven D'Aprano wrote: > I don't think that they will. Being able to register multiple types with a > single call reads very naturally to me, while multiple decorators still > looks weird. Even after many years of seeing them, I still get a momentary > "What the hell...?" moment when I see two decorators on one function. That's > only going to be increased when both decorators are the same (apart from the > argument). The double decorator form above looks to me as weird as: > > x = func(a) > x = func(b) > > > would. I have to stop and think about what is going on, and whether or not > it is a mistake. That's absurd. The above is not comparable to double decorators, the following is: x = func(a) x = func(x) And this is clearly not something anyone has to stop and think about. (more literally, obviously it's actually def x(...): ... ; x = func(a)(x); x = func(b)(x)) There is nothing remotely wrong or distasteful about using multiple decorators. It's a natural thing to want to compose multiple functions together; for example, @functools.lru_cache with @fun.register or @staticmethod or [...]. And it's even natural to want to apply the same decorator with different arguments multiple times to the same thing, if it happens to do something different when given different arguments. -- Devin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 443 - Single-dispatch generic functions
On 24/05/13 15:09, Nick Coghlan wrote: On Fri, May 24, 2013 at 8:40 AM, Steven D'Aprano wrote: I don't think that they will. Being able to register multiple types with a single call reads very naturally to me, while multiple decorators still looks weird. Even after many years of seeing them, I still get a momentary "What the hell...?" moment when I see two decorators on one function. That's only going to be increased when both decorators are the same (apart from the argument). The double decorator form above looks to me as weird as: x = func(a) x = func(b) would. I have to stop and think about what is going on, and whether or not it is a mistake. The difference is that this idiom quickly becomes familiar and unexceptional: @fun.register(float) @fun.register(Decimal) def fun_floating_point(arg1, arg2): ... I initially wrote a reply about the nature of ambiguity, why register(float, Decimal) should not be considered ambiguous, why stacked decorators that are near-duplicates are a code smell, blah blah blah. But for the sake of brevity I'm going to skip it. The important point that you make is here: Is that multiple dispatch? Or is it registering for single dispatch on multiple different types? Sure, we could pick the latter meaning for the standard library, but existing generic function implementations (cited in the PEP) use the tuple-of-types notation for multiple dispatch. This is an excellent point I had not considered. By the way, it seems to me that Guido's multimethod implementation referenced in the PEP actually uses a single decorator argument per function argument, not a tuple-of-types: @multimethod(int, int) def foo(a, b): ...code for two ints... http://www.artima.com/weblogs/viewpost.jsp?thread=101605 You have convinced me: ambiguous or not, for the sake of future expansion I agree that multiple positional arguments to the register method should be left for some hypothetical multiple-dispatch generics: @fun.register(float, Decimal) # not yet supported, but maybe someday would mean "first argument is a float, second argument is a Decimal". But that still leaves open how to specify single dispatch on more than one type: stacking registration decorators is just a new idiom to become accustomed to. Python built-ins and the standard library already have a standard idiom for specifying multiple values at once. A tuple of types is the One Obvious Way to do this: @fun.register((float, Decimal)) which matches the same standard idiom that should be familiar to most people: isinstance(obj, (float, Decimal)) issubclass(cls, (float, Decimal)) And of course it is forward-compatible with our hypothetical future multiple-generics. -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 443 - Single-dispatch generic functions
On Fri, May 24, 2013 at 8:53 PM, Steven D'Aprano wrote: > Python built-ins and the standard library already have a standard idiom for > specifying multiple values at once. A tuple of types is the One Obvious Way > to do this: > > @fun.register((float, Decimal)) It's not obvious, it's ambiguous - some third party libraries use that notation for multi-method dispatch, and they always will, no matter what notation we choose for the standard library. We have three available notations to register the same function for multiple types: stacked decorators, tuple-of-types and multiple arguments. Of those, the first we *cannot avoid* supporting, since we want to return the undecorated function regardless for pickle support and ease of testing. The second two are both used as notations by existing third party multiple dispatch libraries. Thus, your request is that we add a second way to do it that is *known* to conflict with existing third party practices. There is no practical gain on offer, it merely aligns with your current sense of aesthetics slightly better than stacked decorators do. While you're entitled to that aesthetic preference, it isn't a valid justification for adding an unneeded alternate spelling. Furthermore, the proposed registration syntax in the PEP is identical to the syntax which already exists for ABC registration as a class decorator (http://docs.python.org/3/library/abc#abc.ABCMeta.register). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 443 - Single-dispatch generic functions
On Fri, May 24, 2013 at 9:41 PM, Nick Coghlan wrote: > Furthermore, the proposed registration syntax in the PEP is identical > to the syntax which already exists for ABC registration as a class > decorator (http://docs.python.org/3/library/abc#abc.ABCMeta.register). Sorry, I withdraw that observation - it's wrong. ABC registration obviously doesn't need to provide arguments to the decorator at all, since the only necessary info is the ABC itself, and that's providing through the method binding. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 443 - Single-dispatch generic functions
On Fri, May 24, 2013 at 7:54 PM, Sam Partington wrote: > But isn't it much much worse than names in scope, as with assigning > names in a scope it is only your scope that is affected : > > from os.path import join > def join(wibble): > 'overloads join in this module only' > > any other module is unaffected, os.path.join still calls os.path.join > > however with this all scopes globally are affected by the last one wins rule. Indeed, as with any modification of process global state, generic implementation registration is something to be approached with care. ABC registration is similar. There's actually three kinds of registration that can happen, and only two of them are appropriate for libraries to do implicitly, while the last should only be explicitly triggered from main: * registering a class your library defines with a stdlib or third party generic function * registering a stdlib or third party class with a generic function your library defines * registering a stdlib or third party class with a stdlib or third party generic function The first two cases? Those are just part of defining class behaviour or function behaviour. That's entirely up to the library developer and an entirely sensible thing for them to be doing. That third case? It's the moral equivalent of monkey patching, and it's the strict prerogative of application integrators. The core assumption is that on import, you're just providing one component of an application, and you don't know what that application is or what it's needs are. By contrast, when you're explicitly called from main, then you can assume that this is an explicit request from the *integrated* application that wants you to modify the global state. One of the best examples of a project that gets this distinction right is gevent - you can general import gevent without any side effects on the process global state. However, the gevent.monkey module exposes monkeypatching that the *application* developer can request. You know you have a well written library if someone else could import every single one of your modules into their application and it would have *zero* effect on them until they call a function. This is often the tipping point that pushes things over from being libraries to being frameworks: the frameworks have side effects on import that mean they don't play well with others (implicitly configuring the logging system is a common example - the changes to the logging module's default behaviour in 3.2 were designed to make it easier for library developers to *stop doing that*, because it causes spurious incompatibilities. Messing too much with the import system as a side effect of import is also framework-like behaviour). Making *any* changes outside your module scope as a side effect of import can be problematic, since even if it doesn't conflict with another library, it has a tendency to break module reloading. One of the minor reasons that ABC registration, and the proposed single dispatch registration, permit silent overwriting is that being too aggressive about enforcing "once and only once" can make module reloading even more fragile than it is already. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 443 - Single-dispatch generic functions
2013/5/24 Ethan Furman > On 05/23/2013 02:02 PM, Ronan Lamy wrote: > >> 2013/5/23 Łukasz Langa mailto:luk...@langa.pl>> >> >> >> On 23 maj 2013, at 20:13, Éric Araujo > mer...@netwok.org>> wrote: >> >> > Question: what happens if two functions (say in two different >> modules) >> > are registered for the same type? >> >> Last one wins. Just like with assigning names in a scope, defining >> methods >> in a class or overriding them in a subclass. >> >> >> This is a serious annoyance, considering that there are several places >> where a large library can reasonably define the >> implementations (i.e. with the class, with the function, or in some >> utility module). Note that in contrast with the case >> of functions in a module or methods in a class, linting tools cannot be >> expected to detect a duplication between >> functions with different names defined in different modules. >> > > What would you suggest happen in this case? > Raise a ValueError, maybe? In that case, there needs to be a way to force the overriding when it is explicitly desired. One way would be to allow unregistering implementations: overriding is then done by unregistering the old implementation before defining the new one. This is a bit cumbersome, which IMHO is a good thing for an operation that is just as disruptive as monkey-patching a class or a module. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 443 - Single-dispatch generic functions
On 24 maj 2013, at 14:22, Ronan Lamy wrote: >> 2013/5/24 Ethan Furman >> What would you suggest happen in this case? > Raise a ValueError, maybe? In that case, there needs to be a way to force the > overriding when it is explicitly desired. One way would be to allow > unregistering implementations: overriding is then done by unregistering the > old implementation before defining the new one. Unfortunately this isn't going to work because the order of imports might change during the life cycle of a program. Especially if you wish to expose registering to users, the order of imports cannot be guaranteed. I recognize the need for such behaviour to be discoverable. This is important for debugging purposes. This is why I'm going to let users inspect registered overloads, as well as provide their own mapping for the registry. I'm working on the reference implementation now, stay tuned. -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 443 - Single-dispatch generic functions
On Fri, May 24, 2013 at 10:22 PM, Ronan Lamy wrote: > Raise a ValueError, maybe? In that case, there needs to be a way to force > the overriding when it is explicitly desired. One way would be to allow > unregistering implementations: overriding is then done by unregistering the > old implementation before defining the new one. This is a bit cumbersome, > which IMHO is a good thing for an operation that is just as disruptive as > monkey-patching a class or a module. If you're registering an implementation for a type you didn't define on a generic function you didn't define, it's *exactly* as disruptive as monkey-patching. Note that the PEP proposes giving exactly as much of a runtime warning about overwriting a registration as we do about monkeypatching: none. The two cases are exactly analagous: you can do it, you don't get a warning if you do it, but it you do it implicitly as a side effect of import then you will have developers cursing your name. So don't do that, put it in a function that people can call if they want to register your implementations (along the lines of gevent.monkey). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 443 - Single-dispatch generic functions
On 24 maj 2013, at 14:53, Ronan Lamy wrote: >> 2013/5/24 Łukasz Langa >> >> I recognize the need for such behaviour to be discoverable. This is >> important for debugging purposes. This is why I'm going to let users >> inspect registered overloads, as well as provide their own mapping >> for the registry. I'm working on the reference implementation now, >> stay tuned. > > OK, I agree that preventing silent overwriting is actually not desirable. > Specifying an interface for the implementation registry sounds like the best > possible solution to my concerns. Users can now inspect the dispatcher and specify their own mapping for a registry: http://hg.python.org/features/pep-443/file/tip/Lib/functools.py#l359 Documentation: http://hg.python.org/features/pep-443/file/tip/Doc/library/functools.rst#l189 ABC support is still not there, I'm working on it. -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2013-05-17 - 2013-05-24) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open3972 ( +6) closed 25850 (+45) total 29822 (+51) Open issues with patches: 1773 Issues opened (41) == #13146: Writing a pyc file is not atomic http://bugs.python.org/issue13146 reopened by barry #15535: Fix pickling efficiency of named tuples in 2.7.3 http://bugs.python.org/issue15535 reopened by rhettinger #18000: _md5 should be built if _ssl cannot be built http://bugs.python.org/issue18000 opened by jdemeyer #18003: New lzma crazy slow with line-oriented reading. http://bugs.python.org/issue18003 opened by Michael.Fox #18004: test_list.test_overflow crashes Win64 http://bugs.python.org/issue18004 opened by anselm.kruis #18009: os.write.__doc__ is misleading http://bugs.python.org/issue18009 opened by shai #18010: pydoc search chokes on import errors http://bugs.python.org/issue18010 opened by pitrou #18011: Inconsistency between b32decode() documentation, docstring and http://bugs.python.org/issue18011 opened by serhiy.storchaka #18013: cgi.FieldStorage does not parse W3C sample http://bugs.python.org/issue18013 opened by flox #18014: Problem compiling on Windows, VC++Express 2010 http://bugs.python.org/issue18014 opened by terry.reedy #18015: python 2.7.5 fails to unpickle namedtuple pickled by 2.7.3 or http://bugs.python.org/issue18015 opened by anselm.kruis #18016: subprocess should open stdin in mode w+b on windows http://bugs.python.org/issue18016 opened by Jason.Gross #18017: ctypes.PyDLL documentation http://bugs.python.org/issue18017 opened by Marc.Brünink #18018: SystemError: Parent module '' not loaded, cannot perform relat http://bugs.python.org/issue18018 opened by flox #18020: html.escape 10x slower than cgi.escape http://bugs.python.org/issue18020 opened by flox #18021: Update broken link to Apple Publication Style Guide http://bugs.python.org/issue18021 opened by madison.may #18022: Inconsistency between quopri.decodestring() and email.quoprimi http://bugs.python.org/issue18022 opened by serhiy.storchaka #18023: msi product code for 2.7.5150 not in Tools/msi/uuids.py http://bugs.python.org/issue18023 opened by anselm.kruis #18024: dbm module fails to build on SLES11SP1 using 2.7.5 source http://bugs.python.org/issue18024 opened by wempa #18025: Buffer overflow in BufferedIOBase.readinto() http://bugs.python.org/issue18025 opened by serhiy.storchaka #18027: distutils should access stat_result timestamps via .st_*time a http://bugs.python.org/issue18027 opened by jwilk #18028: Warnings with -fstrict-aliasing http://bugs.python.org/issue18028 opened by bkabrda #18029: Python SSL support is missing from SPARC build http://bugs.python.org/issue18029 opened by eeiddne #18032: set methods should specify whether they consume iterators "laz http://bugs.python.org/issue18032 opened by hhm #18033: Example for Profile Module shows incorrect method http://bugs.python.org/issue18033 opened by jough #18034: Last two entries in the programming FAQ are out of date (impor http://bugs.python.org/issue18034 opened by r.david.murray #18035: telnetlib incorrectly assumes that select.error has an errno a http://bugs.python.org/issue18035 opened by gregory.p.smith #18036: "How do I create a .pyc file?" FAQ entry is out of date http://bugs.python.org/issue18036 opened by r.david.murray #18037: 2to3 passes through string literal which causes SyntaxError in http://bugs.python.org/issue18037 opened by vinay.sajip #18038: Unhelpful error message on invalid encoding specification http://bugs.python.org/issue18038 opened by Max.Cantor #18039: dbm.open(..., flag="n") does not work and does not give a warn http://bugs.python.org/issue18039 opened by sonyachiko #18040: SIGINT catching regression on windows in 2.7 http://bugs.python.org/issue18040 opened by David.Gilman #18041: mention issues with code churn in the devguide http://bugs.python.org/issue18041 opened by tshepang #18042: Provide enum.unique class decorator http://bugs.python.org/issue18042 opened by ncoghlan #18043: No mention of `match.regs` in `re` documentation http://bugs.python.org/issue18043 opened by cool-RR #18044: Email headers do not properly decode to unicode. http://bugs.python.org/issue18044 opened by Tim.Rawlinson #18045: get_python_version is not import in bdist_rpm.py http://bugs.python.org/issue18045 opened by maoliping455 #18046: Simplify and clarify logging internals http://bugs.python.org/issue18046 opened by alex #18047: Descriptors get invoked in old-style objects and classes http://bugs.python.org/issue18047 opened by icecrime #18048: Merging test_pep263.py and test_coding.py http://bugs.python.org/issue18048 opened by serhiy.storchaka #18049: Re-enable threading test on OSX http://bugs.python.org/issue180
[Python-Dev] Bilingual scripts
Here's something that seems to come up from time to time in Debian. Take a Python application like tox, nose, or pyflakes. Their executables work with both Python 2 and 3, but require a #! line to choose which interpreter to invoke. When we add Python 3 support in Debian for such a script, all the library installations are handled just fine, but we have conflicts about what to name the thing that lands in /usr/bin. Do we have /usr/bin/pyflakes and /usr/bin/pyflakes3? Do we call the latter py3flakes (as has been convention with some other scripts, but which breaks locate(1))? Do we simply remove the /usr/bin scripts and encourage people to use something like `$python -m nose`? One interesting recent suggestion is to create a shell driver script and make that thing accept --py3 and --py2 flags, which would then select the appropriate interpreter and invoke the actual Python driver. There are some technical details to iron out with that, but there's some appeal to it. Have any other *nix distros addressed this, and if so, how do you solve it? It would be nice if we could have some cross-platform recommendations so things work the same wherever you go. To that end, if we can reach some consensus, I'd be willing to put together an informational PEP and some scripts that might be of general use. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bilingual scripts
On Fri, 24 May 2013 15:56:29 -0400, Barry Warsaw wrote: > Have any other *nix distros addressed this, and if so, how do you solve it? > It would be nice if we could have some cross-platform recommendations so > things work the same wherever you go. To that end, if we can reach some > consensus, I'd be willing to put together an informational PEP and some > scripts that might be of general use. Gentoo has a (fairly complex) driver script that is symlinked to all of these bin scripts. The system then has the concept of the "current python", which can be set to python2 or python3. The default bin then calls the current default interpreter. There are also xxx2 and xxx3 versions of each bin script, which call the 'current' version of python2 or python3, respectively. I'm sure one of the gentoo devs on this list can speak to this more completely...I'm just a user :) But I must say that the system works well from my point of view. --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bilingual scripts
On Fri, May 24, 2013 at 10:23 PM, R. David Murray wrote:
> Gentoo has a (fairly complex) driver script that is symlinked to all
> of these bin scripts. The system then has the concept of the
> "current python", which can be set to python2 or python3. The default
> bin then calls the current default interpreter. There are also
> xxx2 and xxx3 versions of each bin script, which call the 'current'
> version of python2 or python3, respectively.
I'm one of the Gentoo devs, on the python team. I haven't actually
written any code for this, but I can show a little of what's going on.
I think most of the code is actually in
https://bitbucket.org/mgorny/python-exec. We then install three
scripts:
lrwxrwxrwx 1 root root 11 May 20 14:06 /usr/bin/sphinx-build -> python-exec
-rwxr-xr-x 1 root root 311 May 20 14:06 /usr/bin/sphinx-build-python2.7
-rwxr-xr-x 1 root root 311 May 20 14:06 /usr/bin/sphinx-build-python3.2
sphinx-build-python2.7 looks like this:
#!/usr/bin/python2.7
# EASY-INSTALL-ENTRY-SCRIPT: 'Sphinx==1.1.3','console_scripts','sphinx-build'
__requires__ = 'Sphinx==1.1.3'
import sys
from pkg_resources import load_entry_point
if __name__ == '__main__':
sys.exit(
load_entry_point('Sphinx==1.1.3', 'console_scripts', 'sphinx-build')()
)
We now use a python2.7 suffix rather than just a 2.7 suffix because we
will install separate wrappers for e.g. pypy1.9 (and we are also
prepared to support jython or other implementations at some point).
If you have any more questions, I'll try to answer them; or, join
#gentoo-python on Freenode, there are generally people hanging out
there who know much more about our setup than I do.
Cheers,
Dirkjan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
