See issue 22766 for some background on this question.
On Mon, 03 Nov 2014 02:30:53 -0800, Ethan Furman <et...@stoneleaf.us> wrote: > Just to be clear, this is about NotImplemented, not NotImplementedError. > > tl;dr When a binary operation fails, should an exception be raised or > NotImplemented returned? > > > When a binary operation in Python is attempted, there are two possibilities: > > - it can work > - it can't work > > The main reason [1] that it can't work is that the two operands are of > different types, and the first type does not know > how to deal with the second type. > > The question then becomes: how does the first type tell Python that it cannot > perform the requested operation? The most > obvious answer is to raise an exception, and TypeError is a good candidate. > The problem with the exception raising > approach is that once an exception is raised, Python doesn't try anything > else to make the operation work. > > What's wrong with that? Well, the second type might know how to perform the > operation, and in fact that is why we have > the reflected special methods, such as __radd__ and __rmod__ -- but if the > first type raises an exception the __rxxx__ > methods will not be tried. > > Okay, how can the first type tell Python that it cannot do what is requested, > but to go ahead and check with the second > type to see if it does? That is where NotImplemented comes in -- if a > special method (and only a special method) > returns NotImplemented then Python will check to see if there is anything > else it can do to make the operation succeed; > if all attempts return NotImplemented, then Python itself will raise an > appropriate exception [2]. > > In an effort to see how often NotImplemented is currently being returned I > crafted a test script [3] to test the types > bytes, bytearray, str, dict, list, tuple, Enum, Counter, defaultdict, deque, > and OrderedDict with the operations for > __add__, __and__, __floordiv__, __iadd__, __iand__, __ifloordiv__, > __ilshift__, __imod__, __imul__, __ior__, __ipow__, > __irshift__, __isub__, __itruediv__, __ixor__, __lshift__, __mod__, __mul__, > __or__, __pow__, __rshift__, __sub__, > __truediv__, and __xor__. > > Here are the results of the 275 tests: > -------------------------------------------------------------------------------- > testing control... > > ipow -- Exception <unsupported operand type(s) for ** or pow(): 'Control' and > 'subtype'> raised > errors in Control -- misunderstanding or bug? > > testing types against a foreign class > > iadd(Counter()) -- Exception <'SomeOtherClass' object has no attribute > 'items'> raised instead of TypeError > iand(Counter()) -- NotImplemented not returned, TypeError not raised > ior(Counter()) -- Exception <'SomeOtherClass' object has no attribute > 'items'> raised instead of TypeError > isub(Counter()) -- Exception <'SomeOtherClass' object has no attribute > 'items'> raised instead of TypeError > > > testing types against a subclass > > mod(str()) -- NotImplemented not returned, TypeError not raised > > iadd(Counter()) -- Exception <'subtype' object has no attribute 'items'> > raised (should have worked) > iand(Counter()) -- NotImplemented not returned, TypeError not raised > ior(Counter()) -- Exception <'subtype' object has no attribute 'items'> > raised (should have worked) > isub(Counter()) -- Exception <'subtype' object has no attribute 'items'> > raised (should have worked) > -------------------------------------------------------------------------------- > > Two observations: > > - __ipow__ doesn't seem to behave properly in the 3.x line (that error > doesn't show up when testing against 2.7) > > - Counter should be returning NotImplemented instead of raising an > AttributeError, for three reasons [4]: > - a TypeError is more appropriate > - subclasses /cannot/ work with the current implementation > - __iand__ is currently a silent failure if the Counter is empty, and > the other operand should trigger a failure > > Back to the main point... > > So, if my understanding is correct: > > - NotImplemented is used to signal Python that the requested operation > could not be performed > - it should be used by the binary special methods to signal type mismatch > failure, so any subclass gets a chance to work. > > Is my understanding correct? Is this already in the docs somewhere, and I > just missed it? > > -- > ~Ethan~ > > [1] at least, it's the main reason in my code > [2] usually a TypeError, stating either that the operation is not supported, > or the types are unorderable > [3] test script at the end > [4] https://bugs.python.org/issue22766 [returning NotImplemented was rejected] > > -- 8< > ---------------------------------------------------------------------------- > from collections import Counter, defaultdict, deque, OrderedDict > from fractions import Fraction > from decimal import Decimal > from enum import Enum > import operator > import sys > > py_ver = sys.version_info[:2] > > types = ( > bytes, bytearray, str, dict, list, tuple, > Enum, Counter, defaultdict, deque, OrderedDict, > ) > numeric_types = int, float, Decimal, Fraction > > operators = ( > '__add__', '__and__', '__floordiv__', > '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__', > '__imod__', '__imul__', '__ior__', '__ipow__', > '__irshift__', '__isub__', '__itruediv__', '__ixor__', > '__lshift__', '__mod__', '__mul__', > '__or__', '__pow__', '__rshift__', '__sub__', '__truediv__', > '__xor__', > ) > > if py_ver >= (3, 0): > operators += ('__gt__', '__ge__', '__le__','__lt__') > > ordered_reflections = { > '__le__': '__ge__', > '__lt__': '__gt__', > '__ge__': '__le__', > '__gt__': '__lt__', > } > > > # helpers > > class SomeOtherClass: > """" > used to test behavior when a different type is passed in to the > special methods > """ > def __repr__(self): > return 'SomeOtherClass' > some_other_class = SomeOtherClass() > > class MainClassHandled(Exception): > """ > called by base class if both operands are of type base class > """ > > class SubClassCalled(Exception): > """ > called by reflected operations for testing > """ > > def create_control(test_op): > def _any(self, other): > if not type(other) is self.__class__: > return NotImplemented > raise MainClassHandled > class Control: > "returns NotImplemented when other object is not supported" > _any.__name__ = op > setattr(Control, test_op, _any) > return Control() > > def create_subtype(test_op, base_class=object): > def _any(*a): > global subclass_called > subclass_called = True > raise SubClassCalled > class subtype(base_class): > __add__ = __sub__ = __mul__ = __truediv__ = __floordiv__ = _any > __mod__ = __divmod__ = __pow__ = __lshift__ = __rshift__ = _any > __and__ = __xor__ = __or__ = _any > __radd__ = __rsub__ = __rmul__ = __rtruediv__ = __rfloordiv__ = _any > __rmod__ = __rdivmod__ = __rpow__ = __rlshift__ = __rrshift__ = _any > __rand__ = __rxor__ = __ror__ = _any > __le__ = __lt__ = __gt__ = __ge__ = _any > if issubclass(subtype, (bytes, bytearray)): > value = b'hello' > elif issubclass(subtype, str): > value = 'goodbye' > elif issubclass(subtype, (list, tuple)): > value = (1, 2, 3) > elif issubclass(subtype, (int, float, Decimal, Fraction)): > value = 42 > else: > # ignore value > return subtype() > return subtype(value) > > > # test exceptions > > # control against some other class > print('testing control...\n') > errors = False > for op in operators: > control = create_control(op) > op = getattr(operator, op) > try: > op(control, some_other_class) > except TypeError: > # the end result of no method existing, or each method called > returning > # NotImplemented because it does not know how to perform the > requested > # operation between the two types > pass > except Exception as exc: > errors = True > print('%s(%s()) -- Exception <%s> raised instead of TypeError' % > (op.__name__, test_type.__name__, exc)) > else: > errors = True > print('Control -- TypeError not raised for op %r' % op) > if errors: > print('errors in Control -- misunderstanding or bug?\n') > > # control against a subclass > errors = False > for op in operators: > subclass_called = False > control = create_control(op) > subtype = create_subtype(op, control.__class__) > op = getattr(operator, op) > try: > op(control, subtype) > except SubClassCalled: > # if the control class properly signals that it doesn't know how to > # perform the operation, of if Python notices that a reflected > # operation exists, we get here (which is good) > pass > except MainClassHandled: > errors = True > print('Control did not yield to subclass for op %r' % op) > except Exception as exc: > if subclass_called: > # exception was subverted to something more appropriate (like > # unorderable types) > pass > errors = True > print('%s -- Exception <%s> raised' % > (op.__name__, exc)) > else: > errors = True > print('Control -- op %r appears to have succeeded (it should not > have)' % op) > if errors: > print('errors in Control -- misunderstanding or bug?\n') > > > # tests > print('testing types against a foreign class\n') > for test_type in types + numeric_types: > errors = False > for op in operators: > op = getattr(operator, op) > try: > op(test_type(), some_other_class) > except TypeError: > pass > except Exception as exc: > errors = True > print('%s(%s()) -- Exception <%s> raised instead of TypeError' % > (op.__name__, test_type.__name__, exc)) > else: > print('%s(%s()) -- NotImplemented not returned, TypeError not > raised' % > (op.__name__, test_type.__name__)) > if errors: > print() > > print() > > # test subclasses > print('testing types against a subclass\n') > for test_type in types: > errors = False > for op in operators: > subclass_called = False > if not test_type.__dict__.get(op): > continue > subclass = create_subtype(op, test_type) > op = getattr(operator, op) > try: > if test_type is str: > op('%s', subtype) > else: > op(test_type(), subtype) > except SubClassCalled: > # expected, ignore > pass > except Exception as exc: > if subclass_called: > # exception raised by subclass was changed > pass > errors = True > print('%s(%s()) -- Exception <%s> raised (should have worked)' % > (op.__name__, test_type.__name__, exc)) > else: > errors = True > print('%s(%s()) -- NotImplemented not returned, TypeError not > raised' % > (op.__name__, test_type.__name__)) > if errors: > print() > for test_type in numeric_types: > errors = False > for op in operators: > subclass_called = False > if not test_type.__dict__.get(op): > continue > subtype = create_subtype(op, test_type) > op = getattr(operator, op) > try: > op(test_type(), subtype) > except SubClassCalled: > # expected, ignore > pass > except Exception as exc: > if subclass_called: > # exception raised by subclass was changed > pass > errors = True > print('%s(%s()) -- Exception <%s> raised (should have worked)' % > (op.__name__, test_type.__name__, exc)) > else: > errors = True > print('%s(%s)) -- NotImplemented not returned' % > (op.__name__, test_type.__name__)) > if errors: > print() > -- 8< > ---------------------------------------------------------------------------- > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rdmurray%40bitdance.com _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
