Re: [Python-Dev] PEP 562

[Ivan Levkivskyi]

On 15 November 2017 at 08:43, Serhiy Storchaka  wrote:

> 14.11.17 22:34, Ivan Levkivskyi пише:
>
>> This function will be called only if ``name`` is not found in the module
>> through the normal attribute lookup.
>>
>
> It is worth to mention that using name as a module global will bypass
> __getattr__. And this is intentional, otherwise calling __getattr__ for
> builtins will harm a performance.
>
>
Good point!


> Backwards compatibility and impact on performance
>> =
>>
>
> What is affect on pydoc, word completion, inspect, pkgutil, unittest?
>
>
This is rather gray area. I am not sure that we need to update them in any
way, just the people who use __getattr__ should be aware that
some tools might not yet expect it. I will add a note to the PEP about this.


>def keep_pickleable(func):
>>func.__name__ = func.__name__.replace('_deprecated_', '')
>>func.__qualname__ = func.__qualname__.replace('_deprecated_', '')
>>return func
>>
>>@keep_pickleable
>>def _deprecated_old_function(arg, other):
>>...
>>
>
> I would create more standard helpers (for deprecation, for lazy
> importing). This feature is helpful not by itself, but because it will be
> used for implementing new features. Using __getattr__ directly will need to
> write a boilerplate code. Maybe when implementing these helper you will
> discover that this PEP needs some additions.
>
>
>
But in which module these helpers should live?

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 562

[Serhiy Storchaka]

15.11.17 12:53, Ivan Levkivskyi пише:
On 15 November 2017 at 08:43, Serhiy Storchaka > wrote:


It is worth to mention that using name as a module global will
bypass __getattr__. And this is intentional, otherwise calling
__getattr__ for builtins will harm a performance.


Good point!


And please document idiomatic way of using a module global with 
triggering __getattr__. For example if you want to use a lazy loaded 
submodule.


sys.modules[__name__].foobar

or

from . import foobar

The difference between them that the latter sets the module attribute, 
thus __getattr__ will be called only once.



Backwards compatibility and impact on performance
=


What is affect on pydoc, word completion, inspect, pkgutil, unittest?


This is rather gray area. I am not sure that we need to update them in 
any way, just the people who use __getattr__ should be aware that
some tools might not yet expect it.. I will add a note to the PEP about 
this.


This problem is not new, since it was possible to replace a module with 
a module subclass with overridden __getattr__ and __dir__ before, but 
now this problem can occur more often.



I would create more standard helpers (for deprecation, for lazy
importing). This feature is helpful not by itself, but because it
will be used for implementing new features. Using __getattr__
directly will need to write a boilerplate code. Maybe when
implementing these helper you will discover that this PEP needs some
additions.



But in which module these helpers should live?


Good question. lazy_import() could be added in importlib (or 
importlib.util?). The helper that just adds deprecation on importing a 
name, could be added in importlib too. But I think that it would be 
better if the deprecated() helper will also create a wrapper that raises 
a deprecation warning on the use of deprecated function. It could be 
added in the warnings or functools modules.


I would add also a more general lazy_initialized(). It is something like 
cached module property. Executes the specified code on first use, and 
cache the result as a module attribute.


In all these cases the final __getattr__ method should be automatically 
constructed from different chunks. At the end it could call a user 
supplied __getattr__. Or maybe the module method __getattr__ should look 
first at special registry before calling the instance attribute 
__getattr__()?


def ModuleType.__getattr__(self, name):
if name in self.__properties__:
call self.__properties__[name]()
elif '__getattr__' in self.__dict__:
call self.__dict__['__getattr__'](name)
else:
raise AttributeError

I'm wondering if the __set_name__ mechanism can be extended to modules. 
What if call the __set_name__() method for all items in a module dict 
after finishing importing the module?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 562

[Koos Zevenhoven]

On Tue, Nov 14, 2017 at 10:34 PM, Ivan Levkivskyi 
wrote:
​[..]​


> Rationale
> =
>
> It is sometimes convenient to customize or otherwise have control over
> access to module attributes. A typical example is managing deprecation
> warnings. Typical workarounds are assigning ``__class__`` of a module
> object
> to a custom subclass of ``types.ModuleType`` or replacing the
> ``sys.modules``
> item with a custom wrapper instance. It would be convenient to simplify
> this
> procedure by recognizing ``__getattr__`` defined directly in a module that
> would act like a normal ``__getattr__`` method, except that it will be
> defined
> on module *instances*. For example::
>
>
  # lib.py
>
>   from warnings import warn
>
>   deprecated_names = ["old_function", ...]
>
>   def _deprecated_old_function(arg, other):
>   ...
>
>   def __getattr__(name):
>   if name in deprecated_names:
>   warn(f"{name} is deprecated", DeprecationWarning)
>   return globals()[f"_deprecated_{name}"]
>   raise AttributeError(f"module {__name__} has no attribute {name}")
>
>   # main.py
>
>   from lib import old_function  # Works, but emits the warning
>
>
​Deprecating functions is already possible, so I assume the reason for this
would be performance? If so, are you sure this would help for performance?
​
​Deprecating module attributes / globals is indeed difficult to do at
present. This PEP would allow deprecation warnings for accessing
attributes, which is nice!  However, as thread-unsafe as it is, many
modules use module attributes to configure the state of the module. In that
case, the user is more likely to *set* the attribute that to *get* it. Is
this outside the scope of the PEP?


​[..]​


> There is a related proposal PEP 549 that proposes to support instance
> properties for a similar functionality. The difference is this PEP proposes
> a faster and simpler mechanism, but provides more basic customization.
>

​I'm not surprised that the comparison is in favor of this PEP ;-).​


​[..]​


> Specification
> =
>
> The ``__getattr__`` function at the module level should accept one argument
> which is the name of an attribute and return the computed value or raise
> an ``AttributeError``::
>
>   def __getattr__(name: str) -> Any: ...
>
> This function will be called only if ``name`` is not found in the module
> through the normal attribute lookup.
>
>
The Rationale (quoted in the beginning of this email) easily leaves a
different impression of this.​


​[..]
​

>
> Discussion
> ==
>
> Note that the use of module ``__getattr__`` requires care to keep the
> referred
> objects pickleable. For example, the ``__name__`` attribute of a function
> should correspond to the name with which it is accessible via
> ``__getattr__``::
>
>   def keep_pickleable(func):
>   func.__name__ = func.__name__.replace('_deprecated_', '')
>   func.__qualname__ = func.__qualname__.replace('_deprecated_', '')
>   return func
>
>   @keep_pickleable
>   def _deprecated_old_function(arg, other):
>   ...
>
> One should be also careful to avoid recursion as one would do with
> a class level ``__getattr__``.
>
>
Off-topic: In some sense, I'm happy to hear something about pickleability.
But in some sense not.

I think there are three kinds of people regarding pickleability:

1. Those who don't care about anything being pickleable

2. Those ​who care about some things being picklable

​3. ​Those who care about all things being picklable

Personally, I'd like to belong to group 3, but because group 3 cannot even
attempt to coexist with groups 1 and 2, I actually belong to group 1 most
of the time.

​––Koos
​


> References
> ==
>
> .. [1] PEP 484 section about ``__getattr__`` in stub files
>(https://www.python.org/dev/peps/pep-0484/#stub-files)
>
> .. [2] The reference implementation
>(https://github.com/ilevkivskyi/cpython/pull/3/files)
>
>
> Copyright
> =
>
> This document has been placed in the public domain.
>
>
>
> ..
>Local Variables:
>mode: indented-text
>indent-tabs-mode: nil
>sentence-end-double-space: t
>fill-column: 70
>coding: utf-8
>End:
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> k7hoven%40gmail.com
>
>


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 560: bases classes / confusion

[Jim J. Jewett]

(1)  I found the following (particularly "bases classes") very confusing:

"""
If an object that is not a class object appears in the bases of a class

definition, then ``__mro_entries__`` is searched on it. If found,
it is called with the original tuple of bases as an argument. The result
of the call must be a tuple, that is unpacked in the bases classes in place
of this object. (If the tuple is empty, this means that the original bases
is
simply discarded.)
"""

Based on the following GenericAlias/NewList/Tokens example, I think I
now I understand what you mean, and would have had somewhat less
difficulty if it were expressed as:

"""
When an object that is not a class object appears in the (tuple of)
bases of a class
definition, then attribute ``__mro_entries__`` is searched on that
non-class object.  If ``__mro_entries__`` found,
it is called with the entire original tuple of bases as an argument. The result
of the call must be a tuple, which is unpacked and replaces only the
non-class object in the tuple of bases.  (If the tuple is empty, this
means that the original bases
is
simply discarded.)
"""

Note that this makes some assumptions about the __mro_entries__
signature that I wasn't quite sure about from the example.  So
building on that:

class ABList(A, NewList[int], B):

I *think* the following will happen:

"NewList[int]" will be evaluated, and __class_getitem__ called, so
that the bases tuple will be (A, GenericAlias(NewList, int), B)

# (A)  I *think* __mro_entries__ gets called with the full tuple,
# instead of just the object it is found on.
# (B) I *think* it is called on the results of evaluating
# the terms within the tuple, instead of the original
# string representation.
_tmp = __mro_entries__(A, GenericAlias(NewList, int), B)

# (C)  I *think* __mro_entries__ returns a replacement for
# just the single object, even though it was called on
# the whole tuple, without knowing which object it
# represents.
bases = (A, _tmp, B)

# (D) If there are two non-class objects, I *think* the
# second one gets the same arguments as the first,
# rather than an intermediate tuple with the first such
# object already substituted out.

-jJ
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 560: bases classes / confusion

[Koos Zevenhoven]

For anyone confused about similar things, I expect you to be interested in
my post on python-ideas from today:

https://mail.python.org/pipermail/python-ideas/2017-November/047896.html

––Koos


On Wed, Nov 15, 2017 at 4:20 PM, Jim J. Jewett  wrote:

> (1)  I found the following (particularly "bases classes") very confusing:
>
> """
> If an object that is not a class object appears in the bases of a class
>
> definition, then ``__mro_entries__`` is searched on it. If found,
> it is called with the original tuple of bases as an argument. The result
> of the call must be a tuple, that is unpacked in the bases classes in place
> of this object. (If the tuple is empty, this means that the original bases
> is
> simply discarded.)
> """
>
> Based on the following GenericAlias/NewList/Tokens example, I think I
> now I understand what you mean, and would have had somewhat less
> difficulty if it were expressed as:
>
> """
> When an object that is not a class object appears in the (tuple of)
> bases of a class
> definition, then attribute ``__mro_entries__`` is searched on that
> non-class object.  If ``__mro_entries__`` found,
> it is called with the entire original tuple of bases as an argument. The
> result
> of the call must be a tuple, which is unpacked and replaces only the
> non-class object in the tuple of bases.  (If the tuple is empty, this
> means that the original bases
> is
> simply discarded.)
> """
>
> Note that this makes some assumptions about the __mro_entries__
> signature that I wasn't quite sure about from the example.  So
> building on that:
>
> class ABList(A, NewList[int], B):
>
> I *think* the following will happen:
>
> "NewList[int]" will be evaluated, and __class_getitem__ called, so
> that the bases tuple will be (A, GenericAlias(NewList, int), B)
>
> # (A)  I *think* __mro_entries__ gets called with the full tuple,
> # instead of just the object it is found on.
> # (B) I *think* it is called on the results of evaluating
> # the terms within the tuple, instead of the original
> # string representation.
> _tmp = __mro_entries__(A, GenericAlias(NewList, int), B)
>
> # (C)  I *think* __mro_entries__ returns a replacement for
> # just the single object, even though it was called on
> # the whole tuple, without knowing which object it
> # represents.
> bases = (A, _tmp, B)
>
> # (D) If there are two non-class objects, I *think* the
> # second one gets the same arguments as the first,
> # rather than an intermediate tuple with the first such
> # object already substituted out.
>
> -jJ
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> k7hoven%40gmail.com
>



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 560: bases classes / confusion

[Nick Coghlan]

On 16 November 2017 at 00:20, Jim J. Jewett  wrote:

> I *think* the following will happen:
>
> "NewList[int]" will be evaluated, and __class_getitem__ called, so
> that the bases tuple will be (A, GenericAlias(NewList, int), B)
>
> # (A)  I *think* __mro_entries__ gets called with the full tuple,
> # instead of just the object it is found on.
> # (B) I *think* it is called on the results of evaluating
> # the terms within the tuple, instead of the original
> # string representation.
> _tmp = __mro_entries__(A, GenericAlias(NewList, int), B)
>
> # (C)  I *think* __mro_entries__ returns a replacement for
> # just the single object, even though it was called on
> # the whole tuple, without knowing which object it
> # represents.
> bases = (A, _tmp, B)
>

My understanding of the method signature:

def __mro_entries__(self, orig_bases):
...
return replacement_for_self

My assumption as to the purpose of the extra complexity was:

- given orig_bases, a method could avoid injecting bases already listed if
it wanted to
- allowing multiple items to be returned provides a way to programmatically
combine mixins without having to define a new subclass for each combination

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 562

[Guido van Rossum]

I think it's reasonable for the PEP to include some examples, consequences
and best practices. I don't think it's reasonable for the PEP to also
define the API and implementation of helper functions that might be added
once the mechanisms are in place. Those are better developed as 3rd party
packages first.

On Wed, Nov 15, 2017 at 3:59 AM, Serhiy Storchaka 
wrote:

> 15.11.17 12:53, Ivan Levkivskyi пише:
>
>> On 15 November 2017 at 08:43, Serhiy Storchaka > > wrote:
>>
>> It is worth to mention that using name as a module global will
>> bypass __getattr__. And this is intentional, otherwise calling
>> __getattr__ for builtins will harm a performance.
>>
>>
>> Good point!
>>
>
> And please document idiomatic way of using a module global with triggering
> __getattr__. For example if you want to use a lazy loaded submodule.
>
> sys.modules[__name__].foobar
>
> or
>
> from . import foobar
>
> The difference between them that the latter sets the module attribute,
> thus __getattr__ will be called only once.
>
> Backwards compatibility and impact on performance
>> =
>>
>>
>> What is affect on pydoc, word completion, inspect, pkgutil, unittest?
>>
>>
>> This is rather gray area. I am not sure that we need to update them in
>> any way, just the people who use __getattr__ should be aware that
>> some tools might not yet expect it.. I will add a note to the PEP about
>> this.
>>
>
> This problem is not new, since it was possible to replace a module with a
> module subclass with overridden __getattr__ and __dir__ before, but now
> this problem can occur more often.
>
> I would create more standard helpers (for deprecation, for lazy
>> importing). This feature is helpful not by itself, but because it
>> will be used for implementing new features. Using __getattr__
>> directly will need to write a boilerplate code. Maybe when
>> implementing these helper you will discover that this PEP needs some
>> additions.
>>
>>
>>
>> But in which module these helpers should live?
>>
>
> Good question. lazy_import() could be added in importlib (or
> importlib.util?). The helper that just adds deprecation on importing a
> name, could be added in importlib too. But I think that it would be better
> if the deprecated() helper will also create a wrapper that raises a
> deprecation warning on the use of deprecated function. It could be added in
> the warnings or functools modules.
>
> I would add also a more general lazy_initialized(). It is something like
> cached module property. Executes the specified code on first use, and cache
> the result as a module attribute.
>
> In all these cases the final __getattr__ method should be automatically
> constructed from different chunks. At the end it could call a user supplied
> __getattr__. Or maybe the module method __getattr__ should look first at
> special registry before calling the instance attribute __getattr__()?
>
> def ModuleType.__getattr__(self, name):
> if name in self.__properties__:
> call self.__properties__[name]()
> elif '__getattr__' in self.__dict__:
> call self.__dict__['__getattr__'](name)
> else:
> raise AttributeError
>
> I'm wondering if the __set_name__ mechanism can be extended to modules.
> What if call the __set_name__() method for all items in a module dict after
> finishing importing the module?
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
> 40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 562

[Ethan Furman]

On 11/15/2017 04:55 AM, Koos Zevenhoven wrote:

On Tue, Nov 14, 2017 at 10:34 PM, Ivan Levkivskyi wrote:





Rationale
=

[...] It would be convenient to simplify this
procedure by recognizing ``__getattr__`` defined directly in a module that
would act like a normal ``__getattr__`` method

>>
>> [...]
>>

Specification
=


>> The ``__getattr__`` function at the module level should accept one argument

which is the name of an attribute and return the computed value or raise
an ``AttributeError``::



   def __getattr__(name: str) -> Any: ...



This function will be called only if ``name`` is not found in the module
through the normal attribute lookup.


The Rationale (quoted in the beginning of this email) easily leaves a different 
impression of this.​


I don't see how.  This is exactly the way normal __getattr__ works.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 560: bases classes / confusion

[Ivan Levkivskyi]

Nick is exactly right here. Jim, if you want to propose alternative
wording, then we could consider it.

--
Ivan


On 15 November 2017 at 16:37, Nick Coghlan  wrote:

> On 16 November 2017 at 00:20, Jim J. Jewett  wrote:
>
>> I *think* the following will happen:
>>
>> "NewList[int]" will be evaluated, and __class_getitem__ called, so
>> that the bases tuple will be (A, GenericAlias(NewList, int), B)
>>
>> # (A)  I *think* __mro_entries__ gets called with the full tuple,
>> # instead of just the object it is found on.
>> # (B) I *think* it is called on the results of evaluating
>> # the terms within the tuple, instead of the original
>> # string representation.
>> _tmp = __mro_entries__(A, GenericAlias(NewList, int), B)
>>
>> # (C)  I *think* __mro_entries__ returns a replacement for
>> # just the single object, even though it was called on
>> # the whole tuple, without knowing which object it
>> # represents.
>> bases = (A, _tmp, B)
>>
>
> My understanding of the method signature:
>
> def __mro_entries__(self, orig_bases):
> ...
> return replacement_for_self
>
> My assumption as to the purpose of the extra complexity was:
>
> - given orig_bases, a method could avoid injecting bases already listed if
> it wanted to
> - allowing multiple items to be returned provides a way to
> programmatically combine mixins without having to define a new subclass for
> each combination
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 560: bases classes / confusion

[Koos Zevenhoven]

On Wed, Nov 15, 2017 at 5:37 PM, Nick Coghlan  wrote:

> On 16 November 2017 at 00:20, Jim J. Jewett  wrote:
>
>> I *think* the following will happen:
>>
>> "NewList[int]" will be evaluated, and __class_getitem__ called, so
>> that the bases tuple will be (A, GenericAlias(NewList, int), B)
>>
>> # (A)  I *think* __mro_entries__ gets called with the full tuple,
>> # instead of just the object it is found on.
>> # (B) I *think* it is called on the results of evaluating
>> # the terms within the tuple, instead of the original
>> # string representation.
>> _tmp = __mro_entries__(A, GenericAlias(NewList, int), B)
>>
>> # (C)  I *think* __mro_entries__ returns a replacement for
>> # just the single object, even though it was called on
>> # the whole tuple, without knowing which object it
>> # represents.
>> bases = (A, _tmp, B)
>>
>
> My understanding of the method signature:
>
> def __mro_entries__(self, orig_bases):
> ...
> return replacement_for_self
>
> My assumption as to the purpose of the extra complexity was:
>
> - given orig_bases, a method could avoid injecting bases already listed if
> it wanted to
> - allowing multiple items to be returned provides a way to
> programmatically combine mixins without having to define a new subclass for
> each combination
>
>

​Thanks, this might provide an answer to my question about multiple mro
entries here

https://mail.python.org/pipermail/python-ideas/2017-November/047897.html​

​––Koos​



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 562

[Koos Zevenhoven]

On Wed, Nov 15, 2017 at 8:02 PM, Ethan Furman  wrote:

> On 11/15/2017 04:55 AM, Koos Zevenhoven wrote:
>
>> On Tue, Nov 14, 2017 at 10:34 PM, Ivan Levkivskyi wrote:
>>
>
>
>> Rationale
>>> =
>>>
>>> [...] It would be convenient to simplify this
>>> procedure by recognizing ``__getattr__`` defined directly in a module
>>> that
>>> would act like a normal ``__getattr__`` method
>>>
>> >>
> >> [...]
> >>
>
>> Specification
>>> =
>>>
>>> >> The ``__getattr__`` function at the module level should accept one
> argument
>
>> which is the name of an attribute and return the computed value or raise
>>> an ``AttributeError``::
>>>
>>
>>def __getattr__(name: str) -> Any: ...
>>>
>>
>> This function will be called only if ``name`` is not found in the module
>>> through the normal attribute lookup.
>>>
>>
>> The Rationale (quoted in the beginning of this email) easily leaves a
>> different impression of this.​
>>
>
> I don't see how.  This is exactly the way normal __getattr__ works.
>
>
>
​Oh sorry, I think I put this email together too quickly. I was writing
down a bunch of thoughts I had earlier but hadn't written down.​ I think I
was mixing this up in my head with overriding __getitem__ for the module
namespace dict and __class_getitem__ from PEP 560, which only gets called
if the metaclass doesn't implement __getitem__ (IIRC).

But I did have another thought related to this. I was wondering whether the
lack of passing the module to the methods as `self` would harm future
attempts to generalize these ideas.

-- Koos

-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 560

[Terry Reedy]

On 11/14/2017 3:26 PM, Ivan Levkivskyi wrote:
After some discussion on python-ideas, see 
https://mail.python.org/pipermail/python-ideas/2017-September/047220.html, 
this PEP received positive comments. The updated version that takes into 
account the comments that appeared in the discussion so far is available 
at https://www.python.org/dev/peps/pep-0560/


Here I post the full text for convenience:

++

PEP: 560
Title: Core support for typing module and generic types
Author: Ivan Levkivskyi mailto:levkivs...@gmail.com>>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 03-Sep-2017
Python-Version: 3.7
Post-History: 09-Sep-2017

...

Suggested wording improvements:


Performance
---

The ``typing`` module is one of the heaviest and slowest modules in
the standard library even with all the optimizations made.



Mainly this is
because of subscripted generic types (see PEP 484 for definition of terms
used in this PEP) are class objects (see also [1]_).


Delete 'of' after 'because' to make this a proper sentence.


The three main ways how


"There are three ..." reads better to me.

the performance can be improved with the help of the proposed special 
methods:


- Creation of generic classes is slow since the ``GenericMeta.__new__`` is
   very slow; we will not need it anymore.

- Very long MROs for generic classes will be twice shorter;


I  believe by 'twice shorter', which is meaningless by itself, you mean 
'half as long'.  If so, please say the latter.

 > they are present

   because we duplicate the ``collections.abc`` inheritance chain
   in ``typing``.

- Time of instantiation of generic classes will be improved


Instantiation of generic classes will be faster.


   (this is minor however).


--
Terry Jan Reedy


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] module customization

[Ethan Furman]

So there are currently two ways to customize a module, with PEP 562 proposing a 
third.

The first method involves creating a standard class object, instantiating it, 
and replacing the sys.modules entry with it.

The second way is fairly similar, but instead of replacing the entire sys.modules entry, its class is updated to be the 
class just created -- something like sys.modules['mymod'].__class__ = MyNewClass .


My request:  Can someone write a better example of the second method?  And 
include __getattr__ ?

My question:  Does that __getattr__ method have 'self' as the first parameter?  If not, why not, and if so, shouldn't 
PEP 562's __getattr__ also take a 'self'?


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] module customization

[Guido van Rossum]

On Wed, Nov 15, 2017 at 4:27 PM, Ethan Furman  wrote:

> So there are currently two ways to customize a module, with PEP 562
> proposing a third.
>
> The first method involves creating a standard class object, instantiating
> it, and replacing the sys.modules entry with it.
>
> The second way is fairly similar, but instead of replacing the entire
> sys.modules entry, its class is updated to be the class just created --
> something like sys.modules['mymod'].__class__ = MyNewClass .
>
> My request:  Can someone write a better example of the second method?  And
> include __getattr__ ?
>
> My question:  Does that __getattr__ method have 'self' as the first
> parameter?


It does.


> If not, why not, and if so, shouldn't PEP 562's __getattr__ also take a
> 'self'?
>

Not really, since there's only one module (the one containing the
__getattr__ function). Plus we already have a 1-argument module-level
__getattr__ in mypy. See PEP 484.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python possible vulnerabilities in concurrency

[Armin Rigo]

Hi,

On 14 November 2017 at 14:55, Jan Claeys  wrote:
> Sounds like https://www.iso.org/standard/71094.html
> which is updating https://www.iso.org/standard/61457.html
> (which you can download from there if you search a bit; clearly either
> ISO doesn't have a UI/UX "standard" or they aren't following it...)

Just for completeness, I think that what you can download for free
from that second page only contains the first few sections ("Terms and
definitions").  It doesn't even go to "Purpose of this technical
report"---we need to pay $200 just to learn what the purpose is...

*Shrug*

Armin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python possible vulnerabilities in concurrency

[Guido van Rossum]

On Wed, Nov 15, 2017 at 6:37 PM, Armin Rigo  wrote:

> Hi,
>
> On 14 November 2017 at 14:55, Jan Claeys  wrote:
> > Sounds like https://www.iso.org/standard/71094.html
> > which is updating https://www.iso.org/standard/61457.html
> > (which you can download from there if you search a bit; clearly either
> > ISO doesn't have a UI/UX "standard" or they aren't following it...)
>
> Just for completeness, I think that what you can download for free
> from that second page only contains the first few sections ("Terms and
> definitions").  It doesn't even go to "Purpose of this technical
> report"---we need to pay $200 just to learn what the purpose is...
>
> *Shrug*
>

Actually it linked to
http://standards.iso.org/ittf/PubliclyAvailableStandards/index.html from
which I managed to download what looks like the complete
c061457_ISO_IEC_TR_24772_2013.pdf (336 pages) after clicking on an "I
accept" button (I didn't read what I accepted :-). The $200 is for the
printed copy I presume.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a developer mode to Python: -X dev command line option

[Victor Stinner]

Hi,

Since Brett and Nick like the idea and nobody complained against it, I
implemented the -X dev option:
https://bugs.python.org/issue32043
(Right now, it's a pull request.)

I removed the -b option.

Victor

2017-11-14 3:57 GMT+01:00 Nick Coghlan :
> On 14 November 2017 at 02:08, Victor Stinner  wrote:
>> My "-X dev" idea is not incompatible with Nick's PEP 565 "Show
>> DeprecationWarning in __main__" and it's different: it's an opt-in
>> option, while Nick wants to change the default behaviour.
>
> I'm +1 on a `-X dev` mode, since it enables a lot of things that are
> useful for making an application more robust (extension module
> debugging, explicit scope-controlled resource management) that I
> wouldn't want turned at the REPL by default. It also implicitly
> adjusts over time as we add more debugging capabilities.
>
> I don't consider it a replacement for tweaking how we handle
> DeprecationWarning by default, though :)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python possible vulnerabilities in concurrency

[Guido van Rossum]

On Wed, Nov 15, 2017 at 6:50 PM, Guido van Rossum  wrote:

> On Wed, Nov 15, 2017 at 6:37 PM, Armin Rigo  wrote:
>
>> Hi,
>>
>> On 14 November 2017 at 14:55, Jan Claeys  wrote:
>> > Sounds like https://www.iso.org/standard/71094.html
>> > which is updating https://www.iso.org/standard/61457.html
>> > (which you can download from there if you search a bit; clearly either
>> > ISO doesn't have a UI/UX "standard" or they aren't following it...)
>>
>> Just for completeness, I think that what you can download for free
>> from that second page only contains the first few sections ("Terms and
>> definitions").  It doesn't even go to "Purpose of this technical
>> report"---we need to pay $200 just to learn what the purpose is...
>>
>> *Shrug*
>>
>
> Actually it linked to http://standards.iso.org/ittf/
> PubliclyAvailableStandards/index.html from which I managed to download
> what looks like the complete c061457_ISO_IEC_TR_24772_2013.pdf (336
> pages) after clicking on an "I accept" button (I didn't read what I
> accepted :-). The $200 is for the printed copy I presume.
>

So far I learned one thing from the report. They use the term
"vulnerabilities" liberally, defining it essentially as "bug":

All programming languages contain constructs that are incompletely
> specified, exhibit undefined behaviour, are implementation-dependent, or
> are difficult to use correctly. The use of those constructs may therefore
> give rise to *vulnerabilities*, as a result of which, software programs
> can execute differently than intended by the writer.
>

They then go on to explain that sometimes vulnerabilities can be exploited,
but I object to calling all bugs vulnerabilities -- that's just using a
scary word to get attention for a sleep-inducing document containing such
gems as "Use floating-point arithmetic only when absolutely needed" (page
230).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python possible vulnerabilities in concurrency

[Wes Turner]

CWE (Common Weakness Enumeration) has numbers (and URLs) and a graph model,
and code examples, and mitigations for bugs, vulnerabilities, faults,
design flaws, weaknesses.
https://cwe.mitre.org/

Research Concepts
https://cwe.mitre.org/data/definitions/1000.html

Development Concepts
https://cwe.mitre.org/data/definitions/699.html

CWE CATEGORY: Time and State
https://cwe.mitre.org/data/definitions/361.html

  CWE CATEGORY: Concurrency Issues
  https://cwe.mitre.org/data/definitions/557.html

17. Concurrent Execution
https://docs.python.org/3/library/concurrency.html


> 1. activating threads, tasks or pico-threads

https://docs.python.org/3/library/threading.html#threading.Thread.start
https://docs.python.org/3/library/threading.html#threading.Thread.run

> 2. Directed termination of threads, tasks or pico-threads

So, I looked this up:
https://stackoverflow.com/questions/323972/is-there-any-way-to-kill-a-thread-in-python

Do asynchronous programming patterns actually make this basically never
necessary? (asyncio coroutines, greenlet (eventlet, gevent, ), twisted)

https://docs.python.org/3/library/asyncio.html

https://docs.python.org/3/library/asyncio-task.html

If you really feel like you need the overhead of threads instead of or in
addition to coroutines (they won't use multiple cores without going to IPC,
anyway), you can.

> 3. Premature termination of threads, tasks or pico-threads

What is this referring to?
Does it release handles and locks on exception? (try/finally?)


> 4. Concurrent access to data shared between threads, tasks or
pico-threads,   and

CWE-362: Concurrent Execution using Shared Resource with Improper
Synchronization ('Race Condition')
https://cwe.mitre.org/data/definitions/362.html

CWE-567: Unsynchronized Access to Shared Data in a Multithreaded Context
https://cwe.mitre.org/data/definitions/567.html


> 5. Lock protocol errors for concurrent entities

CWE-667: Improper Locking
https://cwe.mitre.org/data/definitions/667.html

CWE-366: Race Condition within a Thread
https://cwe.mitre.org/data/definitions/366.html

The ``mutex`` module is removed in Python 3:
https://docs.python.org/2/library/mutex.html

17.1. threading — Thread-based parallelism
https://docs.python.org/3/library/threading.html


... Are there other good resources (in addition to Chapter 17) for
concurrency in CPython and/or PyPy and/or Stackless Python, MicroPython,
IronPython, Jython?

- [ ] How do we add Python to the excellent CWE reference?

- How can/could/should one add the things with labels (*) from the ISO PDF
you speak of to thr CWE graph? (* schema:name, rdfs:label)

On Wednesday, November 15, 2017, Guido van Rossum  wrote:

> On Wed, Nov 15, 2017 at 6:50 PM, Guido van Rossum  > wrote:
>
>> On Wed, Nov 15, 2017 at 6:37 PM, Armin Rigo > > wrote:
>>
>>> Hi,
>>>
>>> On 14 November 2017 at 14:55, Jan Claeys >> > wrote:
>>> > Sounds like https://www.iso.org/standard/71094.html
>>> > which is updating https://www.iso.org/standard/61457.html
>>> > (which you can download from there if you search a bit; clearly either
>>> > ISO doesn't have a UI/UX "standard" or they aren't following it...)
>>>
>>> Just for completeness, I think that what you can download for free
>>> from that second page only contains the first few sections ("Terms and
>>> definitions").  It doesn't even go to "Purpose of this technical
>>> report"---we need to pay $200 just to learn what the purpose is...
>>>
>>> *Shrug*
>>>
>>
>> Actually it linked to http://standards.iso.org/ittf/
>> PubliclyAvailableStandards/index.html from which I managed to download
>> what looks like the complete c061457_ISO_IEC_TR_24772_2013.pdf (336
>> pages) after clicking on an "I accept" button (I didn't read what I
>> accepted :-). The $200 is for the printed copy I presume.
>>
>
> So far I learned one thing from the report. They use the term
> "vulnerabilities" liberally, defining it essentially as "bug":
>
> All programming languages contain constructs that are incompletely
>> specified, exhibit undefined behaviour, are implementation-dependent, or
>> are difficult to use correctly. The use of those constructs may therefore
>> give rise to *vulnerabilities*, as a result of which, software programs
>> can execute differently than intended by the writer.
>>
>
> They then go on to explain that sometimes vulnerabilities can be
> exploited, but I object to calling all bugs vulnerabilities -- that's just
> using a scary word to get attention for a sleep-inducing document
> containing such gems as "Use floating-point arithmetic only when absolutely
> needed" (page 230).
>
> --
> --Guido van Rossum (python.org/~guido)
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] module customization

[Greg Ewing]

Ethan Furman wrote:
The second way is fairly similar, but instead of replacing the entire 
sys.modules entry, its class is updated to be the class just created -- 
something like sys.modules['mymod'].__class__ = MyNewClass .


If the recent suggestion to replace the global namespace
dict with the module object goes ahead, maybe it would enable
using this idiom to reassign the module class:

   class __class__(__class__):
  # module methods here

Badger-badger-badger-ly,
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] module customization

[Nathaniel Smith]

On Wed, Nov 15, 2017 at 4:27 PM, Ethan Furman  wrote:
> The second way is fairly similar, but instead of replacing the entire
> sys.modules entry, its class is updated to be the class just created --
> something like sys.modules['mymod'].__class__ = MyNewClass .
>
> My request:  Can someone write a better example of the second method?  And
> include __getattr__ ?

Here's a fairly straightforward example:

https://github.com/python-trio/trio/blob/master/trio/_deprecate.py#L114-L140

(Intentionally doesn't include __dir__ because I didn't want
deprecated attributes to show up in tab completion. For other use
cases like lazy imports, you would implement __dir__ too.)

Example usage:

https://github.com/python-trio/trio/blob/master/trio/__init__.py#L66-L98

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] module customization

[Nathaniel Smith]

On Wed, Nov 15, 2017 at 5:49 PM, Guido van Rossum  wrote:
>> If not, why not, and if so, shouldn't PEP 562's __getattr__ also take a
>> 'self'?
>
> Not really, since there's only one module (the one containing the
> __getattr__ function). Plus we already have a 1-argument module-level
> __getattr__ in mypy. See PEP 484.

I guess the benefit of taking 'self' would be that it would make it
possible (though still a bit odd-looking) to have reusable __getattr__
implementations, like:

# mymodule.py
from auto_importer import __getattr__, __dir__

auto_import_modules = {"foo", "bar"}

# auto_importer.py
def __getattr__(self, name):
if name in self.auto_import_modules:
...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] module customization

[Nathaniel Smith]

On Wed, Nov 15, 2017 at 10:14 PM, Nathaniel Smith  wrote:
> On Wed, Nov 15, 2017 at 4:27 PM, Ethan Furman  wrote:
>> The second way is fairly similar, but instead of replacing the entire
>> sys.modules entry, its class is updated to be the class just created --
>> something like sys.modules['mymod'].__class__ = MyNewClass .
>>
>> My request:  Can someone write a better example of the second method?  And
>> include __getattr__ ?

Doh, I forgot to permalinkify those. Better links for anyone reading
this in the future:

> Here's a fairly straightforward example:
>
> https://github.com/python-trio/trio/blob/master/trio/_deprecate.py#L114-L140

https://github.com/python-trio/trio/blob/3edfafeedef4071646a9015e28be01f83dc02f94/trio/_deprecate.py#L114-L140

> (Intentionally doesn't include __dir__ because I didn't want
> deprecated attributes to show up in tab completion. For other use
> cases like lazy imports, you would implement __dir__ too.)
>
> Example usage:
>
> https://github.com/python-trio/trio/blob/master/trio/__init__.py#L66-L98

https://github.com/python-trio/trio/blob/3edfafeedef4071646a9015e28be01f83dc02f94/trio/__init__.py#L66-L98

> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 560: bases classes / confusion

[Nick Coghlan]

On 16 November 2017 at 04:39, Ivan Levkivskyi  wrote:

> Nick is exactly right here. Jim, if you want to propose alternative
> wording, then we could consider it.
>

Jim also raised an important point that needs clarification at the spec
level: given multiple entries in "orig_bases" with __mro_entries__ methods,
do all such methods get passed the *same* orig_bases tuple? Or do they
receive partially resolved ones, such that bases listed before them have
already been resolved to their MRO entries by the time they run.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com