[Python-Dev] PEP 561: Distributing and Packaging Type Information

[Ethan Smith]

Hello all,

I have completed an implementation for PEP 561, and believe it is time to
share the PEP and implementation with python-dev

Python-ideas threads:
 * PEP 561: Distributing and Packaging Type Information
 * PEP 561 v2 - Packaging Static Type Information
 * PEP 561: Distributing Type Information V3

The live version is here: https://www.python.org/dev/peps/pep-0561/

As always, duplicated below.

Ethan Smith

---

PEP: 561
Title: Distributing and Packaging Type Information
Author: Ethan Smith 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Sep-2017
Python-Version: 3.7
Post-History:


Abstract


PEP 484 introduced type hinting to Python, with goals of making typing
gradual and easy to adopt. Currently, typing information must be distributed
manually. This PEP provides a standardized means to package and distribute
type information and an ordering for type checkers to resolve modules and
collect this information for type checking using existing packaging
architecture.


Rationale
=

Currently, package authors wish to distribute code that has
inline type information. However, there is no standard method to distribute
packages with inline type annotations or syntax that can simultaneously
be used at runtime and in type checking. Additionally, if one wished to
ship typing information privately the only method would be via setting
``MYPYPATH`` or the equivalent to manually point to stubs. If the package
can be released publicly, it can be added to typeshed [1]_. However, this
does not scale and becomes a burden on the maintainers of typeshed.
Additionally, it ties bugfixes to releases of the tool using typeshed.

PEP 484 has a brief section on distributing typing information. In this
section [2]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for
shipping stub files. However, manually adding a path to stub files for each
third party library does not scale. The simplest approach people have taken
is to add ``site-packages`` to their ``MYPYPATH``, but this causes type
checkers to fail on packages that are highly dynamic (e.g. sqlalchemy
and Django).


Specification
=

There are several motivations and methods of supporting typing in a package.
This PEP recognizes three (3) types of packages that may be created:

1. The package maintainer would like to add type information inline.

2. The package maintainer would like to add type information via stubs.

3. A third party would like to share stub files for a package, but the
   maintainer does not want to include them in the source of the package.

This PEP aims to support these scenarios and make them simple to add to
packaging and deployment.

The two major parts of this specification are the packaging specifications
and the resolution order for resolving module type information. The packaging
spec is based on and extends PEP 345 metadata. The type checking spec is
meant to replace the ``shared/typehints/pythonX.Y/`` spec of PEP 484 [2]_.

New third party stub libraries are encouraged to distribute stubs via the
third party packaging proposed in this PEP in place of being added to
typeshed. Typeshed will remain in use, but if maintainers are found, third
party stubs in typeshed are encouraged to be split into their own package.

Packaging Type Information
--
In order to make packaging and distributing type information as simple and
easy as possible, the distribution of type information, and typed Python code
is done through existing packaging frameworks. This PEP adds a new item to the
``*.distinfo/METADATA`` file to contain metadata about a package's support for
typing. The new item is optional, but must have a name of ``Typed`` and have a
value of either ``inline`` or ``stubs``, if present.

Metadata Examples::

Typed: inline
Typed: stubs


Stub Only Packages
''''''''''''''''''

For package maintainers wishing to ship stub files containing all of their
type information, it is prefered that the ``*.pyi`` stubs are alongside the
corresponding ``*.py`` files. However, the stubs may be put in a sub-folder
of the Python sources, with the same name the ``*.py`` files are in. For
example, the ``flyingcircus`` package would have its stubs in the folder
``flyingcircus/flyingcircus/``. This path is chosen so that if stubs are
not found in ``flyingcircus/`` the type checker may treat the subdirectory as
a normal package. The normal resolution order of checking ``*.pyi`` before
``*.py`` will be maintained.

Third Party Stub Packages
'''''''''''''''''''''''''

Third parties seeking to distribute stub files are encouraged to contact the
maintainer of the package about distribution alongside the package

Re: [Python-Dev] PEP 561: Distributing and Packaging Type Information

[Ethan Smith]

On Thu, Oct 26, 2017 at 4:48 PM, Mariatta Wijaya 
wrote:

> Not sure if postings to python-ideas count,
>
>
> PEP 1 says:
>
> Post-History is used to record the dates of when new versions of the PEP
> are posted to python-list and/or python-dev.
>
> So, no ?
>


Reading PEP 12, https://www.python.org/dev/peps/pep-0012/#id24


   -

   Leave Post-History alone for now; you'll add dates to this header each
   time you post your PEP to python-l...@python.org or python-dev@python.org.
   If you posted your PEP to the lists on August 14, 2001 and September 3,
   2001, the Post-History header would look like:

   Post-History: 14-Aug-2001, 03-Sept-2001

   You must manually add new dates and check them in. If you don't have
   check-in privileges, send your changes to the PEP editors.

Perhaps it is outdated and needs to have python-ideas added? python-ideas
was created around 2006 (according to the archives), so after PEP 1/12 were
written.


> This PEP adds a new item to the
> ``*.distinfo/METADATA`` file

   *.dist-info/

Thank you for catching that. I will fix that with my next round of edits.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 561: Distributing and Packaging Type Information

[Ethan Smith]

On Fri, Oct 27, 2017 at 12:44 AM, Nathaniel Smith  wrote:

> On Thu, Oct 26, 2017 at 3:42 PM, Ethan Smith  wrote:
> > However, the stubs may be put in a sub-folder
> > of the Python sources, with the same name the ``*.py`` files are in. For
> > example, the ``flyingcircus`` package would have its stubs in the folder
> > ``flyingcircus/flyingcircus/``. This path is chosen so that if stubs are
> > not found in ``flyingcircus/`` the type checker may treat the
> subdirectory as
> > a normal package.
>
> I admit that I find this aesthetically unpleasant. Wouldn't something
> like __typestubs__/ be a more Pythonic name? (And also avoid potential
> name clashes, e.g. my async_generator package has a top-level export
> called async_generator; normally you do 'from async_generator import
> async_generator'. I think that might cause problems if I created an
> async_generator/async_generator/ directory, especially post-PEP 420.)
>

I agree, this is unpleasant, I am now of the thought that if maintainers do
not wish to ship stubs alongside their Python code, they should just create
separate stub-only packages. I don't think there is a particular need to
special case this for minor convenience.



> > Type Checker Module Resolution Order
> > 
> >
> > The following is the order that type checkers supporting this PEP should
> > resolve modules containing type information:
> >
> > 1. User code - the files the type checker is running on.
> >
> > 2. Stubs or Python source manually put in the beginning of the path. Type
> >checkers should provide this to allow the user complete control of
> which
> >stubs to use, and patch broken stubs/inline types from packages.
> >
> > 3. Third party stub packages - these packages can supersede the installed
> >untyped packages. They can be found at ``pkg-stubs`` for package
> ``pkg``,
> >however it is encouraged to check the package's metadata using
> packaging
> >query APIs such as ``pkg_resources`` to assure that the package is
> meant
> >for type checking, and is compatible with the installed version.
>
> Am I right that this means you need to be able to map from import
> names to distribution names? I.e., if you see 'import foo', you need
> to figure out which *.dist-info directory contains metadata for the
> 'foo' package? How do you plan to do this?
>
>
The problem is that technically, import names and distribution names
> are totally unrelated namespaces -- for example, the '_pytest' package
> comes from the 'pytest' distribution, the 'pylab' package comes from
> 'matplotlib', and 'pip install scikit-learn' gives you a package
> imported as 'sklearn'. Namespace packages are also challenging,
> because a single top-level package might actually be spread across
> multiple distributions.
>
>
This is a problem. What I now realize is that the typing metadata is needed
for *packages* and not distributions. I will work on a new proposal that
makes the metadata per-package. It will require a slightly more complicated
proposal, but I feel that it is necessary. Thank you for realizing this
issue with my proposal, I probably should have caught it earlier.

-n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Ethan Smith]

> Beyond the API already proposed in PEP 557, this would mean adding:
>
> * dataclasses.ClassVar (as proposed above)
> * dataclasses.Any (probably just set to the literal string
> "dataclasses.Any")
> * dataclasses.NamedTuple (as a replacement for typing.NamedTuple)
> * potentially dataclasses.is_class_var (instead of dataclasses
> implicitly snooping in sys.modules looking for a "typing" module)
>
> In effect, where we now have "typing" and "typing_extensions", we'd
> instead have "dataclasses" in the standard library (with the subset of
> annotations needed to define data record classes), and "typing" as an
> independently versioned module on PyPI.
>
>
I'm not so keen on this because I think some things in typing (such as
NamedTuple) probably deserve to be in the collections module. And some of
the ABCs could probably also be merged with collections.abc but doing this
correctly and not all at once would be quite difficult.
I do think the typing concepts should be better integrated into the
standard library. However, a fair amount of the clases you list (such as
NamedTuple) are in of themselves dependent on parts of typing.

Cheers,
Ethan

I think such an approach could address a few problems:
>
> - the useful-at-runtime concepts (in dataclasses) would be clearly
> separated from the static-type-analysis enablers (in typing)
> - type hints would still have a clear presence in the regular standard
> library, but the more complex parts would be opt-in (similar to
> statistics vs SciPy, secrets vs cryptography, etc)
> - as various concepts in typing matured and became genuinely stable,
> they could potentially "graduate" into the standard library's
> dataclasses module
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> ethan%40ethanhs.me
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 561 rework

[Ethan Smith]

Hello,

I re-wrote my PEP to have typing opt-in be per-package rather than
per-distribution. This greatly simplifies things, and thanks to the
feedback and suggestions of Nick Coghlan, it is entirely compatible with
older packaging tooling.

The main idea is there are two types of packages:
 - types are packaged with runtime code (inline or stubs in the same
package)
 - types are in a separate package (a third party or maintainer wants to
ship type information, but not with runtime code).

The PEP is live on python.org: https://www.python.org/dev/peps/pep-0561/

And as always, duplicated below.

Cheers,

Ethan Smith

---

PEP: 561
Title: Distributing and Packaging Type Information
Author: Ethan Smith 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Sep-2017
Python-Version: 3.7
Post-History: 10-Sep-2017, 12-Sep-2017, 06-Oct-2017, 26-Oct-2017


Abstract


PEP 484 introduced type hinting to Python, with goals of making typing
gradual and easy to adopt. Currently, typing information must be distributed
manually. This PEP provides a standardized means to leverage existing tooling
to package and distribute type information with minimal work and an ordering
for type checkers to resolve modules and collect this information for type
checking.


Rationale
=

Currently, package authors wish to distribute code that has inline type
information. Additionally, maintainers would like to distribute stub files
to keep Python 2 compatibility while using newer annotation syntax. However,
there is no standard method to distribute packages with type information.
Also, if one wished to ship stub files privately the only method available
would be via setting ``MYPYPATH`` or the equivalent to manually point to
stubs. If the package can be released publicly, it can be added to
typeshed [1]_. However, this does not scale and becomes a burden on the
maintainers of typeshed. In addition, it ties bug fixes in stubs to releases
of the tool using typeshed.

PEP 484 has a brief section on distributing typing information. In this
section [2]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for
shipping stub files. However, manually adding a path to stub files for each
third party library does not scale. The simplest approach people have taken
is to add ``site-packages`` to their ``MYPYPATH``, but this causes type
checkers to fail on packages that are highly dynamic (e.g. sqlalchemy
and Django).


Definition of Terms
===

The definition of "MAY", "MUST", and "SHOULD", and "SHOULD NOT" are
to be interpreted as described in RFC 2119.

"inline" - the types are part of the runtime code using PEP 526 and 3107
syntax.

"stubs" - files containing only type information, empty of runtime code.

"Distributions" are the packaged files which are used to publish and distribute
a release. [3]_

"Module" a file containing Python runtime code or stubbed type information.

"Package" a directory or directories that namespace Python modules.


Specification
=

There are several motivations and methods of supporting typing in a package.
This PEP recognizes three (3) types of packages that users of typing wish to
create:

1. The package maintainer would like to add type information inline.

2. The package maintainer would like to add type information via stubs.

3. A third party or package maintainer would like to share stub files for
   a package, but the maintainer does not want to include them in the source
   of the package.

This PEP aims to support these scenarios and make them simple to add to
packaging and deployment.

The two major parts of this specification are the packaging specifications
and the resolution order for resolving module type information. The type
checking spec is meant to replace the ``shared/typehints/pythonX.Y/`` spec
of PEP 484 [2]_.

New third party stub libraries SHOULD distribute stubs via the third party
packaging methods proposed in this PEP in place of being added to typeshed.
Typeshed will remain in use, but if maintainers are found, third party stubs
in typeshed MAY be split into their own package.


Packaging Type Information
--

In order to make packaging and distributing type information as simple and
easy as possible, packaging and distribution is done through existing
frameworks.

Package maintainers who wish to support type checking of their code MUST add
a ``py.typed`` file to their package supporting typing. This marker is
recursive, if a top-level package includes it, all sub-packages MUST support
type checking as well. To have this file installed with the package,
maintainers can use existing packaging options such as ``package_data`` in
distutils, shown below.

Distutils option example::

...
package_data = {
'pkg': ['py.typed'],
},

Re: [Python-Dev] PEP 561 rework

[Ethan Smith]

On Sun, Nov 12, 2017 at 9:53 AM, Jelle Zijlstra 
wrote:

>
>
> 2017-11-12 3:40 GMT-08:00 Ethan Smith :
>
>> Hello,
>>
>> I re-wrote my PEP to have typing opt-in be per-package rather than
>> per-distribution. This greatly simplifies things, and thanks to the
>> feedback and suggestions of Nick Coghlan, it is entirely compatible with
>> older packaging tooling.
>>
>> The main idea is there are two types of packages:
>>  - types are packaged with runtime code (inline or stubs in the same
>> package)
>>  - types are in a separate package (a third party or maintainer wants to
>> ship type information, but not with runtime code).
>>
>> The PEP is live on python.org: https://www.python.org/dev/peps/pep-0561/
>>
>> And as always, duplicated below.
>>
>> Cheers,
>>
>> Ethan Smith
>>
>> ---
>>
>> PEP: 561
>> Title: Distributing and Packaging Type Information
>> Author: Ethan Smith 
>> Status: Draft
>> Type: Standards Track
>> Content-Type: text/x-rst
>> Created: 09-Sep-2017
>> Python-Version: 3.7
>> Post-History: 10-Sep-2017, 12-Sep-2017, 06-Oct-2017, 26-Oct-2017
>>
>>
>> Abstract
>> 
>>
>> PEP 484 introduced type hinting to Python, with goals of making typing
>> gradual and easy to adopt. Currently, typing information must be distributed
>> manually. This PEP provides a standardized means to leverage existing tooling
>> to package and distribute type information with minimal work and an ordering
>> for type checkers to resolve modules and collect this information for type
>> checking.
>>
>>
>> Rationale
>> =
>>
>> Currently, package authors wish to distribute code that has inline type
>> information. Additionally, maintainers would like to distribute stub files
>> to keep Python 2 compatibility while using newer annotation syntax. However,
>> there is no standard method to distribute packages with type information.
>> Also, if one wished to ship stub files privately the only method available
>> would be via setting ``MYPYPATH`` or the equivalent to manually point to
>> stubs. If the package can be released publicly, it can be added to
>> typeshed [1]_. However, this does not scale and becomes a burden on the
>> maintainers of typeshed. In addition, it ties bug fixes in stubs to releases
>> of the tool using typeshed.
>>
>> PEP 484 has a brief section on distributing typing information. In this
>> section [2]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for
>> shipping stub files. However, manually adding a path to stub files for each
>> third party library does not scale. The simplest approach people have taken
>> is to add ``site-packages`` to their ``MYPYPATH``, but this causes type
>> checkers to fail on packages that are highly dynamic (e.g. sqlalchemy
>> and Django).
>>
>>
>> Definition of Terms
>> ===
>>
>> The definition of "MAY", "MUST", and "SHOULD", and "SHOULD NOT" are
>> to be interpreted as described in RFC 2119.
>>
>> "inline" - the types are part of the runtime code using PEP 526 and 3107
>> syntax.
>>
>> "stubs" - files containing only type information, empty of runtime code.
>>
>> "Distributions" are the packaged files which are used to publish and 
>> distribute
>> a release. [3]_
>>
>> "Module" a file containing Python runtime code or stubbed type information.
>>
>> "Package" a directory or directories that namespace Python modules.
>>
>>
>> Specification
>> =
>>
>> There are several motivations and methods of supporting typing in a package.
>> This PEP recognizes three (3) types of packages that users of typing wish to
>> create:
>>
>> 1. The package maintainer would like to add type information inline.
>>
>> 2. The package maintainer would like to add type information via stubs.
>>
>> 3. A third party or package maintainer would like to share stub files for
>>a package, but the maintainer does not want to include them in the source
>>of the package.
>>
>> This PEP aims to support these scenarios and make them simple to add to
>> packaging and deployment.
>>
>> The two major parts of this specification are the packaging specifications
>> and the resolution order for resolving module type information. The type
>> checking spec is meant

Re: [Python-Dev] PEP 561 rework

[Ethan Smith]

On Sun, Nov 12, 2017 at 8:07 PM, Nathaniel Smith  wrote:

> On Sun, Nov 12, 2017 at 11:21 AM, Ethan Smith  wrote:
> >
> >
> > On Sun, Nov 12, 2017 at 9:53 AM, Jelle Zijlstra <
> jelle.zijls...@gmail.com>
> > wrote:
> >>
> >> 2017-11-12 3:40 GMT-08:00 Ethan Smith :
> >>> The name of the stub
> >>> package
> >>> MUST follow the scheme ``pkg_stubs`` for type stubs for the package
> named
> >>> ``pkg``. The normal resolution order of checking ``*.pyi`` before
> >>> ``*.py``
> >>> will be maintained.
> >>
> >> This is very minor, but what do you think of using "pkg-stubs" instead
> of
> >> "pkg_stubs" (using a hyphen rather than an underscore)? This would make
> the
> >> name illegal to import as a normal Python package, which makes it clear
> that
> >> it's not a normal package. Also, there could be real packages named
> >> "_stubs".
> >
> > I suppose this makes sense. I checked PyPI and as of a few weeks ago
> there
> > were no packages with the name pattern, but I like the idea of making it
> > explicitly non-runtime importable. I cannot think of any reason not to do
> > it, and the avoidance of confusion about the package being importable is
> a
> > benefit. I will make the change with my next round of edits.
>
> PyPI doesn't distinguish between the names 'foo-stubs' and 'foo_stubs'
> -- they get normalized together. So even if you use 'foo-stubs' as the
> directory name on sys.path to avoid collisions at import time, it
> still won't allow someone to distribute a separate 'foo_stubs' package
> on PyPI.
>
> If you do go with a fixed naming convention like this, the PEP should
> probably also instruct the PyPI maintainers that whoever owns 'foo'
> automatically has the right to control the name 'foo-stubs' as well.
> Or maybe some tweak to PEP 541 is needed.
>

As I understand it however, the distribution name need not map to to the
package name in any way. So regardless of if foo-stubs is seen as
foo_stubs, I could name the distribution Albatross if I wished, and install
the foo-stubs package into site/dist-packages, and it would work. Also I'm
not sure if the PyPI change would require an edict from a PEP, but if so, I
wouldn't be opposed to the idea, I think it would be nice to default the
stub packages to the owners of the normal packages (people should, to my
understanding, be able to make alternate distributions without hassle).

Ethan

>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 561 rework

[Ethan Smith]

On Mon, Nov 13, 2017 at 3:50 PM, Sebastian Rittau 
wrote:

> Hello everyone,
>
>
> Am 14.11.2017 um 00:29 schrieb Guido van Rossum:
>
>> This is a nice piece of work. I expect to accept it pretty much verbatim
>> (with some small edits, see https://github.com/python/peps/pull/467). I
>> agree with Nick that we don't have to do anything specifically about
>> control of foo_stubs packages -- nor do I think we need to worry about
>> foo_stubs vs. foo-stubs.
>>
>> Everyone else: if you think this should not go through, now's the time to
>> reply-all here!
>>
> I am really looking forward to the implementation of this PEP and I am
> glad that it is close to acceptance. One thing that is not really clear to
> me is how module-only packages are handled. Say I have a package "foo" that
> installs the file "foo.py" to site-packages, where would I install
> "foo.pyi" and py.typed to? Or is this case not supported and I have to
> convert the foo module into a package containing just __init__.py?
>
> The PEP as of right now provides no support for module only distributions.
I don't think it would be possible to support inline typing in module only
distributions, as a random mymod.py in site/dist-packages is hard to trust.
Refactoring into a package is probably the best option. Alternatively, I
believe a mymod_stubs with just an __init__.pyi might work as a work around
as well.

Ethan


>  - Sebastian
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ethan%
> 40ethanhs.me
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 561 rework

[Ethan Smith]

A note was added [1] about the solution for module only distributions and
is live on Python.org.

[1] https://github.com/python/peps/pull/468

Ethan Smith

On Tue, Nov 14, 2017 at 1:02 AM, Sebastian Rittau 
wrote:

> Am 14.11.2017 um 02:38 schrieb Guido van Rossum:
>
> On Mon, Nov 13, 2017 at 3:50 PM, Sebastian Rittau 
> wrote:
>
>>
>
>> I am really looking forward to the implementation of this PEP and I am
>> glad that it is close to acceptance. One thing that is not really clear to
>> me is how module-only packages are handled. Say I have a package "foo" that
>> installs the file "foo.py" to site-packages, where would I install
>> "foo.pyi" and py.typed to? Or is this case not supported and I have to
>> convert the foo module into a package containing just __init__.py?
>>
>
> Good call. I think that conversion to a package is indeed the best
> approach -- it doesn't seem worth it to add more special-casing for this
> scenario.
>
> Ethan, if you agree, you should just add a sentence about this to the PEP.
>
> This seems like the best solution, especially since setuptools does not
> really support installing package data for pure modules.
>
>  - Sebastian
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> ethan%40ethanhs.me
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Supporting functools.singledispatch with classes.

[Ethan Smith]

Hello all,

In https://bugs.python.org/issue32380 I was hoping to add support for
singledispatch with methods. Unfortunately, this cannot be achieved
internally without ugly attribute or stack hacks.

Therefore, I was thinking it would be nice if singledispatch supported a
keyword argument of the argument index to dispatch on, thus one can say:

class A:
@singledispatch(arg=1)
def method(self, a):
return 'base'
@method.register(int)
def method(self, a):
return 'int'

The other option that could work is to define a special decorator for
methods

def methodsingledispatch(func):
"""Single-dispatch generic method decorator."""
wrapped = singledispatch(func)
def wrapper(*args, **kw):
return wrapped.dispatch(args[1].__class__)(*args, **kw)
wrapper.register = wrapped.register
update_wrapper(wrapper, func)
return wrapper

Since this is an API change, Ivan said I should post here to get feedback.

I prefer the first design, as it is more generic and flexible.

There is also the issue of classmethod and staticmethod. Since these are
descriptors, I'm not sure they will work with singledispatch at all.

if you do

@classmethod
@singledispatch
def foo(cls, arg): ...

You lose register on foo, breaking everything. I believe this would require
changing classmethod thus is a non-starter.

If you do

@singledispatch
@classmethod
def foo(arg): ...

The wrapper in singledispatch needs to be called as the __func__ in
classmethod, but __func__ is readonly.

So at the moment, I don't think it is possible to implement singledispatch
on classmethod or staticmethod decorated functions.

I look forward to people's thoughts on these issues.

Cheers,

Ethan Smith
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supporting functools.singledispatch with classes.

[Ethan Smith]

Okay, if there is no further feedback, I will work on a
singledispatchmethod decorator like partialmethod.

For the future perhaps, would it not be possible to tell that the passed
argument is a descriptor/function and dispatch to the correct
implementation, thus not needing two functions for essentially the same
thing?

It seems more straightforward to make the implementation a bit more complex
to provide a single, simple API to users.

Cheers,

Ethan

On Tue, Dec 26, 2017 at 3:29 PM, Ivan Levkivskyi 
wrote:

> On 26 December 2017 at 01:41, Nick Coghlan  wrote:
>
>> On 25 December 2017 at 12:32, Ethan Smith  wrote:
>> > So at the moment, I don't think it is possible to implement
>> singledispatch
>> > on classmethod or staticmethod decorated functions.
>>
>> I've posted this to the PR, but adding it here as well: I think this
>> is a situation very similar to the case with functools.partialmethod,
>> where you're going to need to write a separate
>> functools.singledispatchmethod class that's aware of the descriptor
>> protocol, rather than trying to add the functionality directly to
>> functools.singledispatch.
>>
>
> I agree with Nick here. Adding a separate decorator looks like the right
> approach,
> especially taking into account the precedent of @partialmethod.
>
> --
> Ivan
>
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Concerns about method overriding and subclassing with dataclasses

[Ethan Smith]

Hello all,

I've recently been experimenting with dataclasses. They totally rock! A lot
of the boilerplate for the AST I've designed in Python is automatically
taken care of, it's really great! However, I have a few concerns about the
implementation.

In a few cases I want to override the repr of the AST nodes. I wrote a
__repr__ and ran the code but lo and behold I got a type error. I couldn't
override it. I quickly learned that one needs to pass a keyword to the
dataclass decorator to tell it *not* to auto generate methods you override.

I have two usability concerns with the current implementation. I emailed
Eric about the first, and he said I should ask for thoughts here. The
second I found after a couple of days sitting on this message.

The first is that needing both a keyword and method is duplicative and
unnecessary. Eric agreed it was a hassle, but felt it was justified
considering someone may accidentally override a dataclass method. I
disagree with this point of view as dataclasses are billed as providing
automatic methods. Overriding via method definition is very natural and
idiomatic. I don't really see how someone could accidentally override a
dataclass method if methods were not generated by the dataclass decorator
that are already defined in the class at definition time.

The second concern, which I came across more recently, is if I have a base
class, and dataclasses inherit from this base class, inherited __repr__ &
co are silently overridden by dataclass. This is both unexpected, and also
means I need to pass a repr=False to each subclass' decorator to get
correct behavior, which somewhat defeats the utility of subclassing. Im not
as sure a whole lot can be done about this though.


I appreciate any thoughts folks have related to this.


Cheers,
~>Ethan Smith
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supporting functools.singledispatch with classes.

[Ethan Smith]

On Fri, Dec 29, 2017 at 7:02 AM, Nick Coghlan  wrote:

> On 28 December 2017 at 04:22, Ethan Smith  wrote:
> > Okay, if there is no further feedback, I will work on a
> singledispatchmethod
> > decorator like partialmethod.
> >
> > For the future perhaps, would it not be possible to tell that the passed
> > argument is a descriptor/function and dispatch to the correct
> > implementation, thus not needing two functions for essentially the same
> > thing?
> >
> > It seems more straightforward to make the implementation a bit more
> complex
> > to provide a single, simple API to users.
>
> "Add 'method' to the decorator name when decorating a method" is a
> pretty simple rule to remember - it's much easier than "Add
> 'arg_index=1'" (which is a comparatively arbitrary adjustment that
> requires a fairly in depth understanding of both the descriptor
> protocol and type-based function dispatch to follow).
>
> And you need the change to be explicitly opt-in *somehow*, in order to
> avoid breaking any existing code that relies on methods decorated with
> "singledispatch" dispatching on the bound class.
>

Good points. I will start working on the singledispatchmethod
implementation.

~>Ethan Smith

>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses

[Ethan Smith]

On Fri, Dec 29, 2017 at 2:45 AM, Antoine Pitrou  wrote:

> On Fri, 29 Dec 2017 02:23:56 -0800
> Ethan Smith  wrote:
> >
> > In a few cases I want to override the repr of the AST nodes. I wrote a
> > __repr__ and ran the code but lo and behold I got a type error. I
> couldn't
> > override it. I quickly learned that one needs to pass a keyword to the
> > dataclass decorator to tell it *not* to auto generate methods you
> override.
> >
> > I have two usability concerns with the current implementation. I emailed
> > Eric about the first, and he said I should ask for thoughts here. The
> > second I found after a couple of days sitting on this message.
> >
> > The first is that needing both a keyword and method is duplicative and
> > unnecessary. Eric agreed it was a hassle, but felt it was justified
> > considering someone may accidentally override a dataclass method. I
> > disagree with this point of view as dataclasses are billed as providing
> > automatic methods. Overriding via method definition is very natural and
> > idiomatic.
>
> Agreed.  We shouldn't take magic too far just for the sake of
> protecting users against their own (alleged) mistakes.  And I'm not
> sure how you "accidentally" override a dataclass method (if I'm
> implementing a __repr__ I'm doing so deliberately :-)).
>

My thinking exactly.

>
> > The second concern, which I came across more recently, is if I have a
> base
> > class, and dataclasses inherit from this base class, inherited __repr__ &
> > co are silently overridden by dataclass. This is both unexpected, and
> also
> > means I need to pass a repr=False to each subclass' decorator to get
> > correct behavior, which somewhat defeats the utility of subclassing. Im
> not
> > as sure a whole lot can be done about this though.
>
> Agreed as well.  If I make the effort of having a dataclass inherit
> from a base class, I probably don't want the base class' methods to be
> silently overriden by machine-generated methods.  Of course, that can
> be worked around by using multiple inheritance, you just need to be
> careful and add a small amount of class definition boilerplate.
>

I am not sure exactly what you mean by "worked around by using multiple
inheritance". Do you mean you think
the dataclass decorator should be made into a dataclass base class for a
dataclass class, or that it should look
at the MRO?


>
> I would expect dataclass parameters such as `repr` to be tri-state:
>
> * repr=None (the default): only provide a machine-generated
>   implementation if none is already defined (either on a base class or
>   in the dataclass namespace... ignoring runtime-provided defaults such
>   as object.__repr__)
> * repr=False: never provide a machine-generated implementation
> * repr=True: always provide a machine-generated implementation, even
>   overriding a previous user-defined implementation
>

This is sensible to me.


Thanks for your thoughts!

~>Ethan Smith


> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> ethan%40ethanhs.me
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses

[Ethan Smith]

On Fri, Dec 29, 2017 at 11:37 AM, Ethan Furman  wrote:

> On 12/29/2017 02:23 AM, Ethan Smith wrote:
>
> The first is that needing both a keyword and method is duplicative and
>> unnecessary. Eric agreed it was a hassle, but
>> felt it was justified considering someone may accidentally override a
>> dataclass method. I disagree with this point of
>> view as dataclasses are billed as providing automatic methods. Overriding
>> via method definition is very natural and
>> idiomatic. I don't really see how someone could accidentally override a
>> dataclass method if methods were not generated
>> by the dataclass decorator that are already defined in the class at
>> definition time.
>>
>
> Accidental or not, the decorator should not be replacing methods defined
> by the class.
>
> The second concern, which I came across more recently, is if I have a base
>> class, and dataclasses inherit from this base
>> class, inherited __repr__ & co are silently overridden by dataclass. This
>> is both unexpected, and also means I need to
>> pass a repr=False to each subclass' decorator to get correct behavior,
>> which somewhat defeats the utility of
>> subclassing. Im not as sure a whole lot can be done about this though.
>>
>
> It is possible to determine whether an existing __repr__ is from 'object'
> or not, and only provide one if that is the case.  I think that should be
> the default, with 'repr = True' for those cases where a new,
> auto-generated, __repr__ is desired.
>

The only other thing you'd want to handle is to cover inheriting from
another dataclass. e.g., if I have dataclass with attribute a, and subclass
it to add attribute b, I'd want both in the repr.

~>Ethan Smith

>
> --
> ~Ethan~
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ethan%
> 40ethanhs.me
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses

[Ethan Smith]

On Fri, Dec 29, 2017 at 12:30 PM, Ethan Furman  wrote:

> On 12/29/2017 11:55 AM, Ethan Smith wrote:
>
>> On Fri, Dec 29, 2017 at 11:37 AM, Ethan Furman wrote:
>>
>
> It is possible to determine whether an existing __repr__ is from 'object'
>>>
>> >> or not, and only provide one if that is the case.  I think that should
> be
> >> the default, with 'repr = True' for those cases where a new,
> auto-generated,
>
>> __repr__ is desired.
>>>
>>
>> The only other thing you'd want to handle is to cover inheriting from
>> another dataclass. e.g., if I have dataclass with
>> attribute a, and subclass it to add attribute b, I'd want both in the
>> repr.
>>
>
> Good point.  So auto-generate a new __repr__ if:
>
> - one is not provided, and
> - existing __repr__ is either:
>   - object.__repr__, or
>   - a previous dataclass __repr__
>
> And if the auto default doesn't work for one's use-case, use the keyword
> parameter to specify what you want.
>
>
> --
> ~Ethan~
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ethan%
> 40ethanhs.me
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses

[Ethan Smith]

attrs just silently overwrites any user provided __repr__ unless you
provide repr=False to attr.s.

I think we can all agree that if nothing else, silently overwriting
unconditionally is not what we want for dataclasses.

On Fri, Dec 29, 2017 at 4:38 PM, Nathaniel Smith  wrote:

> On Fri, Dec 29, 2017 at 12:30 PM, Ethan Furman  wrote:
> > Good point.  So auto-generate a new __repr__ if:
> >
> > - one is not provided, and
> > - existing __repr__ is either:
> >   - object.__repr__, or
> >   - a previous dataclass __repr__
> >
> > And if the auto default doesn't work for one's use-case, use the keyword
> > parameter to specify what you want.
>
> What does attrs do here?
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> ethan%40ethanhs.me
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses

[Ethan Smith]

On Fri, Dec 29, 2017 at 4:52 PM, Guido van Rossum  wrote:

> I still think it should overrides anything that's just inherited but
> nothing that's defined in the class being decorated.
>
>
Could you explain why you are of this opinion? Is it a concern about
complexity of implementation?


> On Dec 29, 2017 5:43 PM, "Nathaniel Smith"  wrote:
>
>> On Fri, Dec 29, 2017 at 12:30 PM, Ethan Furman 
>> wrote:
>> > Good point.  So auto-generate a new __repr__ if:
>> >
>> > - one is not provided, and
>> > - existing __repr__ is either:
>> >   - object.__repr__, or
>> >   - a previous dataclass __repr__
>> >
>> > And if the auto default doesn't work for one's use-case, use the keyword
>> > parameter to specify what you want.
>>
>> What does attrs do here?
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
>> 40python.org
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> ethan%40ethanhs.me
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses

[Ethan Smith]

Okay, I think Guido's proposal is a good compromise.

I already have a branch of dataclasses that should implement that behavior,
so perhaps it was meant to be. :)

~>Ethan Smith

On Fri, Dec 29, 2017 at 5:13 PM, Nick Coghlan  wrote:

>
>
> On 30 Dec. 2017 11:01 am, "Ethan Smith"  wrote:
>
>
>
> On Fri, Dec 29, 2017 at 4:52 PM, Guido van Rossum 
> wrote:
>
>> I still think it should overrides anything that's just inherited but
>> nothing that's defined in the class being decorated.
>>
>>
> Could you explain why you are of this opinion? Is it a concern about
> complexity of implementation?
>
>
> Adding a new method to a base class shouldn't risk breaking existing
> subclasses.
>
> If folks want to retain the base class implementation, they can request
> that explicitly (and doing so isn't redundant at the point of subclass
> definition the way it is for methods defined in the class body).
>
> Cheers,
> Nick.
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses

[Ethan Smith]

On Mon, Jan 1, 2018 at 5:03 PM, Chris Barker  wrote:

> On Sat, Dec 30, 2017 at 7:27 AM, Stephen J. Turnbull <
> turnbull.stephen...@u.tsukuba.ac.jp> wrote:
>
>>   Just use the simple rule that a new
>> __repr__ is generated unless provided in the dataclass.
>>
>
> are we only talking about __repr__ here ???
>
> I interpretted Guido's proposal as being about all methods -- we _may_
> want something special for __repr__, but I hope not.
>
> But +1 for Guido's proposal, not only because it's easy to explain, but
> because it more naturally follows the usual inheritance logic:
>
> The decorator's entire point is to auto-generate boilerplate code for you.
> Once it's done that it shouldn't, in the end, behave any differently than
> if you hand wrote that code. If you hand wrote the methods that the
> decorator creates for you, they would override any base class versions. So
> that's what it should do.
>
> And the fact that you can optionally tell it not to in some particular
> case keeps full flexibility.
>
> -CHB
>

I interpreted this to be for all methods as well, which makes sense.
Special casing just __repr__ doesn't make sense to me, but I will wait for
Guido to clarify.

>
> > I grant that there may be many reasons why one would be deriving
>
>> dataclasses from dataclasses
>
>
> Will you get the "right" __repr__ now if you derive a datacalss from a
> dataclass? That would be a nice feature.
>

The __repr__ will be generated by the child dataclass unless the user
overrides it. So I believe this is the "right" __repr__.

~>Ethan Smith

>
> -CHB
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> ethan%40ethanhs.me
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] libxml2 installation/binding issue

[Ethan Smith]

This list is for the discussion of development *of* Python. For discussion
of development *with* Python, you want python-list.

On Mon, Feb 5, 2018 at 2:41 PM, Priest, Matt  wrote:

> Hello,
>
>
>
> I am not sure if this is the correct place to post an issue/question like
> this, but here goes…
>
>
>
> I’ve successfully (?) installed Python 3.6.4 and libxml2, with the
> ultimate goal of installing GTK+ 3.22.0.
>
> However, I’m running into this error:
>
>
>
>
>
> python3
>
> Python 3.6.4 (default, Feb  5 2018, 13:28:04)
>
> [GCC 4.7.2] on linux
>
> Type "help", "copyright", "credits" or "license" for more information.
>
> >>> import libxml2
>
> Traceback (most recent call last):
>
>   File "", line 1, in 
>
>   File "/nfs/sc/disks/slx_1353/mlpriest/sl1/work_root/a0/
> development/sfwr/lib/python3.6/site-packages/libxml2.py", line 1, in
> 
>
> import libxml2mod
>
> ImportError: /nfs/sc/disks/slx_1353/mlpriest/sl1/work_root/a0/
> development/sfwr/lib/python3.6/site-packages/libxml2mod.so: undefined
> symbol: _PyVerify_fd
>
>
>
>
>
> Here are the details on the version, cflags, and ldflags.
>
> python3 --version ;
>
> Python 3.6.4
>
> python3-config --cflags
>
> -I/nfs/sc/disks/slx_1353/mlpriest/sl1/work_root/a0/
> development/sfwr/include/python3.6m
>
> -I/nfs/sc/disks/slx_1353/mlpriest/sl1/work_root/a0/
> development/sfwr/include/python3.6m
>
> -Wno-unused-result
>
> -Wsign-compare
>
> -fPIC -DNDEBUG
>
> -g
>
> -fwrapv
>
> -O3
>
> -Wall
>
> -Wstrict-prototypes
>
>
>
> python3-config –ldflags;
>
> -L/nfs/sc/disks/slx_1353/mlpriest/sl1/work_root/a0/
> development/sfwr/lib/python3.6/config-3.6m-x86_64-linux-gnu
>
> -L/nfs/sc/disks/slx_1353/mlpriest/sl1/work_root/a0/development/sfwr/lib
>
> -lpython3.6m
>
> -lpthread
>
> -ldl
>
> -lutil
>
> -lrt
>
> -lm
>
> -Xlinker
>
> -export-dynamic
>
>
>
> Anyhelp or hint would be appreciated…
>
>
>
>
>
>
>
> Matt
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> ethan%40ethanhs.me
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 561 implemented and minor clarification

[Ethan Smith]

Hello,

I've updated PEP 561 to clarify that any installed stub package should
supersede an installed inline package. In other words if there is:

/global/site-packages/pkg/
/user/site-packages/pkg-stubs/

Even if pkg in the global site packages is found first and marks that it
supports types, the stub package should supersede it.

I also point to mypy's docs on its implementation of the PEP (which can be
read about here:
https://mypy.readthedocs.io/en/latest/installed_packages.html). The
implementation has been merged into master and will be available in the
0.590 release.

As always, the PEP text is replicated below.

Cheers!
Ethan


==

PEP: 561
Title: Distributing and Packaging Type Information
Author: Ethan Smith 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Sep-2017
Python-Version: 3.7
Post-History: 10-Sep-2017, 12-Sep-2017, 06-Oct-2017, 26-Oct-2017


Abstract


PEP 484 introduced type hinting to Python, with goals of making typing
gradual and easy to adopt. Currently, typing information must be distributed
manually. This PEP provides a standardized means to leverage existing tooling
to package and distribute type information with minimal work and an ordering
for type checkers to resolve modules and collect this information for type
checking.


Rationale
=

Currently, package authors wish to distribute code that has inline type
information. Additionally, maintainers would like to distribute stub files
to keep Python 2 compatibility while using newer annotation syntax. However,
there is no standard method to distribute packages with type information.
Also, if one wished to ship stub files privately the only method available
would be via setting ``MYPYPATH`` or the equivalent to manually point to
stubs. If the package can be released publicly, it can be added to
typeshed [1]_. However, this does not scale and becomes a burden on the
maintainers of typeshed. In addition, it ties bug fixes in stubs to releases
of the tool using typeshed.

PEP 484 has a brief section on distributing typing information. In this
section [2]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for
shipping stub files. However, manually adding a path to stub files for each
third party library does not scale. The simplest approach people have taken
is to add ``site-packages`` to their ``MYPYPATH``, but this causes type
checkers to fail on packages that are highly dynamic (e.g. sqlalchemy
and Django).


Definition of Terms
===

The definition of "MAY", "MUST", and "SHOULD", and "SHOULD NOT" are
to be interpreted as described in RFC 2119.

"inline" - the types are part of the runtime code using PEP 526 and 3107
syntax.

"stubs" - files containing only type information, empty of runtime code.

"Distributions" are the packaged files which are used to publish and distribute
a release. [3]_

"Module" a file containing Python runtime code or stubbed type information.

"Package" a directory or directories that namespace Python modules.


Specification
=

There are several motivations and methods of supporting typing in a package.
This PEP recognizes three (3) types of packages that users of typing wish to
create:

1. The package maintainer would like to add type information inline.

2. The package maintainer would like to add type information via stubs.

3. A third party or package maintainer would like to share stub files for
   a package, but the maintainer does not want to include them in the source
   of the package.

This PEP aims to support these scenarios and make them simple to add to
packaging and deployment.

The two major parts of this specification are the packaging specifications
and the resolution order for resolving module type information. The type
checking spec is meant to replace the ``shared/typehints/pythonX.Y/`` spec
of PEP 484 [2]_.

New third party stub libraries SHOULD distribute stubs via the third party
packaging methods proposed in this PEP in place of being added to typeshed.
Typeshed will remain in use, but if maintainers are found, third party stubs
in typeshed MAY be split into their own package.


Packaging Type Information
--

In order to make packaging and distributing type information as simple and
easy as possible, packaging and distribution is done through existing
frameworks.

Package maintainers who wish to support type checking of their code MUST add
a marker file named ``py.typed`` to their package supporting typing.
This marker applies
recursively: if a top-level package includes it, all its sub-packages
MUST support
type checking as well. To have this file installed with the package,
maintainers can use existing packaging options such as ``package_data`` in
distutils, shown below.

Distutils option example::

setup(
...,
package_data = {
'foopk

Re: [Python-Dev] Compiling without multithreading support -- still useful?

[Ethan Smith]

I think this is useful as it can make porting easier. I am using it in my
attempts to cross compile CPython to WebAssembly (since WebAssembly in its
MVP does not support threading).

On Wed, Sep 6, 2017 at 10:15 AM, Antoine Pitrou  wrote:

>
> I made an experimental PR to remove support for threads-less builds:
> https://github.com/python/cpython/pull/3385
>
> The next effect is to remove almost 2000 lines of code (including many
> #ifdef sections in C code).
>
> Regards
>
> Antoine.
>
>
> On Tue, 5 Sep 2017 18:36:51 +0200
> Antoine Pitrou  wrote:
> > Hello,
> >
> > It's 2017 and we are still allowing people to compile CPython without
> > threads support.  It adds some complication in several places
> > (including delicate parts of our internal C code) without a clear
> > benefit.  Do people still need this?
> >
> > Regards
> >
> > Antoine.
> >
> >
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> ethan%40ethanhs.me
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compiling without multithreading support -- still useful?

[Ethan Smith]

Certainly, I understand it can be burdensome. I suppose I can use 3.6
branch for the initial port, so it shouldn't be an issue.

On Wed, Sep 6, 2017 at 11:13 AM, Antoine Pitrou  wrote:

> On Wed, 6 Sep 2017 10:50:11 -0700
> Ethan Smith  wrote:
> > I think this is useful as it can make porting easier. I am using it in my
> > attempts to cross compile CPython to WebAssembly (since WebAssembly in
> its
> > MVP does not support threading).
>
> The problem is that the burden of maintenance falls on us (core CPython
> developers), while none of us and probably 99.99% of our userbase have
> absolutely no use for the "functionality".
>
> Perhaps there's a simpler, cruder way to "support" threads-less
> platforms.  For example a Python/thread_nothreads.h where
> PyThread_start_new_thread() would always fail (and with trivial
> implementations of locks and TLS keys).  But I'm not sure how much it
> would help those porting attempts.
>
> Regards
>
> Antoine.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> ethan%40ethanhs.me
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: Increase of Spammy PRs and PR reviews

[Ethan Smith]

As a non-committer, I want to make a plea for non-committer approval
reviews, or at least that they have a place. When asked how outsiders can
contribute I frequently see "review open PRs" as a suggested way of
contributing to CPython. Not being able to approve PRs that are good would
be a barrier to those contributions.

Furthermore, I am collaborating with a couple of core devs, it would make
collaboration more difficult if I couldn't review their work and say that I
thought the changes looked good.

I know "drive by approvals" are annoying but I think it is unfortunately
part of open source projects.


On Sat, Jan 29, 2022, 5:51 PM Inada Naoki  wrote:

> On Sun, Jan 30, 2022 at 10:39 AM Ethan Furman  wrote:
> >
> >
> >  > And lots of non-committer PR reviews that only approve.
> >
> > I have seen this.  Quite irritating.
> >
>
> We can prohibit approval from non core developers. Do we use this
> setting already?
>
> https://github.blog/2021-11-01-github-keeps-getting-better-for-open-source-maintainers/
>
>
> --
> Inada Naoki  
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/KHXSE2MSEC5JR2QB5F6HJUFYCC4SHGFF/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BKIUK2KGP2YMN7GBEYYQVTEMECE2WTSD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Increase of Spammy PRs and PR reviews

[Ethan Smith]

On Sat, Jan 29, 2022 at 7:38 PM Inada Naoki  wrote:

> On Sun, Jan 30, 2022 at 12:03 PM Ethan Smith  wrote:
> >
> > As a non-committer, I want to make a plea for non-committer approval
> reviews, or at least that they have a place. When asked how outsiders can
> contribute I frequently see "review open PRs" as a suggested way of
> contributing to CPython. Not being able to approve PRs that are good would
> be a barrier to those contributions.
> >
> > Furthermore, I am collaborating with a couple of core devs, it would
> make collaboration more difficult if I couldn't review their work and say
> that I thought the changes looked good.
> >
>
> You can still write a review comment, including "LGTM".


Fair enough, I suppose commenting with "LGTM" works just as well.


> What you can
> not is labeling PR as "Approved."
> So I don't think it makes collaboration difficult.
>
By preventing approval from others, we can easily find PRs approved
> from core-devs or triage members but not merged yet.
>

> > I know "drive by approvals" are annoying but I think it is unfortunately
> part of open source projects.
> >
>
> Sorry, but I don't think so.
>

Well, if you disallow drive-by approvals, you will still get drive-by LGTM
comments. People seem to believe that adding their approval will expedite a
PR merge, for some reason (or want to bump a stale PR in hopes of a quicker
merge).


>
> --
> Inada Naoki  
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QJ5QBB4XCFU3DFSOFBRUJB5XC4UMX7TK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Stalemate on bringing back PEP 523 support into Python 3.8

[Ethan Smith]

(Replying here since b.p.o doesn't seem to want to let me log in)

I've used the PEP 523 API several times for multiple projects, from
analyzing bytecode frequency (which probably could be done with other
means) to inspecting type information from function calls. I could probably
do such analysis a different way too, (some people do it via tracing). But
from what I have experienced, the PEP 523 API is much faster than
alternatives, and I would be sad to see it go. I also have written some of
these experiments in Rust, and it would make it harder to use if I had to
deal with enabling it. I would much prefer the proposed setters/getters.

Also the discussion is here for those who are interested
https://bugs.python.org/issue38500

Best,
Ethan

On Thu, Nov 21, 2019 at 12:07 PM Brett Cannon  wrote:

> An unfortunate side-effect of making PyInterpreterState in Python 3.8
> opaque is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/)
> support. https://www.python.org/dev/peps/pep-0523/ was opened to try and
> fix this, but there seems to be a stalemate in the issue.
>
> A key question is at what API level should setting the frame evaluation
> function live at? No one is suggesting the stable ABI, but there isn't
> agreement between CPython or the internal API (there's also seems to be a
> suggestion to potentially remove PEP 523 support entirely).
>
> And regardless of either, there also seems to be a disagreement about
> providing getters and setters to continue to try and hide
> PyInterpreterState regardless of which API level support is provided at (if
> any).
>
> If you have an opinion please weight in on the issue.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/4UZJYAZL3NHRAGN5WAMJC4IHAHEXF3QF/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/55CY7JXPYJWXUO7HE4S334AOIWACKV75/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 585: Type Hinting Generics In Standard Collections

[Ethan Smith]

While working on the implementation with Guido I made a list of things that
inherit from typing.Generic in typeshed that haven't been
listed/implemented yet.

https://github.com/gvanrossum/cpython/pull/1#issuecomment-582781121



On Sat, Feb 22, 2020, 3:50 PM Nick Coghlan  wrote:

> This looks like a nice usability improvement to me.
>
> My only suggestion would be that types.MappingProxyType be included on the
> list of types to be updated.
>
> Cheers,
> Nick.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/EYU3VDK7T4OVT7MXM5OWOPFA4YKWXVDE/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z47AGWUR5ZVQQ7GEEFQYPE5HBCUMEL44/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 585: Type Hinting Generics In Standard Collections

[Ethan Smith]

The discussion on the name change came from Łukasz
https://github.com/python/cpython/pull/18239#discussion_r380996908

I suggested "GenericType" to be in line with other things in types.py.

On Mon, Feb 24, 2020 at 8:39 PM Guido van Rossum  wrote:

> I can't find it right now, but IIRC somebody commented that "GenericAlias"
> is a somewhat odd name. I didn't spend much time thinking about the name, I
> just took it from `typing._GenericAlias` (which has a similar role).
>
> It would be hard to change the name later. ATM it's one global substitute
> on my branch. Should we change it? To what?
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/B7P3DITJDZAUHWESPBI34IYMALULGTF2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Need help with test_ctypes failing on Windows (test_load_dll_with_flags)

[Ethan Smith]

(Trusty assistant reporting in) I should also note that further up the
output there is a FileNotFoundError for sqlite3.dll, perhaps it isn't built
or is in the wrong place?

Ethan


On Mon, Apr 6, 2020, 7:19 PM Guido van Rossum  wrote:

> I have a large PR (https://github.com/python/cpython/pull/18239, for PEP
> 585) that's failing in the Azures pipeline on Win32 and Win64 only. My
> trusty assistant who has a Windows laptop couldn't reproduce the failure.
> Can I buy a hint from someone? Steve?
>
> The relevant failure output is:
>
> ==
> ERROR: test_load_dll_with_flags (ctypes.test.test_loading.LoaderTest)
> [WinDLL('_sqlite3.dll', winmode=0)]
> --
> Traceback (most recent call last):
>   File "d:\a\1\s\lib\ctypes\test\test_loading.py", line 140, in should_pass
> subprocess.check_output(
>   File "d:\a\1\s\lib\subprocess.py", line 420, in check_output
> return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
>   File "d:\a\1\s\lib\subprocess.py", line 524, in run
> raise CalledProcessError(retcode, process.args,
> subprocess.CalledProcessError: Command
> '['d:\\a\\1\\s\\PCbuild\\win32\\python.exe', '-c', "from ctypes import *;
> import nt;WinDLL('_sqlite3.dll', winmode=0)"]' returned non-zero exit
> status 1.
>
> --
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/2IMODEGY25VK6R2FPGJ3QKAAVIK5EZKH/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HEESZRLMHUAORUBHSFPKMO6MNAZZMVKE/
Code of Conduct: http://python.org/psf/codeofconduct/