from:"Christopher Barker"

On Wed, Apr 14, 2021 at 7:48 AM David Mertz  wrote:

> >>> vaex.__version__
> {'vaex': '4.1.0', 'vaex-core': '4.1.0', 'vaex-viz': '0.5.0', 'vaex-hdf5':
> '0.7.0', 'vaex-server': '0.4.0', 'vaex-astro': '0.8.0', 'vaex-jupyter':
> '0.6.0', 'vaex-ml': '0.11.1'}
>

Well, THAT is a great argument for some official standardization!

There is sometimes a need for that sort of thing, but I think it's best
handled by either putting __version__ in each sub_package, or having a
different attribute altogether.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DAZGU7U7HU6TEBTM7DKRU2P4L5KBFY6Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

Simon,

I'm not sure your note made it to the list -- bringing it back on.

On Wed, Apr 14, 2021 at 1:15 AM Simon Cross 
wrote:

> I would be +1 on having a small PEP like this one to give __version__
> an official blessing and some well-defined expected behaviour.
>
> I don't have a deep opinion on the specifics of the specification, but
> I'm guessing they should be thought about again even if the answers
> end up being the same. Quite a lot has changed in the Python ecosystem
> in the last 13 years.
>

Yes indeed. Critically, over on the packaging side, standardization of
versioning specs.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/H3U7IFNOBWAZ7GFG3WIGCV4HNUFOUI64/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

On Wed, Apr 14, 2021 at 5:24 AM Victor Stinner  wrote:

> There are two main use cases for versions:
>
> * Display them to the user
> * Compare versions to check if one is newer, older or the same
>
> I dislike using strings for comparison. You need to use
> packaging.version for that:
> https://packaging.pypa.io/en/latest/version.html

Yes, though it looks a little too heavyweight to be required, nifty
features like ``is_prerelease` is going pretty far.

On the other hand, maybe no harm in all that, as long as:

- It's easy to do the easy stuff (which is looks like it is)
- It would be added to the stdlib

However, there is a lot of code out there that's already using strings for
__version__, so it would be a real shame if we had to ask everyone to
upgrade. And Version isn't going to be in older versions of the Python
stdlib, so hard to make code compatible.

But: If we make a Version object that can be compared to (confoming)
strings then there might be a path forward.

> In my Python projects, I like to provide the version as a tuple which
> can be used directly for comparison: version_a <= version_b. Example:
>

Yes, and tuples are already built in to Python, so an easier lift than
adding the Version object.

Another possible issue: using Version would require an extra import in many
module initializations -- is that a performance issue that would matter?

In any case, I think the PEP should specify a standard for what is in
__version__

And ideally it would be backward compatible with strings, which are the
most commonly used currently.

And conform to PEP 440

Sadly, such tuple is no standardized. Which
> part is the major version? How to format it as a string?
>
> Good luck with trying to standardize that ;-)
>

well, PEP 440 gets us close -- and I *think* compliant strings could be
unambiguously converted to-from tuples.

Anyway, the question now for me is whether this is worth any more of my
time.

So:
- Is there someone willing to sponsor?
- Do folks generally think there's a chance of it being accepted without a
huge debate and expansion of scope.

If the answers are not yes, I can better spend my Python time of other
things.

BTW: one source of hope:

__version__ strings are pretty common, but, as evidenced already by this
brief discussion, it's not by any means a universal standard -- so we
really can't break anything that already works in any universal way.

What i mean by that is that anything done here might break someone's
special use case of __version__ (Like the one David Mertz identified), but
that scheme only works with that one package's code anyway. There are no
general tools that expect that scheme.

And it could be preserved by making a subclass of Version (Or str) that was
also a mapping :-)

- CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QUJ6ESEXT2F3OWQIFG4ZFC4RT3CRRPCG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

On Wed, Apr 14, 2021 at 12:10 PM Barry Warsaw  wrote:

I’d forgotten that this PEP was in Deferred state.  I think it should be
> rejected and I plan on making that change.  importlib.metadata is a much
> better approach to programmatically determining package versions.
>
>
> https://docs.python.org/3/library/importlib.metadata.html#distribution-versions

Barry,

You wrote the original PEP, so of course you can withdraw it (or reject
it), but...

Are you sure? See this discussion, I don't think it's as simple as all that.

Honestly, I'm still kind of confused about a "distribution" vs a package or
module, but in general, it seems to me that the packaging community has
done a great job of building a system that can accommodate a lot of
complexity, but as they say:

The easy things should be easy, and the hard things should be possible.

And the full process of using packaging and distributions, and a Version
object and all that is not so easy anymore --at least compared to putting:

__version__ = "1.2.3"

in a package __init__.py.

Anyway, a couple points:

1) it seems self-evident that it would be easier for the whole community if
"how to declare", and "how to find" the version of a package was
standardized: "there's only one way to do it"

1b) It seems like a really good idea that the (or one of the) official
recommended way to do it is supported in the standard library itself, and
using a string is pretty easy to support :-) -- do we really want people to
have to install a third party package to set or get the version of a simple
module?

2) __version__ strings are already pretty darn common -- maybe more common
than proper distribution metadata? -- are we saying that everyone should
stop doing that?

3) I don't know that having __version__ strings is incompatible with
packging.version.Version or importlib.metadata.version

4) Yes, there are real advantages to the more complex system for
programmatically working with versions. But there are real advantages to
something simple for human readability / interactive use -- and, frankly,
simple scripting. See David Mertz's recent post -- I went through a similar
process in iPython.

Consider this analogy: if I want to know what version of git I have
installed, I can run:

$ git --version

And that's a convention adhered to by many *nix command line tools. Would
we all really be happier if we had to go use an rpm or deb or homebrew, or
?? command to see what package i have installed? I think this is similar --
while a good distribution management system is a good thing, it's also good
if I can tell something about the version of a package/module from
the package itself.

With regard to having both __version__ and importlib.metadata.version, see
beautiful soup for example:

In [10]: import beautifulsoup4
...
ModuleNotFoundError: No module named 'beautifulsoup4'

oops, that's not what the importable module is called. So:

In [11]: import bs4

That worked -- now, what version is it?

In [12]: importlib.metadata.version('bs4')
---
PackageNotFoundError  Traceback (most recent call last)
...
PackageNotFoundError: bs4

[*]

That didn't work -- I guess I need to use the distribution name:

In [13]: importlib.metadata.version('beautifulsoup4')
Out[13]: '4.9.3'

OK, that worked.

But I imported "bs4", so what if I do this?

In [14]: bs4.__version__
Out[14]: '4.9.3'

Ah yes, that worked too.

So I guess what I'm suggesting is that we officially recommend that folks
do what the BeautifulSoup folks, and many others, are already doing. In
many (most) cases, the distribution name and the importable package name
are the same, but this does accommodate both.

[*] One other annoyance to importlib.metadata.version is that in addition
to having to import importlib to use it, I also need to use
importlib.metadata.PackageNotFoundError if I want to trap the exception.

BTW: I'm not making that last point up -- I recently updated some in-house
code (that someone else had written) in a web app that presented the
version of various packages core to the system. We had just refactor and
one of these packages what now optional, but the whole system was crashing
on that darn PackageNotFoundError. I ended up jsut commenting out that
package, rather than dealing with how to catch the exception and deal with
it properly. I would have so much rathered a simple:

try:
import the_package
the_pacakge_version = the_package.__version__
except
the_package_version = None

Granted, it's not that hard to "do it right", but I still like things
simple :-)

Anyway, as the original PEP author is no longer supportive, this is dead in
the water unless another core dev is interested in sponsoring this (or a

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

On Wed, Apr 14, 2021 at 2:23 PM Paul Moore  wrote:

> >> because PEP 440 versions are fundamental to packaging.
> >
> > Are you suggesting that users should have to install an external module
> to tell what version of packages they are using?!
>
> No. To tell what version of a package they are using, a string is
> sufficient.
>

exactly -- if we do anything at all here, it should probably be at least.

if you provide a __version__ attribute, it should be a PEP 440 compliant
string.

That would at least make the use __version__ a touch more consistent.

The next, more controversial, step would be to suggest that people SHOULD
provide a __version__string on all top-level packages. I would like that --
the current situation where many, but not all packages have __version__ is
really annoying.

They only need a version object if they want to do additional
> processing (like comparing versions, or checking whether a version
> meets a constraint).
>

And indeed, we could add the ability for packaging.version.Version objects
to be able to compare themselves with compatible strings -- I think that
would be pretty handy.

Given that the packaging ecosystem already has a canonical version
> object (provided by the packaging library), which has been used and
> tested extensively in many environments, inventing a different API
> seems at best ill-advised.

absolutely.

> Whether the stdlib needs a version object.
> rather than leaving that functionality to a 3rd party library, is the
> same question that comes up for *any* functionality that's proposed
> for the stdlib, and I have no particular opinion in this case.
>

I don't think it's the same as any functionality -- if we do want to better
standardize package versioning in Python, and I think we should, then the
Version object may become something useful to, and maybe even needed by,
virtually every third party package. Which makes it a pretty prime
candidate for the stdlib.

Alternatively,  the packaging package is pretty small, but if it grows, it
might be good to split out the run-time vs build-time pieces.

It's designed for programmatic use, not interactive use, yes. But
> that's sort of my point, why do you want anything more than the bare
> string in the REPL? What are you planning on doing with it?
>

there is something in between the REPL and full on system development --
something simple for quickly scripts is nice too. But a simple standardised
version string is fine for that.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BWIF72TLNB6A63SXYUO3KMSR6ZUNTALS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

On Thu, Apr 15, 2021 at 2:40 AM Victor Stinner  wrote:

> Paul Bryan:
> > Seems like this is something that should make its way into stdlib?
>
> In the last 10 years, the trend is more to remove anything related to
> packaging *outside* the stdlib :-)


Well, maybe all of us don't think that's a good idea ;-)

But anyway, I would say removing anything *related* to packaging outside
the stdlib is a bad idea -- requiring an external tool to build a package
is OK, but requireing an external use packages, not so much.

Presumably that's why importlib.metadata exists in the stdlib.

Version is arguably useful from the package user side. As I believe Victor
mentioned, there are two uses for version information: display to the user
-- for which version strings are fine, or programmatic comparison -- for
which something like the Version object is very helpful. Do we only need to
use version information programmatically when we are creating (or
installing) packages? I don't think so -- I know I have code that (poorly)
does version checking programmatically.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LXLG7RPUOCY3J63AFDCAOUL5HHAO33XS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

On Thu, Apr 15, 2021 at 9:38 AM Barry Warsaw  wrote:

> From a library maintainers point of view, I personally want to get away
> from using __version__ strings at all.  They’re kind of a pain to remember
> to bump on every new release.

That’s a tooling issue, and I don’t think that Python itself should take
any official stance on the tooling. But yes, anything that Python takes an
official stance on should be able to be easily supported with
not-too-complex tools.and I think a __version__ string fits this need.

Personally, I use the __version__ string in my packages as the canonical
source when I bold my packages.

And there are fancier tools for more complex needs.

> PEP 396 has been dormant for a long time and I have no interest in
pushing it forward any more.  If someone else wants to take up the cause, I
recommend creating a new PEP and referring back to 396.

Fair enough -- I'm still gathering data to see if I should do just that.

I am leaning toward at least SOME PEP about this -- having  __version__
around as a semi-convention with no official recommendation as to if or how
it should be used is kind of a mess.

-CHB
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FXIM52YHGECNJMKPA4E6UXJNRMZ2CUO2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

On Thu, Apr 15, 2021 at 12:27 PM Terry Reedy  wrote:

> At least some in the stdlib have not been.  For re.__version__, not
> since Nov 2001.
>

>From Serhiy over on python-ideas, he's started an effort to clean that up.

https://mail.python.org/archives/list/python-dev@python.org/thread/KBU4EU2JULXSMUZULD5HJJWCGOMN52MK/
).

I believe there is an unfinished PR

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BRKC3EJWJZFFAFH7VP7W2GCIRFFEIEDK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

I haven't commented on these, as I don't use type annotations in the
"usual" way.,
but:

On Thu, Apr 15, 2021 at 12:06 PM Samuel Colvin  wrote:

> I maintain pydantic <https://pydantic-docs.helpmanual.io/> which uses
> type annotations to provide runtime data validation.
>

I maintain in in-house system kinda similar to pydantic that doesn't use
type annotations for validation, but does use them for other things (mostly
serializing/deserializing to/from JSON), and other kinds of data validation.

> using `typing.get_type_hints()` is not a good replacement for type
annotations that are accessible as python objects.

Absolutely. It is very, very, handy to simply have the object itself easily
accessible.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W7KPZQFMJ2WVW26KKLQDAXHWWTTWFYKQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

On Thu, Apr 15, 2021 at 9:36 AM David Mertz  wrote:

> I was so hopeful about this, but in the end... not really.  I have not
> used this capability before.  Here are a few different situations I know of:
>

...

...
re PackageNotFoundError('re')
statistics PackageNotFoundError('statistics')
pandas 1.2.4
vaex 4.1.0
bs4 PackageNotFoundError('bs4')

>
> It seems like (somehow or another) importlib.metadata is doing something
> perhaps more useful for vaex.  But it is doing distinctly worse for re,
> statistics, and bs4.
>

funny you should try bs4, which I also used as an example in another post.

But what you have found is that there is a difference between a pacakge
(which you can import) and a "distribution" which is something you install.

importlib.metadata is looking for distributions.

re and statistics are not distributions, but rather, built in pacakges
(modules).

bs4 is not a distributiuion, it is a package that is provided by the
"beautifulsoup4" distribution.

In [3]: importlib.metadata.version("beautifulsoup4")
Out[3]: '4.9.3'

Frankly, I'm still a bit confused about the distiction, but I do know for
sure that a single distribution (maybe somethign that you can install via
PyPi) can install more than one top-level package -- and certainly the
packages/modules installed can have a different name than the distribution
name.

And if a distribution installs more than one package, they may have
different version numbers.

I'm not sure what to make of all this, though I'm leaning toward better
suporting the distiction by asking for __version__ strings in top-level
packages -- and maybe making importlib.metadata.version a bit smarter about
looking for packages, and not just distributions.

If you look at the docstring of metadata.version:

"""
In [2]: importlib.metadata.version?
Signature: importlib.metadata.version(distribution_name)
Docstring:
Get the version string for the named package.

:param distribution_name: The name of the distribution package to query.
:return: The version string for the package as defined in the package's
"Version" metadata key.
File:  ~/miniconda3/envs/py3/lib/python3.9/importlib/metadata.py
Type:  function
"""

It's a bit inconsistent with the use of the term "distribution" vs
"package". That should get cleaned up if nothing else.

Also, the Exception raised is "PackageNotFoundError" -- which should maybe
be "DistributionNotFoundError"?

Version is arguably useful from the package user side. As I believe Victor
mentioned, there are two uses for version information: display to the user
-- for which version strings are fine, or programmatic comparison -- for
which something like the Version object is very helpful. Do we only need to
use version information programmatically when we are creating (or
installing) packages? I don't think so -- I know I have code that (poorly)
does version checking programmatically.

> Or rather, the below is what I would find really nice to be able to do.
>
> ver = robust_version(module)
> if ver >= (5, 2, 1):
> doit_modern_style()
> elif ver < (5, 2, 1):
> doit_old_style
> else:
> doit_unversioned_style()
>

Exactly -- and I htink we are close, if pacakges(modules) had compliant
__version__ strings, thenyou could do
(with the Version object from packaging -- maybe why it should be in the
stdlib)

ver = Version(module.__version__)
if ver >= Version("5.2.1"):
doit_modern_style()
elif ver < Version("5.2.1"):
doit_old_style
else:
doit_unversioned_style()

And if my PR is accepted (which looks unlikley) to allow camparison between
Version objects and strings:

ver = Version(module.__version__)
if ver >= "5.2.1":
doit_modern_style()
elif ver < "5.2.1":
doit_old_style
else:
doit_unversioned_style()

A key distiction here from the importlib.metadata approach is the level of
indirection -- I like being able to ask a module itself what version it is,
rather than asking some other part of teh system to go look it up for me.
So I could do:

import numpy as np

print("using np version:", np.__version)

And this is pretty consitent with the rest of Python, where many objects
(functions, classes, modules) have a __name__ attribute if things "know"
their name, shouldn't they "know" their version as well?

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YQYKOZ273ISH3KLWTDGNRTSYUCQGYD33/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

On Thu, Apr 15, 2021 at 3:29 PM Bernat Gabor  wrote:
> I'm a bit surprised that this topic is brought up just days before the
feature freeze of Python 3.10.

I have not idea about the technical details, but I think there is a bit of
a discomment in the community:

Annotations have been around for years.

More recently, there was an official specification of how to use them for
type hinting (was that PEP 484? a bit confused as that appears to still be
provisional -- but anyway)

Since the specification for type hinting, a lot of work has been done
on/with static type checkers (e.g. MyPy).

That work has informed the changes / improvements / proposals to the
annotation and type system in Python.

However, those of us that are not actively involved (and maybe not that
interested) in static typing have been not paying a great deal of attention
to those efforts.

But some of us may, in fact, be using annotations for other reasons, but
since we haven't been involved in the discussion, some issues may have been
missed.

-CHB


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5TGKKRZXGQROQKS2WX6WFGCHTMNUJYBF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

On Thu, Apr 15, 2021 at 4:20 PM Brett Cannon  wrote:

>
>  Making this an "us versus them" discussion just makes the whole situation
> feel confrontational when instead everyone is trying to figure out the best
> thing for everybody when there's no perfect answer.
>

I agree that that was strong language, but from the perspective of folks
not interested in Static Typing, there has been a lot of activity, or at
least discussion,  that makes it seem like this optional feature is driving
the evolution of the language.

So folks are going to get concerned, and maybe upset, if it looks like
changes might be made that may break other features in order to make static
typing work better.

And as I noted in my last post — many folks have not been paying attention
to the typing discussions because they didn’t realize it concerned them.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O57G2ZDD2VW7OB5IMPKACL775HBDSNIP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Relaxing the annotation syntax

2021-04-16 Thread Christopher Barker

On Fri, Apr 16, 2021 at 1:51 AM Sebastian Rittau  wrote:

> I am strongly in favor of diverging type annotation syntax from Python
> syntax. Currently, type annotations are a very useful tool, but often
> clunky to use. Enhancements have been made, but design space is limited
> when working within existing Python syntax. Type annotations have a
> different set of rules, needs, and constraints than general-purpose Python
> code. This is similar to other domain specific languages like regular
> expressions. Ideally, Python itself would not check the syntax of
> annotations, except as needed for determining the end of an annotation.
>
Another example is a discussion a little while back on python-ideas about
extending what's allowed inside square brackets. It started with a use-case
for type specification. It turned out that there were other use cases, more
tightly tied to the original meaning of __getitem__.

Nevertheless, it struck me at the time that it would be nice if the Typing
use case could be addressed without the complication of making something
that made sense in two very different domains.

- Chris

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SG5MCSP6KQDEO2BDZ6LE76TSKGH7TD6E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

2021-04-16 Thread Christopher Barker

I wonder if anyone has considered the impact of PEP 563 on dataclasses ?

I did find this SO post:

https://stackoverflow.com/questions/52171982/new-annotations-break-inspection-of-dataclasses

which is related, but not quite the same -- THAT issue was fixed.

My issue does show up in this BPO:

https://bugs.python.org/issue39442

Somewhere in the middle of that thread, Eric wrote: "At this point, this
seems more like fodder for python-ideas."

Then there's a bit more to the thread, which peters out without resolution.

On to the issue:

dataclasses may be the only part of the stdlib that uses annotations.

dataclasses store the annotations in Field objects type attribute, in
the __dataclass_fields__ attribute.

We can see that with this code:

@dataclass
class Simple:
x : int
y : float
name : str

s = Simple(3, 4.0, "fred")

print("The field types:")
for f in s.__dataclass_fields__.values():
print(f"name: {f.name}, type: {f.type}, type of type: {type(f.type)}")

Which results in:

The field types:
name: x, type: , type of type: 
name: y, type: , type of type: 
name: name, type: , type of type: 

with:

from __future__ import annotations

The result is:

The field types:
name: x, type: int, type of type: 
name: y, type: float, type of type: 
name: name, type: str, type of type: 

This of course is completely as expected.

I have no idea how dataclasses uses the Field type attribute -- as far as I
can tell, for nothing at all. However, it is there, and it is called
"type", rather than say, "annotation".

And I have a whole pile of code that fully expects the Fields' type
attribute to be an actual type object that I can call to crate an instance
of that type (or call a method on it, which is what I am actually doing)

So my code will very much break with this change.

I fully understand that the __dataclass_fields__ attribute was probably
never intended to be part of the public API, so I get what I deserve.

However, the Field object is documented, as such:

"""
class dataclasses.Field

Field objects describe each defined field. These objects are created
internally, and are returned by the fields() module-level method (see
below). Users should never instantiate a Field object directly. Its
documented attributes are:

name: The name of the field.
type: The type of the field.
default, default_factory, init, repr, hash, compare, and metadata have the
identical meaning and values as they do in the field() declaration.

Other attributes may exist, but they are private and must not be inspected
or relied on.
"""

That last sentence implies that the type attribute CAN be inspected and
relied upon, which is what I am doing.

And I haven't tried to solve this problem in my use case, but I'm not sure
it can be done -- when I get around to inspecting the type of the Field
objects, I have no idea what namespace they are in -- so I can't
reconstruct them from the string. I really need the type object itself.

So I'll probably need to re-write much of the dataclasses decorator, to
call eval() early -- though even then I'm not sure I'll have access to the
proper namespace.

Anyway -- one more data point:  PEP 563 changes the (semi-public?) API of
dataclasses.

Though *maybe* that could be addressed with a dataclasses update -- again,
I've only started to think about it -- there was some discussion of that in
the BPO, though Eric didn't seem particularly interested.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4N4FE2Y37WZBILVUPHEHNVRE6NVU6VFO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

2021-04-17 Thread Christopher Barker

On Sat, Apr 17, 2021 at 6:12 AM Eric V. Smith  wrote:

> I wonder if anyone has considered the impact of PEP 563 on dataclasses ?
>
> I have!
>

Thanks Eric.

> In retrospect, that field probably should have been named "annotation".
> Regardless, the intent was always "store what's in
> __annotations__[field_name]", or what the user specified in field(...,
> type=whatever, ...).
>

intent is all well and good, but what was the intent of __annotations__
In the first place. Static typing was certainly one of the intents, but as
it DID store a value, not a string from the beginning, we can only assume
that there was some intent that the value might be used.

As for dataclasses -- why store it at all? It seems that dataclasses use
the fact that a class attribute has an annotation to determine what to add
to the field -- which is a nifty use of teh syntax. But presumably the
intent of storing the annotation in the Field object, and providing a
public attribute for it, was so that it could be used.

I suppose your point is that the type attribute stored the annotation, and
PEP 653 changes what the annotation will be so there's nothing specific at
all about dataclasses here. Which is true. But users of dataclasses may
have been less aware of all the implications as they didn't write that
annotation etrating code themselves. I know I wasn't.

> In any event, all of this mess is one reason I'd like to see PEP 649 get
> accepted:
>
Indeed -- that is the title of this thread, after all :-)

And see others' notes, there seems to be two other places in the stdlib
that will be affected.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3ORLF6IBGMXJMBC2FDGYKBHU445ILK7E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 563 and 649: The Great Compromise

2021-04-18 Thread Christopher Barker

Thanks Larry,

This seems to be going in a good direction.

I do note another motivator -- folks don't want annotations to hurt
run-time performance, particularly when they are being used primarily for
pre-run-time static type checking. Which makes sense. And PEP 649 seems to
affect perfromance.

Which brings me to:

On Sun, Apr 18, 2021 at 9:24 AM Richard Levasseur 
wrote:

>  Java deals with this problem by making its annotations compile-time only
>> unless you mark them to be kept at runtime)
>>
> Maybe we could borrow that as well -- annotations could be marked to not
be used at run time at all. That could be the default.

> 2. It's my belief that the *vast *majority of annotations are unused at
> runtime, so all the extra effort in resolving an annotation expression is
> just wasted cycles. It makes sense for the default behavior to be "string
> annotations", with runtime-evaluation/retention enabled when needed.
>

exactly. And I would prefer that the specification as to whether this was a
"stringified" or not annotation at the annotation level, rather than at the
module level. It's quite conceivable that a user might want to use "run
time" annotations, in say, the specification of a Pydantic class, and
annotations just for type checking elsewhere in the module.

and then would we need the

> "o.__str_annotations__"
>>
> attribute? or even PEP 649 at all? What you'd end up with is
__annotations__ potentially containing both objects and strings that could
be type objects -- which currently works fine:

In [20]: class A:
...: x: "int"
...: y: int
...:

In [21]: A.__annotations__
Out[21]: {'x': 'int', 'y': int}

In [22]: typing.get_type_hints(A)
Out[22]: {'x': int, 'y': int}

Then the only thing that would change with PEP 563 is the default behaviour.

If I'm not mistaken, the complexity (and performance hit) of dealing with
the whole could be string, could be object, evaluate the string process
would be in the type checkers, where performance is much less of an issue.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PWHUB4YULJ33KF7HNMTLDPVK6W2RPW7V/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 563 and 649: The Great Compromise

2021-04-18 Thread Christopher Barker

On Sun, Apr 18, 2021 at 10:12 AM Larry Hastings  wrote:

> On 4/18/21 9:14 AM, Richard Levasseur wrote:
>
> Alternatively: what if the "trigger" to resolve the expression to an
> object was moved from a module-level setting to the specific expression?
> e.g.
>
> I genuinely don't understand what you're proposing.  Could you elaborate?
>
I can't speak for Richard, but Interpreted this as:

Have a way to specify, when you write the annotation, whether you want it
evaluated or kept as a string.

in my previous post, I added the idea of the semantics (am I using that
work right?) as meaning "run-time" vs "non-run time (type check time)" --
that is, do you want this to be a valid value that can be used at run time?
But it could only mean "stringify or not".

> I will note however that your example adds a lot of instances of quoting
> and curly braces and the letter 'f'.  Part of the reason that PEP 563
> exists is that users of type hints didn't like quoting them all the time.
>
I think Richard suggested the f-string because it's currently legal syntax.
And you'd get syntax checking for anything in the brackets.

But we could come up with a nicer notation, maybe a double colon ?

class A:
x: int  # you can stringify this one
y::  int # please don't stringify this one

maybe that's too subtle, but in this cse, maybe subtle is good -- to the
reader of the code they mean pretty much the same thing. To the writer,
they are quite different, but in a very testable way.

And this would preserve:

> PEP 563 meant that syntax errors would be caught at compile-time.
>
It would also open the door to extending the syntax for typing as is also
being discussed.

Granted, adding yet more syntax to Python nis a big deal, but maybe not as
big a deal as adding another dunder, or removing functionality.

Also, could a __past__ import or some such be introduced to make the new
syntax legal in older supported versions of Python?

As I think about this, I like this idea more and more. There are three
groups of folks using annotations:

1) The Static Type Checking folks: This is a large and growing and
important use case, and they want PEP 563, or something like it.

2) The "annotations ARE type objects" folks --this is a much smaller group,
at least primarily -- a much larger group is using that
functionality perhaps without realizing it, via Pydantic and the like, but
the folks actually writing that code are more select.

3) I think BY FAR the largest group: Folks using type annotations primarily
as documentation. (evidenced by a recent paper that whent through PyPi and
found a LOT of code that used type annotations that apparently was not
using a Type checker)

So it seems a good goal would be:

Have nothing change for group 3 -- the largest and probably paying the
least attention to all this.

Have things work well for group 1 -- Type Checking seems to be of growing
importance.

Require only a small manageable update for group 2 -- important, but a
smaller group of folks that would actually have to change code.
(hmm.. maybe not -- not many people write libraries like Pydantic, but all
the users of those libraries would need to update their type annotations
:-( )

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YDGLH25Z3MAXGGK2AJ44CINFPE6A2HIL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Keeping Python a Duck Typed Language.

2021-04-21 Thread Christopher Barker

Thanks Mark for posting this. I know some of us are uneasy about the pace
of the typing train 

On Tue, Apr 20, 2021 at 11:20 AM Nathaniel Smith  wrote:

> > If you guarded your code with `isinstance(foo, Sequence)` then I could
> > not use it with my `Foo` even if my `Foo` quacked like a sequence. I was
> > forced to use nominal typing; inheriting from Sequence, or explicitly
> > registering as a Sequence.
>
> You say this like it's a bad thing, but how is this avoidable, even in
> principle? Structural typing lets you check whether Foo is duck-shaped
> -- has appropriate attribute names, etc. But quacking like a duck is
> harder: you also have to implement the Sequence behavioral contract,
> and realistically the only way to know that is if the author of Foo
> tells you.
>

But that's not what duck typing is (at least to me :-) ) For a given
function, I need the passed in object to quack (and yes, I need that quack
to sound like a duck) -- but I usually don't care whether that object
waddles like a duck.

So yes, isinstance(obj, Sequence) is really the only way to know that obj
is a Sequence in every important way -- but if you only need it to do one
or two things like a Sequence, then you don't care.

And this is not uncommon -- I suspect it's very rare for a single function
to use most of the methods of a given ABC (or protocol, or whatever).

And a lot of the standard library works exactly this way. Two examples
(chosen arbitrarily, I just happen to have thought about how they work):

json.load() simply calls ``fp.read()``, and passes the result on down to
json.loads(). That's it -- no checking of anything.

If fp does not have a read() method, you get an AttributeError. If fp has a
read() method, but it returns something other than a string, then you get
some other Exception. And if it returns a string, but that string isn't
valid JSON, you get yet another kind of error.

In short, json.load(fp, ...) requires fp to have a read() method that
returns a valid JSON string. But it doesn't check, nor does it need to, if
it's getting an actual io.TextIOBase object. Is that the right one? I'm not
totally sure, which I kind of think makes my point -- I've been using
"file-like" objects for years (decades) without worrying about it.

Example 2:

The str.translate method takes:

"a mapping of Unicode ordinals to Unicode ordinals, strings, or None"

Ok, then you need to pass in a Mapping, yes? Well, no you don't. The docs
go on to say:

The table must implement lookup/indexing via __getitem__, for instance a
dictionary or list.

Ah -- so we don't need a Mapping -- we need anything indexable by an
integer that contains "ordinals, strings, or None". What the heck ABC could
we use for that?

The ABCs do have an already complex hierarchy of containers, but there is
no "Indexable", (quacks) and certainly no "indexable and returns these
particular things. (quacks a certain way). (maybe there's something in the
typing module that would work for static typing -- I have no idea).

I'm pretty sure this particular API was designed to accommodate the old py2
str.translate, which took a length-256 sequence, while also
accommodating full Unicode, which would have required a 2^32 length
sequence to do the same thing :-)

But again -- this is duck typing, built into the stdlib, and it works
just fine.

Granted, until PEP 563 (kind of) , there has been nothing that weakens or
disallows such duck typing -- those of us that want to write fully
duck-typed code can continue to do so.

But there is the "culture" of Python -- and it has been very much shifting
toward more typing -- A recent publication (sorry can't find it now -- my
google fu is failing me) examined code on PyPi and found a lot of type
hints -- many of which were apparently not being used with a static type
checker. So why were they there?

And I've seen a lot more isinstance(Some_ABC) code lately as well.

>From looking at the work of my beginning students, I can tell that they are
seeing examples out there that use more typing, to the point that they
think it's a best practice (or maybe even required?). Maybe it is -- but if
the community is moving that way, we should be honest about it.

> I'm not even sure that this *is* nominal typing. You could just as
> well argue that "the operation `isinstance(..., Sequence)` returns
> `True`" is just another of the behavioral constraints that are
> required to quack like a sequence.
>

I'm not sure of the definition of "nominal" typing -- but it absolutely is
NOT duck typing (As Luciano pointed out, Alex Martelli coined the term
Goose Typing for this).

The big distinction is whether we want to know if the object is a duck, or
if we

[Python-Dev] Re: Keeping Python a Duck Typed Language.

2021-04-22 Thread Christopher Barker

On Thu, Apr 22, 2021 at 10:47 AM Matthew Einhorn  wrote:
> In PyCharm, the above code will result in it highlighting the number `12`
with the following warning: "Type 'int' doesn't have expected attribute
'close'"

Which gives yet another use for type hints: helping out IDEs.

> If instead you add an `elif isinstance(t, str):`, under that condition
it'll auto-complete `t.` with all string properties/methods.

now this makes me nervous -- if folks start adding isinstance checks to
make their IDE more helpful , we are rally getting away from Duck Typing.

However, you could have presumably typed it as Sequence[int] and gotten all
the benefits of duck typing and IDE completion.

However, if code were to really use a duck-typed "str-like" i would produce
a failure in the type checker even if it was perfectly functional.

NOTE: str is not a great example, as it's one type that we often do need to
explicitly check -- to make the distinction between a Sequence of strings,
which a str is, and a str itself.  And str is so fundamental and complex a
type it's rarely duck-typed anyway.

-CHB

___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/EGBSQALPGCTLAPM6FLIQLDV2YD2OLVAB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PTVPRVZDZLXGLOM5WREHWQX3ZWUKO5TS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 659: Specializing Adaptive Interpreter

2021-05-13 Thread Christopher Barker

I suggest we keep it really simple, and name the implementation.

Building on Steve Holden’s suggestion:

There is broad interest in improving the performance of the cPython
runtime. (Interpreter?)

-CHB


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XBHAAZIHYPTPBOMNIYQOKZRIIDSTSKSW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The repr of a sentinel

2021-05-13 Thread Christopher Barker

There was a discussion a while back ( a year or so?? ) on Python-ideas that
introduced the idea of having more "sentinel-like" singletons in Python --
right now, we only have None.

I can't remember the context, but the consensus seemed to be that it
was easy to create a custom sentinel object, and it was not worth adding
more "official" ones to the language. But this conversation reminded me
about that, and while I do agree that we don't need more that are elevated
to the status of None, maybe it would be good to have a couple (or only
MISSING) in the standard library somewhere "central" for everyone to use.
"central" rather than in, say, dataclasses.

I'm not sure where that should be, the operator module, maybe??

Ayway, if someone were to put one of the nifty implementations being
discussed here in the stdlib --I'd use it :-)

-CHB




On Thu, May 13, 2021 at 1:14 PM Tal Einat  wrote:

> On Thu, May 13, 2021 at 8:46 PM Eric V. Smith  wrote:
> >
> >
> > On 5/13/2021 1:39 PM, Tal Einat wrote:
> > > Here is my suggestion (also posted on the related bpo-44123), which is
> > > also simple, ensures a single instance is used, even considering
> > > multi-threading and pickling, and has a better repr:
> > >
> > > class Sentinel:
> > >  def __new__(cls, *args, **kwargs):
> > >  raise TypeError(f'{cls.__qualname__} cannot be instantiated')
> > >
> > > class MISSING(Sentinel):
> > >  pass
> >
> >  >>> MISSING
> > 
> >
> > I think a repr of just "MISSING", or maybe "dataclasses.MISSING" would
> > be better.
>
> The repr uses whatever module the class is defined in, so we'd get:
>
> >>> from dataclasses import MISSING
> >>> MISSING
> 
>
> We could override that to something even cleaner with a meta-class. For
> example:
>
> class Sentinel(type):
> @classmethod
> def __prepare__(cls, name, bases, **kwds):
> d = super().__prepare__(name, bases, **kwds)
> def __new__(cls_, *args, **kwargs):
> raise TypeError(f'{cls_!r} is a sentinel and cannot be
> instantiated')
> d.update(__new__=__new__)
> return d
>
> def __repr__(cls):
> return f'{cls.__module__}.{cls.__qualname__}'
>
> Which results in:
>
> >>> from dataclasses import MISSING
> >>> MISSING
> dataclasses.MISSING
> >>> type(MISSING)
> 
> >>> MISSING()
> Traceback (most recent call last): ...
> TypeError: dataclasses.MISSING is a sentinel and cannot be instantiated
>
> - Tal
>
>
> - Tal
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ECYFQWPBQPRN4ZKDU6WNPPAG3Y5XZ2BD/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BMO6NXKJ37LIFQDBOBASEAWWDDAPWTYX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

2021-05-17 Thread Christopher Barker

> > The cost I'm concerned about is the runtime cost of worse code, because
> > the compiler can't perform some optimizations due the constraints of
> > providing the extended debug information.


Python does have an Optimized mode (-O). Granted, it’s not used very often,
but this would be a good use case for it.

-CHB



> Aah thanks for clarifying, I see what you mean now. In cases like this
> where the compiler is making optimizations, I think it is perfectly
> fine to just elide the column information. While it would be nice to
> maintain accurate columns wherever possible, you shouldn't constrain
> improvements and optimizations based on it. The traceback machinery
> will simply not print out the carets in that case and everything
> should just work smoothly.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/EB24LA7L5C35QHQTFLB6QZX26E77O6QM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GBAJSME7P7D6FS4NDCFCJRSJXN6LIYZK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 659: Specializing Adaptive Interpreter

2021-05-20 Thread Christopher Barker

I find this whole conversation confusing -- does anyone really think a
substantial performance boost to cPython is not a "good thing"? Worth the
work? Maybe not, but it seems that Mark, Guido, and MS think it is -- more
power to them!

Anyway:

potential 10% or 20% speedups in Python


I believe the folks involved think they may get a factor of two speedup --
but in any case, Oscar has a point -- there is a trade-off of effort vs
performance, and increasing the performance of cPython moves that trade-off
point, even if just a little.

I like Oscar's example, because it's got hard numbers attached to it, but
the principle is the same for any time you are considering writing, or even
using, a non-python library.


>  (2) bite the bullet and write
> C (or ctypes) that can do the calculations 100x as fast as a
> well-tuned Python program.
>

Oddly missing from this conversation is PyPy -- which can buy you a lot of
performance for some types of code in pure Python, and things like Cython
or numba, which can buy you a lot with slightly modified Python.

All those options are why Python is very useful today -- but none of them
make the case that making cPython run faster isn't a worthy goal.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3WIXL4PIWDXZPVVMQJWWXV3GNO3ED3ZC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The repr of a sentinel

2021-05-23 Thread Christopher Barker

Thanks Tal for writing this up.

A couple comments:

1) “Add a single new sentinel value, e.g. MISSING or Sentinel” (under
rejected)

I was one of the proponent of that -- but not as an alternative to having a
stadardars way to create unique sentinels, but as an addition. That's kind
of orthogonal to the PEP, but it would still be nice to have it at the same
time.

Why? There was a fair bit of discussion as to why you would not want to
require everyone to use  MISSING (dataclasses, in particular), but I still
think better for the readability of everyone's code for the common case(s?)
to use the same singleton, if they can.

2) couldn't a factory function simply return a sentinel subclass? Maybe I'm
missing something, but I can't see why that would require stack frame
inspection.

But frankly Luciano's idea of a base class that can be subclassed seems the
most startightford to me.

-CHB
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7VUT5PLMXBGXHJAWCPQSHX774QLTNSG4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Roundup to GitHub Issues migration

2021-06-21 Thread Christopher Barker

> By contrast, requiring a github account for reporting bugs also makes
> python an unwelcoming place for non-developers in general. Github is a
> developers' social network, "mere" users are much less likely to want to
> be part of it. Many will just silently abandon their bug report.


But you don’t need to be “part of it” in any meaningful way. One only needs
to create an account, which could be quite anonymous, and even temporary.

And is no harder, and probably easier, than creating an account on a
Python-specific site.

Also: cPython is a large, complex, and mature project. I don't think many
non-developers can even identify a true bug, much less write a helpful big
report. There are many other ways to be involved in and contribute to the
Python community that don't require a gitHub (or any) account.

I understand the issue here — I feel that way about businesses that use
Facebook for their website. But in that case, I can’t even read it without
a Facebook account. I don’t mind needing an account to contribute to a
conversation.

And while GitHub  has become the dominant player in Open Source
development— it has not (yet?) reached out to control much else.

-CHB




> Cheers,
> Baptiste
>
> >
> > Victor
> >
> > On Mon, Jun 21, 2021 at 11:46 AM Baptiste Carvello
> >  wrote:
> >>
> >> Hi,
> >>
> >> Le 21/06/2021 à 04:20, Ezio Melotti a écrit :
> >>> This effort is being tracked at
> >>> : this board reflects
> >>> the current status of the project.  The PEPs (including PEP 588 --
> >>> GitHub Issues Migration Plan) haven't been updated yet and might
> >>> contain outdated information, so please refer to the psf/gh-migration
> >>> repo for the latest updates.
> >>
> >> since 2019, I've been waiting for proposed solutions to PEP 588's first
> >> open issue ("A GitHub account should not be a requirement"). Now would
> >> be the time to think about it, but I see no such reflection mentioned in
> >> the repo.
> >>
> >> Cheers,
> >> Baptiste
> >> ___
> >> Python-Dev mailing list -- python-dev@python.org
> >> To unsubscribe send an email to python-dev-le...@python.org
> >> https://mail.python.org/mailman3/lists/python-dev.python.org/
> >> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/2GJQKB27XD6LKGKWJXTXLEURNMF5ZJT7/
> >> Code of Conduct: http://python.org/psf/codeofconduct/
> >
> >
> >
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ETQO4M3IUCXATK2UL56PSYE6FZPNZHIY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6WC5YBPUGL2NDWO26FBIQM2PBEUPV3V3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: cmd.Cmd.prompt should be an instance attribute

2021-06-26 Thread Christopher Barker

A class attribute provides a default that’s the same for all instances, but
does let you customize it with a simple attribute assignment. Which seems
like the right thing in this case.

class Cmd:
> PROMPT = '> '
>
> @property
> def prompt(self) -> str:
> return self.PROMPT


This makes .prompt a read-only instance attribute. So you then wouldn’t be
able to change it for a given instance.

Why make it an instance attribute if all instances will have the same one?
Unless you want to disallow overriding it in an instance.

If the reason is to make MyPy happy, then either there’s a big in your
code, or MyPy is being overly pedantic.

NOTE: accessing class attributes via self is standard practice — after all
methods are simply callable class attributes :-)

-CHB




> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/Q7TSFJD4EPUXXJFT5TMBUODIVL4CLR4O/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GAU3BJFQ2OYLQSRPIWZKH7CKFCK76L32/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is the Python review process flawed?

2021-07-01 Thread Christopher Barker

Just to add a comment from someone who's been around Python a long time --
a (very) occasional contributor, never a maintainer.

I have every sympathy for the core maintainers -- Python is a very large
and complex project that is relied on by an enormous number of people.

The size and complexity (and age of the code base) make it a real challenge
to maintain, and the size of the user community makes the consequences of a
regression massive.

> Tests make us a little more certain, but it is still a guess.

Exactly -- I am a big fan of unit testing and TDD. But whenever I teach
about unit testing, I always say:

"comprehensive tests are a fantasy"

Think of how hard it can be to even get 100% coverage in your tests. Then
think about how the fact that every line of code was run in no way means
every function has been run with every possible input. Then think about the
fact that cPython is a language implementation -- everything in the
interpreter is being driven at another level by arbitrarily complex Python
code.

I think it's a near miracle that anything gets updated or fixed without
major regressions!

Finally -- cPython has been around a long time, and used an enormous amount
-- any existing bugs have been avoided or worked around already by
thousands of projects -- almost by definition, ANY regression is worse than
any bug that still exists today (Security issues being an exception).

Obligatory XKCD:

https://xkcd.com/1172/

Thanks to all contributors and maintainers,

-CHB

PS: All that being said, we, as a community, could do better. For instance,
someone like me could do high-level triage on bug reports -- I need to set
aside some time to do that.


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/32BS57QPQJA7PFGY6J3NYRB44K2QJQLM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 467 feedback from the Steering Council

2021-08-04 Thread Christopher Barker

I see in the PEP:

"the bchr builtin is to recreate the ord/chr/unichr trio from Python 2
under a different naming scheme"

Why recreate that trio? Shouldn't we be moving away from the
bytes-is-a-string concept here?

A byte is not a character -- why would the function that creates a byte
from an integer value be called bchr()? (short for "byte character",
presumably)

There are fewer and fewer people having to translate their code (or their
brains) from py2 to py3.

bytes.fromint() is just fine.

-CHB

BTW -- I really love the rest of the PEP -- it's been too awkward to work
with bytes for too long.



On Wed, Aug 4, 2021 at 9:43 AM Barry Warsaw  wrote:

> On Aug 4, 2021, at 07:31, Victor Stinner  wrote:
> >
> > On Tue, Aug 3, 2021 at 7:54 PM Ethan Furman  wrote:
> >> I would rather keep `bchr` and lose the `.fromint()` methods.
> >
> > I would prefer to only have a bytes.byte(65) method, no bchr()
> > built-in function. I would prefer to keep builtins namespace as small
> > as possible.
>
> The Steering Council is also pretty adamantly against adding a new bchr()
> built-in.
>
> > bytes.byte() name is similar to bytes.getbyte(). I cannot find "int"
> > in the name of other bytes methods.
>
> .byte() seems fine to me too.  I’m not a fan of smushedwords but
> .fromint() seemed better than .fromord().
>
> -Barry
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CPZTRWIWLRKTBHQS6UPY63BEZCV3FDZZ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XPJUILIFBTYMAEQYJVDBQMC3GJI6XX6M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 467 feedback from the Steering Council

2021-08-10 Thread Christopher Barker

On Tue, Aug 10, 2021 at 3:00 PM  wrote:

> The history of bytes/bytearray is a dual-purpose view.  It can be used in
> a string-like way to emulate Python 2 string handling (hence all the usual
> string methods and a repr that displays in a string-like fashion).  It can
> also be used as an array of numbers, 0 to 255 (hence the list methods and
> having an iterator of ints).  ISTM that the authors of this PEP reject or
> want to discourage the latter use cases.
>

I didn't read it that way, but if so, please no, I"d rather see the former
use cases discouraged. ISTM that the Py2 string handling is still needed
for working with mixed binary / text data -- but that should be a pretty
specialized use case. spelling the way to create a byte, byte() sure makes
more sense in any other context.

> ... anything where a C programmer would an array of unsigned chars).
>

or any programmer would use an array of unsigned 8bit integers :-) numpy
spells it: `np.uint8`, and the the type in the C99 stdint.h is `uint8_t`.
My point is that for anyone not an "old time" C programmer, or even a
Python2 programmer, the  "character is an unsigned 8 bit int" concept is
alien and confusing, not a helpful mnemonic.

> For example, creating a single byte with bytes([0x1f]) isn't pleasant,
> obvious, or fast.
>

no, though bytes([31]) isn't horrible ;-)   (despite coding for over four
decades, I'm still not comfortable with hex notation)

I say it's not horrible, because bytes is a Sequence of bytes (or integer
values between 0 and 255), initializing it with an iterable seems pretty
reasonable, that's how we initialize most (all?) other sequences after all.
And compatible with array.array and numpy arrays.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RM4JHK4GIKYYWV7J5F6IQJ66KUIXWMMF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Should PEP 8 be updated for Python 3 only?

2021-08-24 Thread Christopher Barker

I just noticed that PEP 8 has quite a few references to Python 2. Perhaps
it's time to remove those.

There are more and more folks that have never learned or used Python 2 --
having those notes in PEP 8 is just confusing.

And for folks that are still maintaining Py 2 code --  it's probably too
late for PEP 8 for those code bases anyway.

If I make a PR, will it be considered?

-CHB


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MK7N36L3XCXVY5R4DIIYIDU2W7EGJKZ3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should PEP 8 be updated for Python 3 only?

2021-08-24 Thread Christopher Barker

On Tue, Aug 24, 2021 at 9:53 PM Guido van Rossum  wrote:

> Since PEP 8 is a "living document" that is occasionally updated, I think
> that's reasonable. Don't go overboard. I found 10 occurrences of the string
> "python 2" in the text. You might also want to strike the paragraph that
> links to reference [6].
>

will do -- hopefully I'll find some time for this tomorrow.

-CHB




> On Tue, Aug 24, 2021 at 9:42 PM Christopher Barker 
> wrote:
>
>> I just noticed that PEP 8 has quite a few references to Python 2. Perhaps
>> it's time to remove those.
>>
>> There are more and more folks that have never learned or used Python 2 --
>> having those notes in PEP 8 is just confusing.
>>
>> And for folks that are still maintaining Py 2 code --  it's probably too
>> late for PEP 8 for those code bases anyway.
>>
>> If I make a PR, will it be considered?
>>
>> -CHB
>>
>>
>> --
>> Christopher Barker, PhD (Chris)
>>
>> Python Language Consulting
>>   - Teaching
>>   - Scientific Software Development
>>   - Desktop GUI and Web Development
>>   - wxPython, numpy, scipy, Cython
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/MK7N36L3XCXVY5R4DIIYIDU2W7EGJKZ3/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
>


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DRYMTFYBEUYF2KFNHXYUINYSTS5AZ7YM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should PEP 8 be updated for Python 3 only?

2021-08-25 Thread Christopher Barker

I'm working on a PR now. It seems there is little support for keeping the
python2 content in the docs, so I'm re-writing it as though it was never
there. If someone wants to add a note about Python 2, of course that can be
added later.

Note that "moving the Python 2 content to a section at the end" is not all
that straightforward, as it is pretty mixed in with the text at this point.

But now a question -- the current text reads:

"Code in the core Python distribution should always use UTF-8"

and then:

"In the standard library, non-default encodings should be used only for
test purposes or when a comment or docstring needs to mention an author
name that contains non-ASCII characters ..."

I *think* that's a remnant of the Py2 ASCII encoding days -- but I wanted
to make sure, a bit later on, it says:

"The following policy is prescribed for the
standard library ... In addition, string literals and comments must also be
in ASCII."

Is that still correct for string literals and comments? And what
about docstrings?

It seems to me that if we really are utf-8, then there is no need for those
"textual" elements to be ASCII. e.g they can still contain non-ascii
characters, and escaping those makes things less readable, not more.

So I think that section should now read:

"""
Source File Encoding


Code in the core Python distribution should always use UTF-8, and should
not have an encoding declaration.

In the standard library, non-UTF-8 encodings should be used only for
test purposes.

The following policy is prescribed for the standard library (see PEP
3131): All identifiers in the Python standard library MUST use
ASCII-only identifiers, and SHOULD use English words wherever feasible
(in many cases, abbreviations and technical terms are used which aren't
English). In comment and docstrings, authors whose names tht are not
based on the Latin alphabet (latin-1, ISO/IEC 8859-1 character set)
MUST provide a transliteration of their names in this character set.

Open source projects with a global audience are encouraged to adopt a
similar policy.
"""

But maybe we do want to keep comments, docstrings and literals as ASCII
with escapes?

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VGNL46HSGMCCQM466324INLM46HI47CS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Notes on PEP 8

2021-08-25 Thread Christopher Barker

As I was working on removing Python 2 references from PEP 8 (PR:
https://github.com/python/peps/pull/2059), I tried to avoid any other
copy-editing.

However, I noted a few things that maybe could use some attention:

stdlib or not?
##

Right at the top, it says:

"This document gives coding conventions for the Python code comprising the
standard library in the main Python distribution."

However, it has become a de facto standard for all Python code, and in the
document itself, there is frequent wording akin to "Identifiers used in the
standard library must be ASCII compatible ...", and even advice for third
party libraries.

Which I think is acknowledging that PEP 8 is indeed not only about the
standard library.

So maybe there should be a bit of text about that at the top.

Name Mangling
#

In the sections on "Method Names and Instance Variables" and "Designing for
Inheritance", the discussion of name mangling (leading __ names) is a bit
redundant. Nothing incorrect or misleading -- just a copy-editing issue.

Maybe I'll do a copy-editing PR 

-CHB


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WCTRPSZSC2XO4NG5CKMOBLK2P7UUY7RE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should PEP 8 be updated for Python 3 only?

2021-08-25 Thread Christopher Barker

OK,

PR here:

https://github.com/python/peps/pull/2059

Interesting some of the cruft in there e.g.  still referring to "new style
classes" :-)

I'll leave it to the PEP editors to decide about a note about Python 2.

-CHB


On Wed, Aug 25, 2021 at 10:28 PM Stephen J. Turnbull <
stephenjturnb...@gmail.com> wrote:

> Terry Reedy writes:
>
>  > We could add (perhaps at the end of the first paragraph) something like
>  > "(Since the 2.x stdlib is frozen, all 2.x-specific guidelines were
>  > removed in Sept 2021.)"  Anyone interested could then check git log for
>  > the last commit before then.
>
> How about including the most recent 'main' SHA1 (where a 6-character
> prefix should do, and "the" refers to Chris's clone, we don't need to
> be hugely fussy about the exact previous commit when it eventually
> gets merged), or even tag it?
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/TWNYNKGCCMGNK4CXZGVZBCGCPQL7HMYF/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6FBGNA23EYENXC5OIBVU3TXPZOEZ5SA7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should PEP 8 be updated for Python 3 only?

2021-08-26 Thread Christopher Barker

Thanks all.

The PR is in process, and I believe it includes everything brought up here.

If you have any more thoughts, please post them there.

-CHB

On Thu, Aug 26, 2021 at 1:54 AM Petr Viktorin  wrote:

> On 26. 08. 21 9:54, Marc-Andre Lemburg wrote:
> > On 26.08.2021 06:07, Christopher Barker wrote:
> >> I'm working on a PR now. It seems there is little support for keeping
> the
> >> python2 content in the docs, so I'm re-writing it as though it was
> never there.
> >> If someone wants to add a note about Python 2, of course that can be
> added later.
> >>
> >> Note that "moving the Python 2 content to a section at the end" is not
> all that
> >> straightforward, as it is pretty mixed in with the text at this point.
> >>
> >> But now a question -- the current text reads:
> >>
> >> "Code in the core Python distribution should always use UTF-8"
> >>
> >> and then:
> >>
> >> "In the standard library, non-default encodings should be used only for
> >> test purposes or when a comment or docstring needs to mention an author
> >> name that contains non-ASCII characters ..."
> >>
> >> I *think* that's a remnant of the Py2 ASCII encoding days -- but I
> wanted to
> >> make sure, a bit later on, it says:
> >>
> >> "The following policy is prescribed for the
> >> standard library ... In addition, string literals and comments must
> also be in
> >> ASCII."
> >
> > For Python 2 code we mandated ASCII for the stdlib, with some exceptions
> > using the source code encoding for testing purposes or in case e.g.
> > Martin von Löwis or Marc-André Lemburg wanted to put his name into the
> code
> > without escaping part of it ;-)
> >
> > Note that Python 2 defaults to ASCII as source code encoding.
> >
> > With UTF-8 as standard source code encoding, this is no longer
> > necessary.
> >
> > So the second quote can be changed to "In the standard library,
> non-default
> > source code encodings should be used only for test purposes ...".
> >
> >> Is that still correct for string literals and comments? And what
> about docstrings?
> >>
> >> It seems to me that if we really are utf-8, then there is no need for
> those
> >> "textual" elements to be ASCII. e.g they can still contain non-ascii
> characters,
> >> and escaping those makes things less readable, not more.
> >>
> >> So I think that section should now read:
> >>
> >> """
> >> Source File Encoding
> >> 
> >>
> >> Code in the core Python distribution should always use UTF-8, and
> should not
> >> have an encoding declaration.
> >>
> >> In the standard library, non-UTF-8 encodings should be used only for
> >> test purposes.
> >
> > I think the above should be limited to Python code. In C or other
> > source files you may well still need a source code encoding.
> >
> >> The following policy is prescribed for the standard library (see PEP
> >> 3131): All identifiers in the Python standard library MUST use
> >> ASCII-only identifiers, and SHOULD use English words wherever feasible
> >> (in many cases, abbreviations and technical terms are used which aren't
> >> English). In comment and docstrings, authors whose names tht are not
> >> based on the Latin alphabet (latin-1, ISO/IEC 8859-1 character set)
> >> MUST provide a transliteration of their names in this character set.
> >>
> >> Open source projects with a global audience are encouraged to adopt a
> >> similar policy.
> >> """
> >>
> >> But maybe we do want to keep comments, docstrings and literals as ASCII
> with
> >> escapes?
> >
> > No need for the stdlib, since UTF-8 is widely accepted by now
> > and why should people with non-ASCII names not be able to write
> > their true name ?
> >
> > You may have noted that I rarely do... the reason is that in the
> > past, the accent on the "e" caused me too many problems. Perhaps
> > one of these days, I'll go back to adding it again :-)
>
> I would drop the weirdly specific "(latin-1, ISO/IEC 8859-1 character
> set)" note, and only keep "based on the Latin alphabet".
> The Ł in Łukasz's name is not in latin-1, and I don't think it needs
> different treatment than German or French names. (As opposed to a
> R

[Python-Dev] Re: PEP 467 feedback from the Steering Council

2021-09-08 Thread Christopher Barker

What am I missing?

The integers between 0 and 255 map directly to a particular byte value.

But any other integer could be expressed as a wide variety of multiple byte
combinations.

The proposal here covers byte-order, but what about 16 vs 32 vs 64 bits?
Unsigned vs signed?

I thought that’s what the struct module is for.

There is the byte representation of Python’s bignum, but is that consistent
across platforms and implementations?
(Micropytjon, PyPy, IronPython, Jython)

And even if so, is it useful?

NOTE: my objection to “bchr”, whether as a builtin or not is not the
functionality, it’s the name. Equating a byte with a character is a legacy
of C ( and Python 2” — in Python 3, they are completely distinct concepts.
Yes, that is serious bike-shedding :-)

-CHB

On Wed, Sep 8, 2021 at 10:16 AM Barry Scott  wrote:

>
> On 8 Sep 2021, at 06:39, Steven D'Aprano  wrote:
>
>
> On Tue, Sep 07, 2021 at 08:09:33PM -0700, Barry Warsaw wrote:
>
> I think Nick is on board with bytes.fromint() and no bchr(), and my
> sense of the sentiment here is that this would be an acceptable
> resolution for most folks.  Ethan, can you reconsider?
>
>
> I haven't been completely keeping up with the entire thread, so
> apologies if this has already been covered. I assume that the idea is
> that bytes.fromint should return a single byte, equivalent to chr()
> returning a single character.
>
> To me, it sounds like should be the opposite of int.from_bytes.
>
> int.from_bytes(b'Hello world', 'little')
>
>121404708502361365413651784
>
> bytes.from_int(121404708502361365413651784, 'little')
>
># should return b'Hello world'
>
>
> :>>> int.from_bytes(b'\x00\x00\x00\x01', byteorder='big')
> 1
> :>>> bytes.from_int(1)
> would return b'\x01'? Without a length it cannot return
> b'\x00\x00\x00\x01'
>
> Barry
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/TTFJ4VP5PCR557VHEH5LPSWAPNPE4QQU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PANHM75VNG2SX4FUZNKHELKVSDFNFNCT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 467 feedback from the Steering Council

2021-09-09 Thread Christopher Barker

I fully admit serious bikeshedding here, but:

I'm starting to think the name should be `bytes.bchr` -- it avoids any
> confusion with the `int.to_bytes` and
> `int.from_bytes` methods,


are they so different? :-)

In [23]: x.to_bytes(1, 'little')
Out[23]: b'A'

In [27]: int.from_bytes(b'A', 'little')
Out[27]: 65

you can think of this as a useful specific case of the int methods.

(by the way, why do I need to specify the byte order for a 1 byte int? Yes,
I know, it's always a required parameter -- though that 's one reason to
have the special case easily available)

and is an appropriate name for the target domain (where bytes are treated
> as characters).
>

Is that the target domain? Yes, it's an important use case, but
certainly not the only one, and frankly kind of a specialized use case
actually. If you are working with characters (text) in Python 3, you should
be using the str type.

Using bytes for general text (even if you know the text at hand is all
ASCII) is not recommended. It is useful to use bytes for the specialized
use case of mixed text and binary data (which, by the way, I have had to
do) but I don't think we should say that particular use case is what bytes
are  targeted for. Anyone doing that should know what they are doing :-)

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S3Q6NYRXHKUQLMP7WCQMCEEK3MCCGKNJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Regressions caused the recent work on ceval.c and frame objects

2021-09-19 Thread Christopher Barker

Will all packages that use Cython have to upgrade Cython to work with 3.10?

That can be a pretty heavy lift. A lot of us pin Cython.

-CHB


On Sun, Sep 19, 2021 at 4:32 PM Miro Hrončok  wrote:

> On 20. 09. 21 0:10, Thomas Grainger wrote:
> > Are projects that ship pre-compiled wheels impacted? Eg
> twisted-iocpsupport ?
>
> I guess that if they managed to compile with 3.10, they shouldn't be.
>
> --
> Miro Hrončok
> --
> Phone: +420777974800
> IRC: mhroncok
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/TDGG4VFRZO5CDKTUH4AXYDLVXOKNLZZE/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NVVWCZBZLO2T26VQLGLH34TTOHCCHR7F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Type annotations, PEP 649 and PEP 563

2021-10-20 Thread Christopher Barker

Thanks to the SC for such a thoughtful note. I really like where this is
going.

One thought.

On Wed, Oct 20, 2021 at 6:21 AM Thomas Wouters  wrote:

>
>1.
>
>Is the performance of PEP 649 and PEP 563 similar enough that we can
>outright discount it as a concern? Does anyone actually care about the
>overhead of type annotations anymore? Are there other options to alleviate
>this potential issue (like a process-wide switch to turn off annotations)?
>
>
Annotations are used at runtime by at least one std lib module:
dataclasses, and who knows how many third party libs. So that may not be
practical.

-CHB
-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4C3G5M2OUCFHNLHJ6HHBQWLYOOT3SKXD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Type annotations, PEP 649 and PEP 563

2021-10-21 Thread Christopher Barker

On Thu, Oct 21, 2021 at 5:24 PM Steven D'Aprano  wrote:

> Runtime type checkers already have to deal with forward refs that are
> strings, as this is legal, and always will be:
>
> def function(arg:'Spam') -> Any: ...
>
> so we're not putting any extra burden on them. And we had already
> agreed to implicitly use strings for annotations.
>

I'll take your word for it. However, other runtime uses for annotations may
not already need to support strings as types.

Pydantic is the classic example. I'm not entirely sure how it does its
thing, but I have code built on dataclasses that is likely similar -- it
expects the annotation to be an actual type object -- an actual class, one
that can be instantiated, not even the types that are in the typing module,
and certainly not a string that needs to be evaluated.

So having them sometimes, maybe sometimes be strings would be a big pain in
the @$$

On the other hand, having them be some odd "ForwardReference" type, rather
than a NameError might be OK -- as long as SOME exception was raised as
soon as I tried to use it. Though it would make for far more confusing
error messages :-(

My first choice would be to get a NameError at module load time, like we do
now. Second would be a NameError as soon as it is accessed. Getting a
special value is OK though, now that I'm thinking about it, I could
probably put that special case code in one place, and provide a nice error
message.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WAE5LTKPBCEEBPR3ZHRGWJEGIEQ6RI6G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Type annotations, PEP 649 and PEP 563

2021-10-22 Thread Christopher Barker

> Any other runtime annotation tool has to support strings, otherwise the
> "from __future__ import annotations" directive will have already broken
> it.


Exactly- isn’t that it was deferred in 3.10, and may never be implemented?
I’ll leave it to the Pydantic developers to discuss that, but they were
pretty clear that PEP 563 would break things.

If the tool does type-checking, then it should support stringified
> annotations.


I guess I wasn’t clear — my point was there are uses for annotations that
are NOT type checking.

And my understanding of what the SC said was that they want those uses to
continue to be supported.

-CHB


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J2UWKGIGKA7IKYBSKL3DNDWPO3AXH7SF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

2021-10-28 Thread Christopher Barker

Thanks  Skip — nice to see some examples.

Did you try running  the same code with stock Python?

One reason I ask is the IIUC, you are using numpy for the individual
 vector operations, and numpy already releases the GIL in some
circumstances.

It would also be fun to see David Beezley’s example from his seminal talk:


https://youtu.be/ph374fJqFPE

-CHB



On Thu, Oct 28, 2021 at 3:55 AM Skip Montanaro 
wrote:

> Guido> To be clear, Sam’s basic approach is a bit slower for
> single-threaded code, and he admits that. But to sweeten the pot he has
> also applied a bunch of unrelated speedups that make it faster in general,
> so that overall it’s always a win. But presumably we could upstream the
> latter easily, separately from the GIL-freeing part.
>
> Something just occurred to me. If you upstream all the other goodies
> (register VM, etc), when the time comes to upstream the no-GIL parts won't
> the complaint then be (again), "but it's slower for single-threaded
> code!" ? ;-)
>
> Onto other things. For about as long as I can remember, the biggest knock
> against Python was, "You can never do any serious multi-threaded
> programming with it. It has this f**king GIL!" I know that attempts to
> remove it have been made multiple times, beginning with (I think) Greg
> Smith in the 1.4 timeframe. In my opinion, Sam's work finally solves the
> problem.
>
> Not being a serious parallel programming person (I have used
> multi-threading a bit in Python, but only for obviously I/O-bound tasks), I
> thought it might be instructive — for me, at least — to kick the no-GIL
> tires a bit. Not having any obvious application in mind, I decided to
> implement a straightforward parallel matrix multiply. (I think I wrote
> something similar back in the mid-80s in a now defunct Smalltalk-inspired
> language while at GE.) Note that this was just for my own edification. I
> have no intention of trying to supplant numpy.matmul() or anything like
> that. It splits up the computation in the most straightforward (to me)
> way, handing off the individual vector multiplications to a variable
> sized thread pool. The code is here:
>
> https://gist.github.com/smontanaro/80f788a506d2f41156dae779562fd08d
>
> Here is a graph of some timings. My machine is a now decidedly
> long-in-the-tooth Dell Precision 5520 with a 7th Gen Core i7 processor
> (four cores + hyperthreading). The data for the graph come from the
> built-in bash time(1) command. As expected, wall clock time drops as you
> increase the number of cores until you reach four. After that, nothing
> improves, since the logical HT cores don't actually have their own ALU
> (just instruction fetch/decode I think). The slope of the real time
> improvement from two cores to four isn't as great as one to two, probably
> because I wasn't careful about keeping the rest of the system quiet. It was
> running my normal mix, Brave with many open tabs + Emacs. I believe I used
> A=240x3125, B=3125x480, giving a 240x480 result, so 15200 vector multiplies.
> .
>
> [image: matmul.png]
>
> All-in-all, I think Sam's effort is quite impressive. I got things going
> in fits and starts, needing a bit of help from Sam and Vadym Stupakov
> to get the modified numpy implementation (crosstalk between my usual Conda
> environment and the no-GIL stuff). I'm sure there are plenty of problems
> yet to be solved related to extension modules, but I trust smarter people
> than me can solve them without a lot of fuss. Once nogil is up-to-date with
> the latest 3.9 release I hope these changes can start filtering into main.
> Hopefully that means a 3.11 release. In fact, I'd vote for pushing back the
> usual release cycle to accommodate inclusion. Sam has gotten this so close
> it would be a huge disappointment to abandon it now. The problems faced at
> this point would have been amortized over years of development if the GIL
> had been removed 20 years ago. I say go for it.
>
> Skip
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/WBLU6PZ2RDPEMG3ZYBWSAXUGXCJNFG4A/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CR2B3H2WKE6CEHUT22P263F2F4L7F3FU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

2021-10-31 Thread Christopher Barker

On Fri, Oct 29, 2021 at 6:10 AM Skip Montanaro 
wrote:

> 1. I use numpy arrays filled with random values, and the output array is
> also a numpy array. The vector multiplication is done in a simple for loop
> in my vecmul() function.
>

probably doesn't make a difference for this exercise, but numpy arrays make
lousy replacements for a  regular list -- i.e. as a container alone. The
issue is that floats need to be "boxed" and "unboxed" as you put them in
and pull them out of an array. whereas with lists, they float objects
themselves are already there.

OK, maybe not as bad as I remember. but not great:

In [61]: def multiply(vect, scalar, out):
...: """
...: multiply all the elements in vect by a scalar in place
...: """
...: for i, val in enumerate(vect):
...: out[i] = val * scalar
...:

In [62]: arr = np.random.random((10,))

In [63]: arrout = np.zeros_like(arr)

In [64]: l = list(arr)

In [65]: lout = [None] * len(l)

In [66]: %timeit multiply(arr, 1.1, arrout)
19.3 ms ± 125 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [67]: %timeit multiply(l, 1.1, lout)
12.8 ms ± 83.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

> That said, I have now run my example code using both PYTHONGIL=0 and
PYTHONGIL=1 of Sam's nogil branch as well as the following other Python3
versions:

* Conda Python3 (3.9.7)
* /usr/bin/python3 (3.9.1 in my case)
* 3.9 branch tip (3.9.7+)

The results were confusing, so I dredged up a copy of pystone to make sure
I wasn't missing anything w.r.t. basic execution performance. I'm still
confused, so will keep digging.

I'll be interested to see what you find out :-)

It would also be fun to see David Beezley’s example from his seminal talk:
>
> https://youtu.be/ph374fJqFPE
>

Thanks, I'll take a look when I get a chance

That may not be the best source of the talk -- just the one I found first
:-)

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GY7RWKFOPQFGTGD7IUN5JS6FYNXYM22I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

2021-11-01 Thread Christopher Barker

> I think the performance difference is because of different versions of
> NumPy.
>

Good reason to leave numpy completely out of it. Unless you want to test
 nogil’s performance effects on numpy code — an interesting exercise in
itself.

Also — sorry I didn’t look at your code before, but you really want to keep
the generation of large random arrays out of your benchmark if you can. I
suspect that’s what’s changed in numpy versions.

In any case, do time the random number generation…

-CHB



Python 3.9 installs NumPy 1.21.3 by default for "pip install numpy". I've
> only built and packaged NumPy 1.19.4 for "nogil" Python. There are
> substantial performance differences between the two NumPy builds for this
> matmul script.
>
> With NumPy 1.19.4, I get practically the same results for both Python
> 3.9.2 and "nogil" Python for "time python3 matmul.py 0 10".
>
> I'll update the version of NumPy for "nogil" Python if I have some time
> this week.
>
> Best,
> Sam
>
> On Sun, Oct 31, 2021 at 5:46 PM Skip Montanaro 
> wrote:
>
>> > Remember that py stone is a terrible benchmark.
>>
>> I understand that. I was only using it as a spot check. I was surprised
>> at how much slower my (threaded or unthreaded) matrix multiply was on nogil
>> vs 3.9+. I went into it thinking I would see an improvement. The
>> Performance section of Sam's design document starts:
>>
>> As mentioned above, the no-GIL proof-of-concept interpreter is about 10%
>> faster than CPython 3.9 (and 3.10) on the pyperformance benchmark suite.
>>
>>
>> so it didn't occur to me that I'd be looking at a slowdown, much less by
>> as much as I'm seeing.
>>
>> Maybe I've somehow stumbled on some instruction mix for which the nogil
>> VM is much worse than the stock VM. For now, I prefer to think I'm just
>> doing something stupid. It certainly wouldn't be the first time.
>>
>> Skip
>>
>> P.S. I suppose I should have cc'd Sam when I first replied to this
>> thread, but I'm doing so now. I figured my mistake would reveal itself
>> early on. Sam, here's my first post about my little "project."
>> https://mail.python.org/archives/list/python-dev@python.org/message/WBLU6PZ2RDPEMG3ZYBWSAXUGXCJNFG4A/
>>
>>
>> --
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VLSAMFORVMEIQVH3UH6LOK3OA3GL7C6J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Having Sorted Containers in stdlib?

2021-11-09 Thread Christopher Barker

Maybe a stupid question:

What are use cases for sorted dicts?

I don’t think I’ve ever needed one.

Also, I can’t quite tell from the discussion If a “sorted dict” implements
something new, or is an internal data structure that gives better
performance for particular use cases. I.e. is a sorted dict a Mapping?

-CHB



On Tue, Nov 9, 2021 at 9:45 PM Dan Stromberg  wrote:

>
> On Tue, Nov 9, 2021 at 9:00 PM Steven D'Aprano 
> wrote:
>
>> Sorting dicts has been discussed on the Python-Ideas mailing list, it is
>> too hard and expensive to justify for the limited use-cases for it. If
>> you want to sort a dict, you are best to sort the dict's keys, then
>> create a new dict. Or possibly use a dedicated Sorted Mapping type like
>> a red-black tree or similar.
>>
>
> There are several implementations of key-order sorted dicts available in
> Python, and more than a few of them are well tested.  I am the maintainer
> of one of them: https://pypi.org/project/treap/ which I ported from
> Java.  I also did a performance comparison among several of them a few
> years back.
>
> Red-black trees are popular, perhaps especially among Java developers, but
> I've never understood why.  They aren't the fastest.  Among traditional
> tree-based key-sorted dicts, treaps are frequently fastest.
>
> However, SortedDict is not uncommonly faster than treaps - I believe this
> is because SortedDict is very good at maximizing locality of reference.
> Traditional trees are almost always going to do an entire cache line hit
> for every node in large trees, even if those nodes are "next to each other"
> when sorted/traversed.  SortedDicts put keys that are nearly equal, in
> nearly the same part of memory - so multiple values can be retrieved with a
> single cache line hit.
>
> Sorting a dict's keys inside a loop tends to give O(n^2) algorithms, and
> sometimes even O(n^2 * logn).  This is not good.  A traditional tree should
> give O(nlogn) algorithms under similar circumstances, and although my gut
> is telling me SortedDict is similar the presentation linked below suggests
> a different (though still better than sorting in a loop) runtime for
> SortedDict.
>
> It's true that key-sorted dicts are not all that common.  Their most
> common use is probably for implementing finite caches - evictions can
> benefit from ordered keys.  However, we already have functools.lru_cache.
>
> Here's my most recent performance comparison:
> https://stromberg.dnsalias.org/~strombrg/sorted-dictionary-comparison/Datastructure%20comparison.pdf
>
> Here's a PyCon 2016 presentation about SortedContainers, that includes
> SortedDict:
> https://www.youtube.com/watch?v=7z2Ki44Vs4E
>
> I think it would make sense to include at least one key-sorted dict in
> CPython, and feel that SortedDict would not be a bad way to go.  Neither
> would treap.
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/VNK4VPGA4LJH7RMR4LCCACEG2WNDBBFO/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U7FQIE5NDWY3AQI5QDMIJSIKVIQ3YUPC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Having Sorted Containers in stdlib?

2021-11-10 Thread Christopher Barker

On Tue, Nov 9, 2021 at 11:05 PM Paul Bryan  wrote:

> On Tue, Nov 09, 2021 at 10:01:35PM -0800, Christopher Barker wrote:
>
> What are use cases for sorted dicts?
>
>
Good question :-)

It could be handy for deterministic iteration of its values, for example to
>  allow serialized values to be easily compared, or to generate and verify a
> signatures.
>

Yup -- I've done that. But for that, it's fine to sort when you are doing
the comparing / serialization. What I haven't been able to imagine is a use
case for keeping a
Mapping sorted constantly as you insert / remove items. For that you'd need
a use case where you were making sorted queries of various sorts. I
noticed, for instance, that the Java implementation posted earlier had
methods like "all the items after this one" (can't remember how that was
spelled).

I'm sure there are use cases for this, I'm probably lacking imagination :-)

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/X7CGUBYQ3PEIYNMYAG3Z4SMXWILR5BKT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Having Sorted Containers in stdlib?

2021-11-11 Thread Christopher Barker

Earlier in the thread, we were pointed to multiple implementations.

Is this particular one clearly the “best”[*]?

If so, then sure.

-CHB

[*] best meaning “most appropriate for the stdlib”. A couple folks have
already pointed to the quality of the code. But my understanding is that
different algorithms are more or less appropriate for different use cases.
So is this one fairly “universal”?



On Thu, Nov 11, 2021 at 4:31 AM Paul Moore  wrote:

> On Thu, 11 Nov 2021 at 11:51, Antoine Pitrou  wrote:
> >
> > On Wed, 10 Nov 2021 21:12:17 -0600
> > Tim Peters  wrote:
> > > [Bob Fang ]
> > > > This is a modest proposal to consider having sorted containers
> > > > (http://www.grantjenks.com/docs/sortedcontainers/) in standard
> library.
> > >
> > > +1 from me, but if and only if Grant Jenks (its author) wants that too.
> > >
> > > It's first-rate code in all respects, including that it's a fine
> > > example _of_ Python programming (it's not written in C - in Python).
> >
> > Agreed with Tim.  This is a perfect example of some basic and perennial
> > facility that would fit very well in the stdlib.
>
> I agree as well. Is anyone interested enough to ask the library author
> if he supports doing this? That seems to be the main unanswered
> question here.
>
> But if anyone wants to argue the "the stdlib should be shrinking, not
> growing" position, I suggest they do so *before* someone reaches out
> to the module author. No point in us making the suggestion and then
> being forced to withdraw it.
>
> Paul
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/5SURNB4C5FGJ6LSXUPVW2EFP22ERKSGB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/35YUCU4UZET6NDHCI3N2AY5YQSOX6DTF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Do we need to remove everything that's deprecated?

2021-11-12 Thread Christopher Barker

I sympathize with the OP, but I think never removing deprecated names is
the wrong solution.

If never removing those names is the appropriate action, then they never
should have been changed in the first place. That is, we should be ( and I
think are) very careful about gratuitously changing things.

Frankly, changing names in your code is a light lift. If you really want
your code to not be touched, keep using an older version of Python.

BTW, tools can help a lot here. I make heavy use of Pytest, and it recently
changed its default warning policy so that I get the depreciation warnings
early and often. It’s just not that hard to keep up if the code is seeing
any maintenance at all.

It’s a simple reality that code needs to be maintained.

Mass-replacing "failUnless" with
> "assertTrue" just because someone decided it's a better name

If there really was no better reason than that, the change never should
have been made. But once made, keeping multiple names around forever is not
a good option.

-CHB
--
Christopher Barker, PhD (Chris)

Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/HRD3NSIAI57CDRCP2CNXBL4722CUNGLX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Do we need to remove everything that's deprecated?

2021-11-13 Thread Christopher Barker

On Sat, Nov 13, 2021 at 12:01 AM Stephen J. Turnbull

> What I think would make a difference is a six-like tool for making
> "easy changes" like substituting aliases and maybe marking other stuff
> that requires human brains to make the right changes.


I think a “2to3” like or “futurize” like tool is a better idea, but yes.

The real challenge with the 2-3 transition was that many of us needed to
keep the same code base running on both 2 and 3. But do we need to support
running the same code on 3.5 to 3.10? I don’t think so. If you can’t
upgrade Python to a supported version, you probably shouldn’t upgrade your
code or libraries.

Which is a thought — maybe the policy should be that we remove things when
the new way is usable in all supported versions of Python. So as of today (
if I’m correct) anything needed in 3.5 can be dropped.

I'm not volunteering to do this, I don't even know that it's actually
> feasible.


It’s clearly feasible— if the transition from 2 to 3 could be done, this is
easy :-)

Not that I’m volunteering either.

But maybe the folks that find updating deprecated features onerous might
want to do it (or already have — I haven’t looked)

 Deprecated code does normally more or less work, and often
> it never gets close to dangerous behavior.  On the flip side, it often
> can cause dangerous behavior,


I’m confused — did you mean “sometimes cause dangerous behavior”? That’s
pretty rare isn’t it?

-CHB


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UPOHK3SKHZZHNBRCCHNZQ24AT5L252NW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)

On Sat, Nov 13, 2021 at 2:03 PM  wrote:

> def 𝚑𝓮𝖑𝒍𝑜():
>
> try:
>
> 𝔥e𝗅𝕝𝚘︴ = "Hello"
>
> 𝕨𝔬r𝓵ᵈ﹎ = "World"
>
> ᵖ𝖗𝐢𝘯𝓽(f"{𝗵ｅ𝓵𝔩º_}, {𝖜ₒ𝒓lⅆ︴}!")
>
> except 𝓣𝕪ᵖｅ𝖤𝗿ᵣ𝖔𝚛 as ⅇ𝗑c:
>
> 𝒑rℹₙₜ("failed: {}".𝕗𝗼ʳᵐªｔ(ᵉ𝐱𝓬))
>

Wow. Just Wow.

So why does Python apply  NFKC normalization to variable names?? I can't
for the life of me figure out why that would be helpful at all.

The string methods, sure, but names?

And, in fact, the normalization is not used for string comparisons or
hashes as far as I can tell.

In [36]: weird
Out[36]: 'ᵖ𝖗𝐢𝘯𝓽'

In [37]: normal
Out[37]: 'print'

In [38]: eval(weird + "('yup, that worked')")
yup, that worked

In [39]: weird == normal
Out[39]: False

In [40]: weird[0] in normal
Out[40]: False

This seems very odd (and dangerous) to me.

Is there a good reason? and is it too late to change it?

-CHB









>
>
> if _︴ⁿ𝓪𝑚𝕖__ == "__main__":
>
> 𝒉eℓˡ𝗈()
>
>
>
>
>
> # snippet from unittest/util.py
>
> _𝓟Ⅼ𝖠𝙲𝗘ℋ𝒪Lᴰ𝑬𝕽﹏𝕷𝔼𝗡 = 12
>
> def _𝔰ʰ𝓸ʳ𝕥𝙚𝑛(𝔰, p𝑟𝔢ﬁ𝖝𝕝𝚎𝑛, ｓᵤ𝑓𝗳𝗂𝑥𝗹ₑ𝚗):
>
> ˢ𝗸ｉ𝗽 = 𝐥ｅ𝘯(𝖘) - ｐr𝚎𝖋𝐢x𝗅ᵉ𝓷 - 𝒔𝙪ﬀｉ𝘅𝗹𝙚ₙ
>
> if sｋi𝘱 > _𝐏𝗟𝖠𝘊𝙴H𝕺Ｌ𝕯𝙀𝘙﹏L𝔈𝒩:
>
> 𝘴 = '%s[%d chars]%s' % (𝙨[:𝘱𝐫𝕖𝑓𝕚ｘℓ𝒆𝕟], ₛ𝚔𝒊p, 𝓼[𝓁𝒆𝖓(
> 𝚜) - 𝙨𝚞𝒇ﬁx𝙡ᵉ𝘯:])
>
> return ₛ
>
>
>
>
>
> You should able to paste these into your local UTF-8-aware editor or IDE
> and execute them as-is.
>
>
>
> (If this doesn’t come through, you can also see this as a GitHub gist at 
> Hello,
> World rendered in a variety of Unicode characters (github.com)
> <https://gist.github.com/ptmcg/bf35d5ada416080d481d789988b6b466>. I have
> a second gist containing the transformer, but it is still a private gist
> atm.)
>
>
>
>
>
> Some other discoveries:
>
> “·” (ASCII 183) is a valid identifier body character, making “_···” a
> valid Python identifier. This could actually be another security attack
> point, in which “s·join(‘x’)” could be easily misread as “s.join(‘x’)”, but
> would actually be a call to potentially malicious method “s·join”.
>
> “_” seems to be a special case for normalization. Only the ASCII “_”
> character is valid as a leading identifier character; the Unicode
> characters that normalize to “_” (any of the characters in “︳︴﹍﹎﹏＿”) can
> only be used as identifier body characters. “︳” especially could be
> misread as “|” followed by a space, when it actually normalizes to “_”.
>
>
>
>
>
> Potential beneficial uses:
>
> I am considering taking my transformer code and experimenting with an
> orthogonal approach to syntax highlighting, using Unicode groups instead of
> colors. Module names using characters from one group, builtins from
> another, program variables from another, maybe distinguish local from
> global variables. Colorizing has always been an obvious syntax highlight
> feature, but is an accessibility issue for those with difficulty
> distinguishing colors. Unlike the “ransom note” code above, code
> highlighted in this way might even be quite pleasing to the eye.
>
>
>
>
>
> -- Paul McGuire
>
>
>
>
> ___________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/GBLXJ2ZTIMLBD2MJQ4VDNUKFFTPPIIMO/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7RPAZIGJYTOR76IVTAI5NA5NA2HEHDPE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Do we need to remove everything that's deprecated?

On Sun, Nov 14, 2021 at 8:19 AM Stephen J. Turnbull <
stephenjturnb...@gmail.com> wrote:

>  > But do
>  > we need to support running the same code on 3.5 to 3.10?
>
> Need?  No.  Want to not raise a big middle finger to our users?

Note that I said 3.5, not 3.6 -- 3.5 is no longer supported. If we feel the
need to be backward compatible with unsupported versions that we can't ever
remove anything.

I wouldn't mind if the tool gently suggests, ''' hey, folks, you can't
> really support both 3.5 AND 3.10 without a lot of "if hasattr(foo,
> 'frob')", so maybe you can drop support for 3.5? ''', though.
>

now I'm confused -- if you need the hasattr() calls, then you aren't
supporting it.I guess I meant:

runinng the same code without special case code to handle the differences.
Which is why I said "like 2to3" rather than "like six". I always hated six,
even though it was a necessary evil.

 > I’m confused — did you mean “sometimes cause dangerous behavior”? That’s
>  > pretty rare isn’t it?
>
> FVO of "often" == "yeah, I've heard enough stories that I worry about
> it", I mean "often".
>

hmm -- is there any way to know which deprecations might actually be
dangerous? -- for instance, it's hard to imagine a name change alone would
be that, but I have a failure of imagination.

Eric V. Smith wrote:
> I write systems that support old versions of python. We just moved to
3.7, for example.

But do you need to support non longer supported versions of Python -- 3.7
is still supported, for just these reasons.

Can we remove stuff that's only needed by unsupported versions of Python?

So you have an application running on 3.5. You really should upgrade Python
anyway. When you do, you will need to run an "update_deprecated_stuff"
script, and test and you're good. Is that too much a burden?

Frankly, even the 2to3 transition was less painful than I thought it would
be -- I had a substantial codebase written for py2 -- we couldn't go to
three for quite some time, as we have a LOT of dependencies, and it was a
while before they were all supported. When we finally made the transition
it was less painful than I thought it would be and it would have been even
less painful if we hadn't had a both-2-and-3 stage. And we've got a bunch
of Cython code that bridges strings between Python and C++, too.

For all of the testing and signoffs we do for a single release,
> I've calculated that it costs us $5k just to release code to prod, even
> if there are no actual code changes.

But that's a fixed cost -- any maintained codebase is going to need updates
and re-releases. I don't think anyone's suggesting that you do a release
only to remove deprecations.

For the example above -- if ALL you are doing is moving from running on
Python 3.5 to running on a newer version, wouldn't that $5k cost have to be
absorbed anyway?

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VIC7RSZ3RSXXOSQFWUMMFJ5HOMMTITC7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)

On Sun, Nov 14, 2021 at 10:27 AM MRAB  wrote:

> > So why does Python apply  NFKC normalization to variable names??

> It's probably to deal with "é" vs "é", i.e. "\N{LATIN SMALL LETTER
> E}\N{COMBINING ACUTE ACCENT}" vs "\N{LATIN SMALL LETTER E WITH ACUTE}",
> which are different ways of writing the same thing.
>

sure, but this is code, written by humans (or meta-programming). Maybe I'm
showing my english bias, but would it be that limiting to have identifiers
be based on codepoints, period?

Why does someone that wants to use, .e.g. "é" in an identifier have to be
able to represent it two different ways in a code file?

But if so ...

> Unfortunately, it goes too far, because it's unlikely that we want "ᵖ"
> ("\N{MODIFIER LETTER SMALL P}') to be equivalent to "P" ("\N{LATIN
> CAPITAL LETTER P}".
>

Is it possible to only capture things like the combining characters and not
the "equivalent" ones like the above?

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QAR3TNRPNW7OXTGWKBDZHNVRKZGMCFZS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)

On Sun, Nov 14, 2021 at 4:53 PM Steven D'Aprano  wrote:

> Out of all the approximately thousand bazillion ways to write obfuscated
> Python code, which may or may not be malicious, why are Unicode
> confusables worth this level of angst and concern?
>

I for one am not full of angst nor particularly concerned. Though ti's a
fine idea to inform folks about h this issues.

I am, however, surprised and disappointed by the NKFC normalization.

For example, in writing math we often use different scripts to mean
different things (e.g. TeX's
Blackboard Bold). So if I were to use some of the Unicode Mathematical
Alphanumeric Symbols, I wouldn't want them to get normalized.

Then there's the question of when this normalization happens (and when it
doesn't). If one is doing any kind of metaprogramming, even just using
getattr() and setattr(), things could get very confusing:

In [55]: class Junk:
...: 𝗵ｅ𝓵𝔩º = "hello"
...:

In [56]: setattr(Junk, "ᵖ𝖗𝐢𝘯𝓽", "print")

In [57]: dir(Junk)
Out[57]:
 '__weakref__',

 'hello',
 'ᵖ𝖗𝐢𝘯𝓽']

In [58]: Junk.hello
Out[58]: 'hello'

In [59]: Junk.𝗵ｅ𝓵𝔩º
Out[59]: 'hello'

In [60]: Junk.print
---
AttributeErrorTraceback (most recent call last)
 in 
> 1 Junk.print

AttributeError: type object 'Junk' has no attribute 'print'

In [61]: Junk.ᵖ𝖗𝐢𝘯𝓽
---
AttributeErrorTraceback (most recent call last)
 in 
> 1 Junk.ᵖ𝖗𝐢𝘯𝓽

AttributeError: type object 'Junk' has no attribute 'print'

In [62]: getattr(Junk, "ᵖ𝖗𝐢𝘯𝓽")
Out[62]: 'print'

Would a proposal to switch the normalization to NFC only have any hope of
being accepted?

and/or adding normaliztion to setattr() and maybe other places where names
are set in code?

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z2AIS6Y6NVNF5QSD7GMTB76NSP6NAIKV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Do we need to remove everything that's deprecated?

On Sun, Nov 14, 2021 at 10:06 PM Stephen J. Turnbull <
stephenjturnb...@gmail.com> wrote:

> I'm not saying *Python* can't remove anything.  I'm saying downstream,
> *GNU Mailman* has users it *may* want to support.
>

So a project (not to pick on Mailman) may want to support its users
running old versions of the code on new versions of Python?

I've been confused about that kind of thing for years -- e.g. numpy has to
support Python versions from ten years ago so users can get the latest
numpy while running a really old version of Python? I understand that there
are sometimes IT policy issues, like "you have to use the system Python, on
an old system, but you can still install the latest Python packages' '. And
believe me, I work in an institution with many such irrational policies,
but they are,indeed, irrational, and sometimes I can use the ammunition of
saying, no, I can not run this application on that old OS. Period -- give
me an updated system to get my job done.

Unlike Petr, I'm not <0 on removals.  But they are costly, keeping up
> with deprecations is costly.

absolutely, and hopefully that cost was considered when the depreciation is
made -- we shouldn't second guess it when it's time to actually remove the
old stuff.

 In this thread, I'm most interested in exploring
> tooling to make it easier for those who express reservations about
> removals.
>

Maybe reviving the future package to cover Python 3 changes. It appears not
to have been updated for two years, which makes sense, as Py2 is no longer
supported. But maybe its infrastructure could be updated to accommodate the
newer changes. It should be an easier job than making a 2-3 compatible
codebase.

I found future really helpful in the 2-3 transition, and one nice thing
about it is that it provided both translation ala 2to3 and
compatibility ala six.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UXMR6X3ZBK3QJZXAXXQVEC52GJAYGL2Y/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

2021-11-20 Thread Christopher Barker

ocumenting dataclasses. However, PEP 567
does make a number of references to typing, and was clearly written with
the idea in mind that the annotations would, in fact, be used for type
analysis. But again, no mention of PEP 563, or what might be in the
field.type attribute -- one might wonder why it was there at all if it was
not intended to be used.

We need help in defining those requirements clearly, in uncovering the use
> cases we’re not aware of, and most importantly, being an interface to
> typing enthusiasts


Again -- is it only "typing enthusiasts" that you want to engage? Or "users
of annotations"? -- maybe it is, but it would be nice if that was a clear
statement from the SC.

- CHB


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MAPFC2O5YRCH6LFGBIX4YEFG4G5X7VSH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

2021-11-25 Thread Christopher Barker

ah, probably doable one way or another.


>   > But I suspect I'm not alone in not really noticing that statement,
>
>   I am
> surprised that a lot of folks haven't noticed that this comes up
> occasionally and the answer every time is "we're not going to go out
> of our way to break other use cases, but typing *is* the primary use
> case and will take precedence if the question comes up".
>

I think the "breaking other use cases" has been subtle to those not paying
attention to typing.


>  > But the fact is that I, among others, have been a bit uncomfortable
>  > about the focus on typing in Python for years.
>
> You mean y'all are uncomfortable with the popularity of typing.  You
> wouldn't care if it wasn't used outside of hugely complex proprietary
> codebases you'll never see.
>

well, yes. The issue is that, intended or not, typing is making it's way
into Python culture. As an instructor of beginning python users, I am
unsure at this point when to introduce type annotations.

What is their role? Up to today, I have treated them as an advanced
feature, useful for "complex codebases". But there are any number of
examples springing up on the internet, to the point where many students now
think they are "best practice", if not actually required.

But that's getting pretty OT here.

>  Anything that helps

> *other people* to write better code, I can get behind.  My code, I'm
> going to write as incompetently as ever.[1] ;-)
>

me too -- actually, not quite. I write a lot of "scripts" that I will
continue to keep simple, and I also develop complex systems -- and those
require more care.


>   > But when issues are raised, we have been repeatedly told that
>  > typing is, and always will remain, optional.
>
> This has always been in the context of *your* source code.


Sure -- my point with that is that if I don't want to use typing in my
code, then I didn't have to pay attention to the development of the typing
systems.

But my point here is that what changed with PEP  563 is that while typing
is still optional, this is the first time that the language itself may be
changed to accommodate typing -- so it can no longer be completely ignored.

But dunders are the property of the language (or sometimes the
> implementation), and they always have been.  If you use them in a way
> that's not documented to work, you're at risk even if it happens to
> work now.  PEP 3107 didn't document that anything would work. ;-)
>

no, but it did document that annotations were "arbitrary Python
expressions" -- that is changed in PEP 563 -- intentionally, but it is
changing something about the language that was documented.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ML3OSGC2DPLLAEDOXGT2XK2S3UJ5AY23/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

2021-11-26 Thread Christopher Barker

> There may > be scoping issues to be sorted out,

> Yes, the scoping issues are the main problem.


> > but I don't think they are
> > insurmountable.

 Probably not — and LLukas Langa has some good ideas that the SC is
considering.

(Sorry I can’t get your name right in a phone)

Maybe inspect.get_annotations() does, or will be able to, solve many of
these issues.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z74WROINRFUWXNLDSC4WYBSRCHHDNJSC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

2021-11-26 Thread Christopher Barker

On Fri, Nov 26, 2021 at 1:47 PM Guido van Rossum  wrote
>  It's easy enough to do something at runtime with `def f(a: list[int]) ->
int`. It's not so simple to handle `def f(a: Sequence[T]) -> T` or `def
f(cb: (T) -> tuple[str, T], Sequence[T]) -> Mapping[str, T]`. Presumably
frameworks like pydantic just don't support such things and tell the user
not to do that.

I don’t know about Pydantic, but that’s exactly my use case: the type needs
to be an actual intstantiatable type.

So list and tuple is fine, and Sequence[t]  won’t work.  But it’s fine that
this use case is restricted.

In fact, other than the basic core types, you need to use specialized types
with this system anyway.

-CHB
-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CFRHK622JYFQ3MD6UBTAUS3UDY2UQ52F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

2021-11-27 Thread Christopher Barker

On Fri, Nov 26, 2021 at 5:47 PM Jim J. Jewett  wrote:

> Steven D'Aprano wrote:
> > Maybe PEP 563 could include a decorator in the typing module to
> > destringify all the annotations in a class or function?
>
> If it were in an annotations module, that would probably be sufficient.
>
> If it is in typing, then it is a very heavyweight dependency --


As of Py 3.10 there is:

inspect.get_annotations

one could write a decorator around that, but I think the real question is
when do you want the annotations to be "finalized"?

-CHB



heavy enough that even the people actually using that module for
> development (and not for production runs) are worried about the costs.  If
> the costs of the typing module are that high, it is not acceptable to
> impose them on people not otherwise using the module.
>
> -jJ
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZQHP24T2PKRTDBGZ36KLBHLLOKITP5ON/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OID2CGJ2L5IA7WGDRPO45BAPECFGWO7B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Optimizing literal comparisons and contains

2021-11-28 Thread Christopher Barker

I will frequently do simple computation with literals to make my code more
clear:

t = 2 * 3600  # 2 hours in seconds

But I see no need to optimize this kind of thing -- it would never be in a
tight loop.

-CHB


On Sun, Nov 28, 2021 at 3:06 PM Rob Cliffe via Python-Dev <
python-dev@python.org> wrote:

> I am slightly surprised that it seems to be *easier* to fold selected
> constant expressions than to have more generic code to fold them all.
> Or at least, all those that don't contain containers, such as
>  1 in [0,1,2]
> Rob Cliffe
>
> On 28/11/2021 21:10, Eric V. Smith wrote:
> >> On Nov 28, 2021, at 3:03 PM, Serhiy Storchaka 
> wrote:
> >>
> >> 28.11.21 17:13, Skip Montanaro пише:
> >>>> That is not entirely true:
> https://github.com/python/cpython/pull/29639#issuecomment-974146979
> >>> The only places I've seen "if 0:" or "if False:" in live code was for
> >>> debugging. Optimizing that hardly seems necessary. In any case, the
> >>> original comment was about comparisons of two constants. I suppose
> >>> sweeping up all of that into a constant expression folding/elimination
> >>> step performed on the AST and/or during peephole optimization would
> >>> cover both cases.
> >> "if 0:" and "if False:" is already optimized by the compiler. The OP
> >> proposes to optimize "2 < 1", and I cannot imagine any use case for
> this.
> > I agree. I suggest we don’t add this optimization.
> >
> > Eric
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/JP6FF2RHDQSIOS6ZAI45S7X6UXWGPBKR/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/LD3XHINO7VPDE5EMNFD5ON4FQIJLY4DI/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NJWCKNV4ULEBHSLR44G3HNVROE4IAM47/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Preventing Unicode-related gotchas (Was: pre-PEP: Unicode Security Considerations for Python)

2021-11-29 Thread Christopher Barker

On Mon, Nov 29, 2021 at 1:21 AM Steve Holden  wrote:

> It's interesting that the egalitarian wish to allow use of native
> "alphabetics" has turned out to be such a viper's nest.
>

Indeed.

However, is there no way to restrict identifiers at least to the alphabets
of natural languages? Maybe it wouldn’t help much, but does anyone need to
use letter-like symbols designed for math expressions? I would say maybe,
but certainly not have them auto-converted to the “normal” letter?

For that matter, why have any auto-conversion all?

The answer may be that it’s too late to change now, but I don’t think I’ve
seen a compelling (or any?) use case for that conversion.

-CHB

Particular thanks to Stephen J. Turnbull for his thoughtful and
> well-informed contribution above.
>
> Kind regards,
> Steve
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/FNSI6EXCWMMCXEJNYWVVR5LMFOM6M5ZB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/H4QOXCLPBMGNGQHNFFQDYJQANUH6323C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

2021-11-29 Thread Christopher Barker

I don't think the issue of how typing is making its way intot he community
is particular relevant to this thread, but I was directly asked a question,
so I'll answer it:

I don't know exactly where the impression is coming from that typing is a
best practice, but it's certainly creeping into example code, etc. So folks
will see examples of code that have nothing to do with typing, and probably
have no need for it, and there will be type annotations. Many (most?) folks
learn from examples more than targeted instruction, so as those examples
grow, so will the impression that it's an inherent part of Python.

David Mertz' experience is similar to mine as an instructor.

Another anecdotal example: I've been teaching an intro to Python class for
years, based on the same material, that we try to keep up to date. But
typing is not yet included. However, about a year ago, one of the other
instructors (that had not previously developed materials for the class)
added a new example / exercise to the "intro to classes" lesson. And that
example had type annotations. I ended up removing them, as I thought it
could be pretty confusing to students to see them for the first time with
no explanation (would they think it was something specific to classes?). I
wish I'd discussed it with the other instructor, I don't know why that was
done -- was it standard practice in their workplace? Seemed like a best
practice? I have no idea.

Another observation is that more and more library authors are being asked
to add type annotations -- maybe that IS a best practice for widely used
libraries, but that's one more piece of data.

Also: I know this isn't intentional, but Guido has an enormous influence on
the Python community -- so I suspect that the fact that he's taken an
active role in developing static typing has influenced how it's being
perceived.

I do think this is a topic for the community to grapple with, but not
really a Python-dev responsibility.

-CHB

--
Christopher Barker, PhD (Chris)

Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/PTOBKJ6ILORMRORXLTL2WXADIOMO3AXX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

2021-11-30 Thread Christopher Barker

This is great Patrick, thanks.

My use case is similar to Patrick's, except it builds on dataclasses
directly:

It's part of a larger system, but I've just pulled it out into its own
poorly documented and poorly tested package:

https://github.com/PythonCHB/flexi

(I think it's generally useful, so may one day make a proper package out of
it)

Anyway, the principle is this:

The goal is to have a flexible data model that enforces the structure, but
not necessarily the validity of values, or even types. This is useful
because we need to store / edit / import complex hierarchy of data, and we
very much don't want to have all of it to have to be valid in order to even
store it. An example is a dataset we recently imported (from
spreadsheets--arrgghh), where in a thousand records, they had put "too
viscous to measure" instead of a value for interfacial tension. That, of
course, would be helpful to a person reading the spreadsheet, but would
have been a pain if we had enforced that that field had to be a float > 0.
Instead, it imported fine, and then our validation code flagged the issue
to be fixed later. And all the rest of the data could be used in the
meantime.

Anyway, that was a digression (but explains why we don't use Pydantic).
Functionally, we need the "nodes" in the data model to be able to convert
themselves to/from JSON compatible Python, and validate themselves, etc.
This is done by using the actual type object, as stored in
dataclasses.Field.type

This has worked nicely for us, and dataclasses saved us a lot of
boilerplate code.

if we do

from future import Annotations

then the system breaks, as we have a string instead of the actual type that
is needed.

Solutions:

I *think* PEP 649 would work fine for us, as the annotation would get
evaluated when it was added to the field object.

But inspect.get_annotations isn't really helpful for this use, as the
Field.type attributes aren't annotations anymore. So we would have to
eval() the strings ourselves, and I think it would be unclear what
namespaces to use for that.

If dataclasses were to use

inspect.get_annotations(cls, eval_str=rue)

in order to extract the types to put into the Field objects, then I think
all would be good for us. And as I understand it, dataclasses itself
doesn't do anything with that attribute anyway.

If that's not decided to be a good thing for dataclasses, then I think we'd
have to re-implement a lot ourselves, though it could probably be hacked
in - I haven't tried that yet.

BTW, I don't think we've heard from the attrs (and especially cattrs) folks
in this thread. A search of the repo issues indicates that there has been
some discussion of PEP 563's impact, but it's not totally clear to me if
it's been resolved.

-CHB

--
Christopher Barker, PhD (Chris)

Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/ZYWLGMP5UO7NTABVJJOHN7ZSS4KUM6UG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

2021-11-30 Thread Christopher Barker

Another concern I have is the over specification of types.

I have seen many examples of, e.g.

func(x: int, y: float, stuff: List(int]):

but very few of:

func(x: SupportsInt, y: SupportsFloat, stuff: Sequence[SupportsInt]):

(or even Iterable[int])

Is that even the right thing to do to get generic number types? How do I
specify a type that could be any number type (float, int, Fraction,
Decimal, numpy.float32) ... I just spent a few minutes perusing the typing
module and MyPy docs, and didn't find a discussion of that.

But I'd really love to see the community as a whole start with more
generic types in examples, and be clear that the concrete types should be
used only when necessary -- even the typing module docs use a lot of
concrete types in the examples.

I think this is an issue for two reasons:

1) Folks learn from examples more than instruction -- people are very
likely to follow the first example they see that seems to do the job.

2) as mentioned in this thread, library authors are being asked to type
hint their libraries. I think it will be all too common for that type
hinting to be more specific than required. For the most part, that won't
cause any real issues right away, but as soon as someone uses that lib in a
large project that enforces static type checking, it's could get ugly.

-CHB






On Tue, Nov 30, 2021 at 1:37 PM Paul Moore  wrote:

> On Tue, 30 Nov 2021 at 19:07, Brett Cannon  wrote:
> >
> > On Tue, Nov 30, 2021 at 9:09 AM Steven D'Aprano 
> wrote:
> >>
> >> On Tue, Nov 30, 2021 at 02:30:18PM +, Paul Moore wrote:
> >>
> >> > And to be clear, it's often very non-obvious how to annotate something
> >> > - in https://github.com/pfmoore/editables I basically gave up because
> >> > I couldn't work out how to write a maintainable annotation for an
> >> > argument that is "a Path, or something that can be passed to the Path
> >> > constructor to create a Path" (it's essentially impossible without
> >> > copy/pasting the argument annotation for the Path constructor).
> >
> > You're after
> https://docs.python.org/3/library/os.html?highlight=pathlike#os.PathLike:
> `str | PathLike[str]` (if you're only accepting string paths).
>
> Well, it's not really. What I'm after, as I stated, is "anything that
> can be passed to the Path constructor". Yes, str | PathLike[str] is
> probably close enough (although why is it OK for me to prohibit bytes
> paths?) but that's what I mean about copying the Path constructor's
> annotations. If Path changes, I have to change my code.
>
> This is a very common idiom:
>
> def f(p: ???):
> p = Path(p)
> ...
>
> Why isn't it correspondingly straightforward to annotate?
>
> If PathLike[str] included str, then it would be a lot easier. It's not
> at all obvious to me why it doesn't (well, that's not entirely true -
> it's because PathLike is an ABC, not a protocol, and it's not intended
> to define "the type of objects that the Path constructor takes"). It
> would still not be documented anywhere, though.
>
> >> I thought that type inference was supposed to solve that sort of
> >> problem? If the typechecker can see that an argument is passed to the
> >> Path constructor, it should be able to infer that it must be the same
> >> types as accepted by Path.
> >
> > I would change that "should" to "may". Python's dynamism makes
> inferencing really hard.
>
> That's fair. That's why I think it should be straightforward for the
> user to explicitly say "this argument should accept the same types as
> pathlib.Path does". If inference can't do it automatically, and the
> user can't (easily) let the checker know that it's OK to do it, then
> we're left with no easy way to express a very common pattern.
>
> >> Aside: I'm a little disappointed in the way the typing ecosystem has
> >> developed. What I understood was that we'd get type inference like ML or
> >> Haskell use, so we wouldn't need to annotate *everything*, only the bits
> >> needed to resolve ambiguity. But what we seem to have got is typing like
> >> C, Pascal and Java, except gradual. Am I being unreasonable to be
> >> disappointed? I'm not a heavy mypy user, I just dabble with it
> >> occasionally, so maybe I've missed something.
> >
> >
> > It really depends on the code base. Type checkers can make guesses based
> on the code they have available to them, but that only works if the usage
> is really clear and the dynamic nature of the code doesn't make things
> murky. For instance, look at open() and how whether you opened a file with
> `b` or not influences whether the object's methods return strings or bytes.
> What would you expect to be inferred in that case if you didn't annotate
> open() with overrides to specify how its arguments influence the returned
> object?
>
> Personally, I'd be quite happy leaving open() as duck typed. I see
> this as what Steven was getting at - the "typing ecosystem" has moved
> into a situation where it's acknowledged that some typing problems are
> really hard,

[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

2021-11-30 Thread Christopher Barker

On Tue, Nov 30, 2021 at 3:31 PM Guido van Rossum  wrote:

>
>> There is some discussion of the numeric tower in PEP 484 but the PEP says
> you should just use 'int', 'float' and be happy.
>

Thanks -- I didn't think to look there. And this: "when an argument is
annotated as having type float, an argument of type int is acceptable"
makes it pretty workable.

I do wonder what happens  with oddball numbers, like, say numpy.float32.
But maybe that's Ok, as for though most part one will be dealing with
arrays of them -- the scalars are fairly rare in real code.

however, IIRC, __index__ was introduced at least partially so that numpy
integer types could be used as indexes. Which would mean SupportsIndex for
a type that's going to be used as an index, yes?

> solutions is just terrible (too much typing for too little benefit) and
> honestly in many cases the runtime is not very good at these either (e.g.
> Decimal refuses to play). In practice it is just fiction that you can write
> your own numeric type.
>

Yes, Decimal and Fraction are pretty special purpose, really.

> Regarding Iterable[int] vs. Sequence[int], there are subtle semantic
> differences so it depends, but these have much better support and *are*
> encouraged. (What helps is that List is invariant but Sequence is
> covariant, and people often want covariance, so there is a real incentive
> to use Sequence, Iterable etc.)
>
>
>> But I'd really love to see the community as a whole start with more
>> generic types in examples, and be clear that the concrete types should be
>> used only when necessary -- even the typing module docs use a lot of
>> concrete types in the examples.
>>
>
> Maybe you are speaking with a lot of enthusiasm but not a lot of
> experience?
>

well, yes, for sure.

> Generics aren't always better, and in many cases they just aren't worth
> the effort to get them right
>

I suspect that depends a bit on the use case. For something internal to one
system, probably not worth the effort at all. But for a general purpose
library, I'm (perhaps needlessly) concerned about types getting locked down
more than they need to be, just because its easier, and "it works for me"

> (just like not every bit of code you write needs to be published on PyPI).
>

Someone should tell that to all the folks publishing 0.0.0.1 version of
packages :-)

> I also have a feeling that most frameworks that use *runtime*
> introspection of types strongly prefer concrete types, i.e. list[int]
> rather than Sequence[T].
>

This is getting into what it means to use Types at runtime -- if it's
runtime type checking, then more abstract types might be fine. But if it's
using the type itself (like my use case), then absolutely -- my system will
only work with real concrete type objects. Though that may not play that
well with type checkers :-(

> It would be nice if someone did some work and collected a list of
> tutorials about type annotations that exist (especially the ones that are
> discoverable with a simple Bing query) and ranked them by quality.
>

yes, it would.

> We should definitely push back on zealous new converts to typing who
> insist that everything should be annotated.
>

well, we got folks wanting to change PEP 8 becuase they don't want their
linter to complain -- so it will be a battle.

> (Have you run into VS Code yet? It gets tremendous value from typing
> stubs, in the form of improved auto-complete and hover-doc functionality.)
>

Some of the folks on my team use it -- I've been meaning to check it out --
one more reason now.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PTZBYWYU66JVOWVHKISKDBMQZDLWEK43/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

2021-12-01 Thread Christopher Barker

I know this isn't really the place for this conversation, but:


> which is what `os.PathLike` represents, hence why `str` isn't covered by
> it);
>

wait, what? It seems so clear to me that "PathLike" (as a type specifier)
would mean: anything that can be passed into os.fspath to give me a path.
(or, of course to the stdlib functions that take paths)

Isn't the entire purpose of os.fspath that you can write code like:

def fun(some_kind_of_path):
   some_kind_of_path = os.fspath(some_kind_of_path)

   (or just pass it to a function you takes PathLIke)

and go on your merry way -- e.g. duck typing, baby!

Is there really no way to annotate that simply now?

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AYFULIM3SJ7DZK7SHGK436KGHX2M3WP5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

>for library authors.

> Providing high quality stubs and the best user experience is not easy.
> But I believe that referring people to typeshed can help.


This is actually very helpful. It provides an answer for open source
projects for which do users want typing.

One can say to the users that want type stubs for the library that they are
encouraged to contribute those stubs to type shed themselves and once they
have been tested and vetted they can be included in the project.

it’s not unlike any other feature request. The developer of an open source
project is under no obligation to provide any feature asked for, but a
well-managed project will encourage useful contributions from users.

-CHB
-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ASO2PUTECTOB25HC7DURFOGOHOUQOCD5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

On Thu, Dec 2, 2021 at 9:48 AM Eric Fahlgren  wrote:

> wait, what? It seems so clear to me that "PathLike" (as a type specifier)
> would mean: anything that can be passed into os.fspath to give me a path.
> (or, of course to the stdlib functions that take paths)
>>
>>
>> Isn't the entire purpose of os.fspath that you can write code like:
>>
>> def fun(some_kind_of_path):
>>some_kind_of_path = os.fspath(some_kind_of_path)
>>
>>(or just pass it to a function you takes PathLIke)
>>
>> and go on your merry way -- e.g. duck typing, baby!
>>
>> Is there really no way to annotate that simply now?
>>
>
> Assuming you want the return value of 'fun' to be covariant with the path
> input, I believe you would say this:
>

I just read the wikipedia page on  "Covariant return type", and I either
misunderstood what it means, or it's not relevant to my intended example.

> def fun(some_kind_of_path: str) -> str: ...
> def fun(some_kind_of_path: bytes) -> bytes: ...
> def fun(some_kind_of_path: os.PathLike[AnyStr]) -> AnyStr:
> some_kind_of_path = os.fspath(some_kind_of_path)
> # transform it
> return some_kind_of_path
>

Ahh I see, I wasn't intending, in my example, to return the path -- my
intent was a function that would need to use the path, to, e.g. open a file.

This Strikes me as a very common use case -- you want your users to be able
to pass multiple different types in, as long as they can be used as a path.
I have a LOT of code that does this -- for years it accepted only string
paths, and now I'm updating it to take PathLike as well.

My usual code is one of:

def fun_that_opens_a_file(the_path):
# if I'm working with an "old" library, or one that, e.g. wraps a C lib:
the_path = os.fspath(the_path)
# if I'm working with things I know will take a Path object:
the_path = Path(the_path)

>
> I would love to be shown how to do this with just a one-line declaration
> of 'fun', but I've given up trying to figure it out.
>

Darn. could you at least use a Union type:

def fun_that_opens_a_file(the_path: Union[str, Path]):
...

which isn't too bad. Personally, I think accepting bytes is the way of
madness, but you could add that too.

On Thu, Dec 2, 2021 at 12:32 PM Brett Cannon  wrote:

>
> wait, what? It seems so clear to me that "PathLike" (as a type specifier)
> would mean: anything that can be passed into os.fspath to give me a path.
> (or, of course to the stdlib functions that take paths)
>
> That is not what the docs say:
> https://docs.python.org/3/library/os.html#os.PathLike. And as the creator
> of that ABC it's very much on purpose (see
> https://www.python.org/dev/peps/pep-0519/ for details).
>

Sorry, I wasn't clear. By "PathLike" (as a type specifier)" I meant using
it as, well, a type specifier, not as an ABC -- often we can use an ABC as
a type specifier, but not always. But you've prompted me to go re-read the
PEP and I see this:

"""
Provide specific type hinting support

There was some consideration to providing a generic typing.PathLike class
which would allow for e.g. typing.PathLike[str] to specify a type hint for
a path object which returned a string representation. While potentially
beneficial, the usefulness was deemed too small to bother adding the type
hint class.

This also removed any desire to have a class in the typing module which
represented the union of all acceptable path-representing types as that can
be represented with typing.Union[str, bytes, os.PathLike] easily enough and
the hope is users will slowly gravitate to path objects only.
"""

I didn't pay any attention to that at the time, as I wasn't paying
attention to typing. But personally, I think I do have that desire :-)

But I can see that "the hope is users will slowly gravitate to path objects
only".

would lead to:

>  in my code I just take pathlib.Path or pathlib.PurePath and I'm done as
I don't want to be dealing with encoded file paths and instead with objects
that represent file paths.

For my part, I'm not interested  in encoded paths (e.g. bytes) -- that is
absolutely not the right boundary to deal with file system encoding.
(that's mentioned in the PEP) But I also don't want my users (or me, with
old code, scripts, etc) to have to suddenly go in and wrap a call to Path()
everywhere in order to continue to use my library. String paths are
ubiquitous, and I think here to stay.

I really like PEP 519 -- I think it not only provides a transition, but
also a future in which Path objects and string paths can continue to play
well together.

While Union[PathLike, str] is a pretty light lift, th

[Python-Dev] Re: The current state of typing PEPs

resent -- damn iPhone!

On Thu, Dec 2, 2021 at 3:20 PM Christopher Barker 
wrote:

> >for library authors.
> >
> > Providing high quality stubs and the best user experience is not easy.
> > But I believe that referring people to typeshed can help.
>
>
> This is actually very helpful. It provides an answer for open source
> projects for which do users want typing.
>
> One can say to the users that want type stubs for the library that they
> are encouraged to contribute those stubs to type shed themselves and once
> they have been tested and vetted they can be included in the project.
>
> it’s not unlike any other feature request. The developer of an open source
> project is under no obligation to provide any feature asked for, but a
> well-managed project will encourage useful contributions from users.
>
> -CHB



> --
> Christopher Barker, PhD (Chris)
>
> Python Language Consulting
>   - Teaching
>   - Scientific Software Development
>   - Desktop GUI and Web Development
>   - wxPython, numpy, scipy, Cython
>


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/B2LV7W3XU2VW4YYHP3K34VTJ4N3WZAHA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

On Thu, Dec 2, 2021 at 3:35 PM Rob Cliffe via Python-Dev <
python-dev@python.org> wrote:

> I assume you accidentally pressed Send prematurely.
>

Actually, it was my phone making the text white -- what the heck?

why is it so hard to send plain text email?

resent now.

-CHB



> Still, maybe you have inadvertently listed everything that is agreed about
> typing PEPs. 😂
> Rob Cliffe
>
> On 02/12/2021 23:20, Christopher Barker wrote:
>
>
> >for library authors.
>
>> Providing high quality stubs and the best user experience is not easy.
>> But I believe that referring people to typeshed can help.
>
>
> This is actually very helpful. It provides an answer for open source
> projects for which do users want typing.
>
> One can say to the users that want type stubs for the library that they
> are encouraged to contribute those stubs to type shed themselves and once
> they have been tested and vetted they can be included in the project.
>
> it’s not unlike any other feature request. The developer of an open source
> project is under no obligation to provide any feature asked for, but a
> well-managed project will encourage useful contributions from users.
>
> -CHB
> --
> Christopher Barker, PhD (Chris)
>
> Python Language Consulting
>   - Teaching
>   - Scientific Software Development
>   - Desktop GUI and Web Development
>   - wxPython, numpy, scipy, Cython
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to 
> python-dev-leave@python.orghttps://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/ASO2PUTECTOB25HC7DURFOGOHOUQOCD5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/7AF22G3S23S67FAHH7BI4XUNHWOU4YXT/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J3MG3ULSGWM7JBJ6HADRMNJOD2MV5W7S/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

On Thu, Dec 2, 2021 at 6:57 PM Stephan Richter 
wrote:

> So maybe this is my time to chime in. I have used annotations for runtime
> behavior. My primary use case is an injection library that I wrote.


Does it work with __future__ annotations?

-CHB





> It allows
> something along the lines of:
>
> class IMyService(IService):
> pass
>
> @inject
> def call_something(arg1: str, arg2: int, svc: IMyService = None):
> assert svc is not None
> # do stuff...
>
> At runtime the `inject()` decorator will scan the function signature and
> insert an instance of `IMyService` based on a registry lookup. (We use
> zope.component utilities, but that's an implementation detail.) When
> testing,
> one can simply pass a mock version of the service.
>
> Using the mypy-zope plugin, the above also passes all mypy type checking.
>
> It is a simple pattern in terms of annotation but a very effective pattern
> that mimics other injection systems in Java and TypeScript (Angular). We
> have
> used it for a few years now and like it a lot.
>
> Regards,
> Stephan
>
> On Monday, November 29, 2021 6:00:04 PM EST Barry Warsaw wrote:
> > On Nov 26, 2021, at 01:13, Paul Moore  wrote:
> > > I'd therefore interpret Barry's plea as being for *anyone* with a use
> > > for annotations to provide their feedback (at least, anyone who
> > > accepts that annotations are types), with particular emphasis on
> > > people who want to use the types declared in annotations to affect
> > > runtime behaviour, as that's the most under-represented group at the
> > > moment (and it's not clear whether it's under-represented because
> > > there aren't many such uses, or because the users aren't being heard
> > > from).
> >
> > Spot on.
> >
> > -Barry
>
>
> --
> Stephan Richter
> Entrepreneur & Geek
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/XRJJ4ZMOQJXOTRFT3FQ25QXHXOKBVXKP/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VF274OZPBKJ4OGF2ZWGLIES2GQF35VEZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should dataclasses addset (and possibly get ) descriptors ?

2021-12-13 Thread Christopher Barker

I'm also confused about what you are trying to achieve here. Note that:

assert Entries.ENTRY1.value.a == 1

should work.

But my main thought is that dataclasses are really just a way to
auto-generate classes without having to write a bunch of boilerplate. So
you example above is equivalent to:

class Foo:
def __init__(self, a=0):
self.a = a

Which behaves the same way when used the same way with Enum.

So why should dataclasses be special in this particular way?

-CHB


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J7EEZUO7MI7RZT347C6VETUPCGBZIUYJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: my plans for subinterpreters (and a per-interpreter GIL)

2021-12-15 Thread Christopher Barker

On Wed, Dec 15, 2021 at 3:00 PM Guido van Rossum  wrote:


> who cares if the refcount for None is 5000 or 1610612736? As long as the
> refcount of *mortal* objects is the same as it was before, this shouldn't
> be a problem.
>

indeed:

$ python -c "import sys; print(sys.getrefcount(None))"
4110

and a newly started iPython session:

In [2]: sys.getrefcount(None)
Out[2]: 28491

It does seem a bit silly to actually be tracking that refcount :-)

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/36N4I4CJ53OR3CLDJSJUIXEAS3NIFURP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python 3.10 vs 3.8 performance degradation

On Sun, Dec 19, 2021 at 1:46 PM MRAB  wrote:

> On 2021-12-19 20:06, Tigran Aivazian wrote:
> > So far I have narrowed it down to a block of code in solve.py doing a
> lot of multi-threaded FFT (i.e. with fft(..., threads=6) of pyFFTW), as
> well as numpy exp() and other functions and pure Python heavy list
> manipulation (yes, lists, not numpy arrays).

> The Python interpreter does have the GIL (Global Interpreter Lock). It
> can't execute Python bytecodes in parallel, but timeshares between the
> threads.
>

Sure. But what the OP seems to have discovered is that there is some
difference in behavior between 3.8 and 3.10 -- and AFIAK, there are not
intended major changes in the GIL between those two releases.

I *think* that all of the issues have involved numpy (pyFFTW depends on
numpy as well), and certainly matplotlib does) -- but I think the OP has
made sure that the numpy (and other libs) versions are all the same. There
still remains to confirm that numpy (and other libs) are built exactly the
same way in the py3.8 and 3.10 versions -- this can be a very complicated
stack!

But it seems either cPython itself, or numpy (or Cyhton?) is doing
something different. Still to be discovered what that is.

Note the OP: make sure that it's not as simple as a change to the default
for the threads parameter.

Note2: even if this is a regression cPython itself, I suspect the numpy
list may be a better wey to get it figured out.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DEBHSISDF45B3XTOYNM34WUB7GOPJRNV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Syntax PEP

> On 12/18/2021 3:13 PM, Batuhan Taskaya wrote:
>
> > tl;dr: I find it very troubling that we are going on a path where need
> > to increase the language complexity (syntax) only in the cause
> > 'easier' typing.


Which brings up the question is whether it's worth adding  syntax for
typing, but only in the context of typing. As of right now,
typing.get_type_hints() will evaluate a string annotation, e.g.

In [62]: def f(x:"int"):
...: pass
...:

In [63]: typing.get_type_hints(f)
Out[63]: {'x': int}

so get_type_hints could extend its acceptable syntax with this new use of
-> -- and it could get used by wrapping it in quotes. And depending on
how PEP 563 gets resolved, the quotes may not be necessary in the future.

And this could open up some other nifty things, like extending what's
allowable inside [] -- there was a discussion a while back on python-ideas
about extending the __getitem__ protocol, partly motivated by type hints.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JWPMHLSPFTYAYSMWCX7LOXWBEU4FA6DC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Syntax PEP

A question that came up for me is:

How common is it to need to use Callable for type hints? particularly
complex versions, specifying what parameters the Callable takes? A more
compact and easier to read syntax is nice, but not very important if it
isn't used much.

My first thought on this was that I can't remember a single time that I
wrote production code that took a Callable as a function parameter -- or
returned one -- OK maybe a few times, but it's certainly rare in my
production code.

So I looked in the PEP to see if that issue was addressed, and indeed it is:

"The Callable type is widely used. For example, as of October 2021 it was
the fifth most common complex type in typeshed,"

That did surprise me, but on thinking about it, maybe not so much. It
strikes me that Callable is most likely to be used in fairly low level,
general purpose functions, like map(), sort(), various functions in
itertools, etc. Just the sort of functions that are common in the standard
library, but may not so much in production code.

I have no idea how to evaluate how common it is in production code -- maybe
type hinting is common enough now that PyPi could be searched -- but even
PyPi is a biased sample, as it is full of, by definition, libraries for
others' use -- i.e. general purpose tools (less general that the stad lib,
but still not specialty production code, which I suspect is the majority of
Python code out there).

Perhaps some folks that have been type=hinting their production code bases
could provide anecdotal evidence.

Anyway, if my hypothesis is correct, then it's not so bad that not-so-nice
syntax is required to type hint general purpose utilities.

-CHB

--
Christopher Barker, PhD (Chris)

Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/ROPQTGYCTCL6N37MJHFFFWSPPGGS2662/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Syntax PEP