Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

[Gregory P. Smith]

+1 on getting this in for 3.5.

If the only objection people are having is the stupid paint color of the
name I don't care what it's called!  scandir matches the libc API of the
same name.  iterdir also makes sense to anyone reading it.  Whoever checks
this in can pick one and be done with it.  We have other Python APIs with
iter in the name and tend not to be trying to mirror C so much these days
so the iterdir folks do have a valid point.

I'm not a huge fan of the DirEntry object and the method calls on it
instead of simply yielding tuples of (filename,
partially_filled_in_stat_result) but I don't *really* care which is used as
they both work fine and it is trivial to wrap with another generator
expression to turn it into exactly what you want anyways.

Python not having the ability to operate on large directories means Python
simply cannot be used for common system maintenance tasks.  Python being
slow to walk a file system due to unnecessary stat calls (often each an
entire io op. requiring a disk seek!) due to the existing information that
it throws away not being used via listdir is similarly a problem. This
addresses both.

IMNSHO, it is a single function, it belongs in the os module right next to
listdir.

-gps



On Thu, Jun 26, 2014 at 6:37 PM, Ben Hoyt  wrote:

> I don't mind iterdir() and would take it :-), but I'll just say why I
> chose the name scandir() -- though it wasn't my suggestion originally:
>
> iterdir() sounds like just an iterator version of listdir(), kinda
> like keys() and iterkeys() in Python 2. Whereas in actual fact the
> return values are quite different (DirEntry objects vs strings), and
> so the name change reflects that difference a little.
>
> I'm also -1 on windows_wildcard. I think it's asking for trouble, and
> wouldn't gain much on Windows in most cases anyway.
>
> -Ben
>
> On Thu, Jun 26, 2014 at 7:43 PM, Ethan Furman  wrote:
> > On 06/26/2014 04:36 PM, Tim Delaney wrote:
> >>
> >> On 27 June 2014 09:28, MRAB wrote:
> >>>
> >>>
> >>> Personally, I'd prefer the name 'iterdir' because it emphasises that
> >>> it's an iterator.
> >>
> >>
> >> Exactly what I was going to post (with the added note that thee's an
> >> obvious symmetry with listdir).
> >>
> >> +1 for iterdir rather than scandir
> >>
> >> Other than that:
> >>
> >> +1 for adding [it] to the stdlib
> >
> >
> > +1 for all of above
> >
> > --
> > ~Ethan~
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

[Gregory P. Smith]

On Fri, Jun 27, 2014 at 2:58 PM, Nick Coghlan  wrote:
>
>  * -1 on including Windows specific globbing support in the API
> * -0 on including cross platform globbing support in the initial iteration
> of the API (that could be done later as a separate RFE instead)
>
Agreed.  Globbing or filtering support should not hold this up.  If that
part isn't settled, just don't include it and work out what it should be as
a future enhancement.

> * +1 on a new section in the PEP covering rejected design options (calling
> it iterdir, returning a 2-tuple instead of a dedicated DirEntry type)
>
+1.  IMNSHO, one of the most important part of PEPs: capturing the entire
decision process to document the "why nots".

> * regarding "why not a 2-tuple", we know from experience that operating
> systems evolve and we end up wanting to add additional info to this kind of
> API. A dedicated DirEntry type lets us adjust the information returned over
> time, without breaking backwards compatibility and without resorting to
> ugly hacks like those in some of the time and stat APIs (or even our own
> codec info APIs)
> * it would be nice to see some relative performance numbers for NFS and
> CIFS network shares - the additional network round trips can make excessive
> stat calls absolutely brutal from a speed perspective when using a network
> drive (that's why the stat caching added to the import system in 3.3
> dramatically sped up the case of having network drives on sys.path, and why
> I thought AJ had a point when he was complaining about the fact we didn't
> expose the dirent data from os.listdir)
>
fwiw, I wouldn't wait for benchmark numbers.

A needless stat call when you've got the information from an earlier API
call is already brutal. It is easy to compute from existing ballparks
remote file server / cloud access: ~100ms, local spinning disk seek+read:
~10ms. fetch of stat info cached in memory on file server on the local
network: ~500us.  You can go down further to local system call overhead
which can vary wildly but should likely be assumed to be at least 10us.

You don't need a benchmark to tell you that adding needless >= 500us-100ms
blocking operations to your program is bad. :)

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

[Gregory P. Smith]

On Jun 28, 2014 12:49 PM, "Ben Hoyt"  wrote:
>
> >> But the underlying system calls -- ``FindFirstFile`` /
> >> ``FindNextFile`` on Windows and ``readdir`` on Linux and OS X --
> >
> > What about FreeBSD, OpenBSD, NetBSD, Solaris, etc. They don't provide
readdir?
>
> I guess it'd be better to say "Windows" and "Unix-based OSs"
> throughout the PEP? Because all of these (including Mac OS X) are
> Unix-based.

No, Just say POSIX.

>
> > It looks like the WIN32_FIND_DATA has a dwFileAttributes field. So we
> > should mimic stat_result recent addition: the new
> > stat_result.file_attributes field. Add DirEntry.file_attributes which
> > would only be available on Windows.
> >
> > The Windows structure also contains
> >
> >   FILETIME ftCreationTime;
> >   FILETIME ftLastAccessTime;
> >   FILETIME ftLastWriteTime;
> >   DWORDnFileSizeHigh;
> >   DWORDnFileSizeLow;
> >
> > It would be nice to expose them as well. I'm  no more surprised that
> > the exact API is different depending on the OS for functions of the os
> > module.
>
> I think you've misunderstood how DirEntry.lstat() works on Windows --
> it's basically a no-op, as Windows returns the full stat information
> with the original FindFirst/FindNext OS calls. This is fairly explict
> in the PEP, but I'm sure I could make it clearer:
>
> DirEntry.lstat(): "like os.lstat(), but requires no system calls on
Windows
>
> So you can already get the dwFileAttributes for free by saying
> entry.lstat().st_file_attributes. You can also get all the other
> fields you mentioned for free via .lstat() with no additional OS calls
> on Windows, for example: entry.lstat().st_size.
>
> Feel free to suggest changes to the PEP or scandir docs if this isn't
> clear. Note that is_dir()/is_file()/is_symlink() are free on all
> systems, but .lstat() is only free on Windows.
>
> > Does your implementation uses a free list to avoid the cost of memory
> > allocation? A short free list of 10 or maybe just 1 may help. The free
> > list may be stored directly in the generator object.
>
> No, it doesn't. I might add this to the PEP under "possible
> improvements". However, I think the speed increase by removing the
> extra OS call and/or disk seek is going to be way more than memory
> allocation improvements, so I'm not sure this would be worth it.
>
> > Does it support also bytes filenames on UNIX?
>
> > Python now supports undecodable filenames thanks to the PEP 383
> > (surrogateescape). I prefer to use the same type for filenames on
> > Linux and Windows, so Unicode is better. But some users might prefer
> > bytes for other reasons.
>
> I forget exactly now what my scandir module does, but for os.scandir()
> I think this should behave exactly like os.listdir() does for
> Unicode/bytes filenames.
>
> > Crazy idea: would it be possible to "convert" a DirEntry object to a
> > pathlib.Path object without losing the cache? I guess that
> > pathlib.Path expects a full  stat_result object.
>
> The main problem is that pathlib.Path objects explicitly don't cache
> stat info (and Guido doesn't want them to, for good reason I think).
> There's a thread on python-dev about this earlier. I'll add it to a
> "Rejected ideas" section.
>
> > I don't understand how you can build a full lstat() result without
> > really calling stat. I see that WIN32_FIND_DATA contains the size, but
> > here you call lstat().
>
> See above.
>
> > Do you plan to continue to maintain your module for Python < 3.5, but
> > upgrade your module for the final PEP?
>
> Yes, I intend to maintain the standalone scandir module for 2.6 <=
> Python < 3.5, at least for a good while. For integration into the
> Python 3.5 stdlib, the implementation will be integrated into
> posixmodule.c, of course.
>
> >> Should there be a way to access the full path?
> >> --
> >>
> >> Should ``DirEntry``'s have a way to get the full path without using
> >> ``os.path.join(path, entry.name)``? This is a pretty common pattern,
> >> and it may be useful to add pathlib-like ``str(entry)`` functionality.
> >> This functionality has also been requested in `issue 13`_ on GitHub.
> >>
> >> .. _`issue 13`: https://github.com/benhoyt/scandir/issues/13
> >
> > I think that it would be very convinient to store the directory name
> > in the DirEntry. It should be light, it's just a reference.
> >
> > And provide a fullname() name which would just return
> > os.path.join(path, entry.name) without trying to resolve path to get
> > an absolute path.
>
> Yeah, fair suggestion. I'm still slightly on the fence about this, but
> I think an explicit fullname() is a good suggestion. Ideally I think
> it'd be better to mimic pathlib.Path.__str__() which is kind of the
> equivalent of fullname(). But how does pathlib deal with unicode/bytes
> issues if it's the str function which has to return a str object? Or
> at least, it'd be very weird if __str__() returned bytes. But I think
> it'd need to if you passed bytes i

Re: [Python-Dev] Sad status of Python 3.x buildbots

[Gregory P. Smith]

On Wed, Sep 3, 2014 at 2:52 PM, Victor Stinner 
wrote:

>
> > ARMv7 3.x: "Read-only file system", Mercurial fails...
>
> I sent an email to Gregory P. Smith (owner).
>

For mine, its a case of me not having any monitoring for it. Sending me an
email worked.

In this case the SSD (a very fast usb3 memory stick) died after 9 months of
use. Not a surprise. I brought it back up on the internal flash for the
time being. I am in the process of setting up a new system (nvidia jetson
tk1) to replace this one that will be 1.5-3x faster depending on how you
measure it.

Monitoring idea for buildbots: If builders for all non-custom branches on a
buildbot are failing for more than some period of time or number of builds
each in a row, that is likely a sign of broken infrastructure or a common
platform specific code change that needs to be made.

anyone want to implement that?

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Gregory P. Smith]

On Wed, Sep 3, 2014 at 3:48 PM, Stephen J. Turnbull 
wrote:

> Guido van Rossum writes:
>
>  > lot: five years ago (when I worked at Google!) it was common to find
>  > internal services that required SSL but had a misconfigured certificate,
>  > and the only way to access those services was to override the browser
>  > complaints. Today (working at Dropbox, a much smaller company!) I don't
>  > even remember the last time I had to deal with such a browser complaint
> --
>
> I would tend to discount your recent experience, then.  Smaller (and
> possibly even more important in this fast-developing area, younger)
> organizations are a lot more nimble about things like this.
>

As a defensive data point: I don't remember a single instance of this
happening for Google internal services, at least since I arrived in 2007.
I'm not doubting that Guido remembers some thing(s) but in general people
here at Google would not stand for that, then or now. I would not call it
common, especially five years ago.

Common things I _have_ encountered over the years everywhere I've been both
internal and external: services that listen on the https port 443 but don't
have a valid cert as they are intended only for http port 80 access. Those
are becoming somewhat less common, the only thing I regularly see that on
anymore is random home router web config UIs as issuing a signed server
certificate for security hole ridden commodity embedded box is... a
challenge.

(I'm not commenting on the PEP plans as it seems the right things are
happening for now)

-gps @ Google
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (3.4): Remove the unimplemented but ignored without='-E' parameters being passed to

[Gregory P. Smith]

I misinterpreted the side effect this code was triggering. It turns out
that any kwarg would have the desired effect as a side effect (as seen in
dummyvar='' being passed later in this file for the same purpose).  I'm
looking at refactoring various uses of -E that interfere with running the
test suite in an environment where you need PYTHONHOME set. I will revert
this change or otherwise do the equivalent of reinstating its behavior in a
followup check-in as appropriate.

-gregory.p.smith

On Thu Jan 22 2015 at 10:05:56 PM gregory.p.smith <
python-check...@python.org> wrote:

> https://hg.python.org/cpython/rev/7b833bd1f509
> changeset:   94249:7b833bd1f509
> branch:  3.4
> parent:  94247:7f3ac2ce24ed
> user:Gregory P. Smith 
> date:Thu Jan 22 22:04:16 2015 -0800
> summary:
>   Remove the unimplemented but ignored without='-E' parameters being
> passed to
> script_helper.assert_python_failure().  No such feature has ever existed,
> thus it doesn't do what the comment claims.  (It does add a 'without'
> variable to the environment of the child process but that was not intended)
>
> files:
>   Lib/test/test_cmd_line.py |  6 ++
>   1 files changed, 2 insertions(+), 4 deletions(-)
>
>
> diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py
> --- a/Lib/test/test_cmd_line.py
> +++ b/Lib/test/test_cmd_line.py
> @@ -426,13 +426,11 @@
>  self.assertIn(b'Unknown option: -z', err)
>  self.assertEqual(err.splitlines().count(b'Unknown option: -z'),
> 1)
>  self.assertEqual(b'', out)
> -# Add "without='-E'" to prevent _assert_python to append -E
> -# to env_vars and change the output of stderr
> -rc, out, err = assert_python_failure('-z', without='-E')
> +rc, out, err = assert_python_failure('-z')
>  self.assertIn(b'Unknown option: -z', err)
>  self.assertEqual(err.splitlines().count(b'Unknown option: -z'),
> 1)
>  self.assertEqual(b'', out)
> -rc, out, err = assert_python_failure('-a', '-z', without='-E')
> +rc, out, err = assert_python_failure('-a', '-z')
>  self.assertIn(b'Unknown option: -a', err)
>  # only the first unknown option is reported
>  self.assertNotIn(b'Unknown option: -z', err)
>
> --
> Repository URL: https://hg.python.org/cpython
> ___
> Python-checkins mailing list
> python-check...@python.org
> https://mail.python.org/mailman/listinfo/python-checkins
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can Python Be Built Without Distutils

[Gregory P. Smith]

On Fri Jan 23 2015 at 11:20:02 AM M.-A. Lemburg  wrote:

> On 23.01.2015 19:48, Matthias Klose wrote:
> > On 01/23/2015 06:30 PM, Cyd Haselton wrote:
> >> Related to my earlier question regarding building Python on Android
> >> and an undefined reference to dlopen error...I have the following
> >> question:  Is it possible to build and install Python without having
> >> to build and install...or use...distutils?
> >>
> >> Some background:
> >> I can build the python interpreter on my device, and I can build a
> >> bunch of modules.  The problem appears when make reaches the part
> >> where setup.py is used to build and import modules...specifically when
> >> setup.py attempts to import distutils.core.
> >
> > you can do this using Setup.local. This works for me building additional
> > extensions as builtins.  It might require some tweaking to build
> everything.
>
> You may want to have a look at the Setup files we're using
> in eGenix PyRun, which uses them to force static builds of the
> various built-in extensions.
>
> Look for these files:
>
> PyRun/Runtime/Setup.PyRun-2.7
> PyRun/Runtime/Setup.PyRun-3.4
>
> in the source archives:
>
> http://www.egenix.com/products/python/PyRun/
>
> > Otoh, I would like to get rid off the setup.py altogether (/me ducks
> ...).
>
> Why ? It's great for finding stuff on your system and configuring
> everything without user intervention (well, most of the time :-)).
>

Because our setup.py is a nightmare of arbitrary code run in a linear
fashion with ad-hoc checks for things that are unlike how any other project
on the planet determines what is available on your system.  It may have
seemed "great" when it was created in 2001.  It really shows its age now.

It defeats build parallelism and dependency declaration.
It also prevents cross compilation.

Building an interpreter with a limited standard library on your build host
so that you can run said interpreter to have it drive the remainder of your
build is way more self hosting that we rightfully need to be for CPython.

-gps


> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source  (#1, Jan 23 2015)
> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
> >>> mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
> >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
> 
>
> : Try our mxODBC.Connect Python Database Interface for free ! ::
>
>eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>Registered at Amtsgericht Duesseldorf: HRB 46611
>http://www.egenix.com/company/contact/
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can Python Be Built Without Distutils

[Gregory P. Smith]

Why doesn't our Makefile supply that flag with the make parallelism level
in the sharedmods step?

On Sat Jan 24 2015 at 2:25:45 PM Antoine Pitrou  wrote:

> On Sat, 24 Jan 2015 21:53:06 +0100
> "M.-A. Lemburg"  wrote:
> >
> > BTW: Parallel execution, cross compilation can be added
> > to setup.py. I don't think parallel execution is all
> > that important,
>
> $ ./python setup.py build_ext --help
> [...]
>   --parallel (-j)  number of parallel build jobs
>
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] is a new buildbot gps-debian-profile-opt possible?

[Gregory P. Smith]

I want this buildbot to run "make profile-opt" instead of "make all" and
it'd be best if it could _not_ have --with-pydebug on the ./configure
command line.

I've got a small VM at the moment with one CPU and 600mb ram so just give
it a make -j2 at most to start with.

If that is possible, please assign the new buildbot name and password for
me and i'll get it up and running.

motivation for such a buildbot: https://bugs.python.org/issue22904 and any
potential similar issues.  fwiw, the major linux distros ship with
profile-opt interpreter builds as they are significantly faster.

thanks,
-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 468 (Ordered kwargs)

[Gregory P. Smith]

On Tue Jan 27 2015 at 2:13:08 PM Armin Rigo  wrote:

> Hi all,
>
> On 24 January 2015 at 11:50, Maciej Fijalkowski  wrote:
> > I would like to point out that we implemented rhettingers idea in PyPy
> > that makes all the dicts ordered by default and we don't have any
> > adverse performance effects (in fact, there is quite significant
> > memory saving coming from it). The measurments on CPython could be
> > different, but in principle OrderedDict can be implemented as
> > efficiently as normal dict.
>
> I would like to add that http://bugs.python.org/issue16991 is the same
> as today's dicts with an additional doubly-linked list for the order.
> I'm unsure why you would do that after the 2012 thread started by
> Raymond Hettinger, but anyway, don't conclude from only this that in
> the CPython case ordered dictionaries would be slower and bigger.  My
> guess is that, with a simple port of what is now in PyPy, they would
> not be (but of course no-one can be sure at this point).  Let's say,
> if you could imagine that CPython's dictionaries, tomorrow, are always
> magically fully ordered, then would it still be a bad idea?
>

It is a potentially bad idea if order is the default behavior of iteration,
items(), keys() and values(). Ideally order should only be exposed when
explicitly asked for to help prevent bugs and mitigate potential
information leaks.

But I'm not sure how big of a deal this actually is. The insertion order
nicely doesn't give away anything related to the hash seed used for hash
randomization which is a nice bonus over today's implementation (and 2.7 &
3.3's very poor hash randomization implementation).

Experience cleaning up our huge code base at work to turn on hash
randomization by default a couple years ago has shown that people depend on
iteration order in code often without intending to. This often leads to
latent bugs. Keep iteration order unstable by default and you prevent
people from doing that. Make people request an ordered or stable iteration
when their code explicitly needs it.

If such a discussion would resurface (soon or "one day"), and if other
> related issues are resolved (like what to do in Jython and
> IronPython), and if the conclusion would tentatively turn out
> positive... then, provided there would at that point still be no
> "Raymond-style" implementation of dicts, I would volunteer to port
> PyPy's one to CPython[1].  As you may have guessed I don't consider
> this particularly likely to occur, but it is a standing offer
> nevertheless :-)
>

CPython should benefit from it regardless for the memory savings alone.

-gps


>
> A bientôt,
>
> Armin.
>
>
> [1]  Someone could also do such a port for the goal of getting an
> alternate `odictobject.c`.  He would be welcome to #pypy to get some
> help from the PyPy guys, including me --- but my offer above doesn't
> apply in this case.  I want to remove a thorn in the foot of
> python-dev discussing about the language; I'm not really interested in
> contributing to the `collections.OrderedDict` type.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can Python Be Built Without Distutils

[Gregory P. Smith]

On Sun Jan 25 2015 at 7:08:53 AM Antoine Pitrou  wrote:

> On Sun, 25 Jan 2015 05:22:48 +
> "Gregory P. Smith"  wrote:
> > Why doesn't our Makefile supply that flag with the make parallelism level
> > in the sharedmods step?
>
> If I run "make -j4" here, it works (on the default branch).
>

Thanks, glad to see that this was added! :)

This looks like a feature that was added in 3.5.  Rather than passing the
flag to setup.py from the Makefile (where I had been looking for it),
setup.py looks at the MAKEFLAGS environment variable and turns on
distutils.common.build_ext's parallel flag if "-j" is found in that. By
default it will then do a parallel build using os.cpu_count() number of
tasks.

-gps


>
> Regards
>
> Antoine.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Encoding of PyFrameObject members

[Gregory P. Smith]

On Thu Feb 05 2015 at 4:36:30 PM Francis Giraldeau <
francis.girald...@gmail.com> wrote:

> I need to access frame members from within a signal handler for tracing
> purpose. My first attempt to access co_filename was like this (omitting
>  error checking):
>
> PyFrameObject *frame = PyEval_GetFrame();
> PyObject *ob = PyUnicode_AsUTF8String(frame->f_code->co_filename)
> char *str = PyBytes_AsString(ob)
>
> However, the function PyUnicode_AsUTF8String() calls PyObject_Malloc(),
> which is not reentrant. If the signal handler nest over PyObject_Malloc(),
> it causes a segfault, and it could also deadlock.
>
> Instead, I access members directly:
> char *str = PyUnicode_DATA(frame->f_code->co_filename);
> size_t len = PyUnicode_GET_DATA_SIZE(frame->f_code->co_filename);
>
> Is it safe to assume that unicode objects co_filename and co_name are
> always UTF-8 data for loaded code? I looked at the PyTokenizer_FromString()
> and it seems to convert everything to UTF-8 upfront, and I would like to
> make sure this assumption is valid.
>

The faulthandler module calls into Python/traceback.c in signal context
which has some low level code for extracting something useful from
PyUnicode objects without triggering a conversion:

https://hg.python.org/cpython/file/f1a82e949fb8/Python/traceback.c#l531

That code is written to write output to a fd. It's much more useful for
signal handler tracing data collection from a signal handler if you
refactor it to dump its output into a preallocated char* buffer.

-gps


>
> Thanks!
>
> Francis
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Encoding of PyFrameObject members

[Gregory P. Smith]

To get at the Python thread state from a signal handler (using 2.7 as a
reference here; but i don't believe 3.4 has changed this part much) you
need to modify the interpreter to expose pystate.c's "autoTLSkey" and
thread.c's "struct key" as well as "keyhead" and "keymutex".

>From there, in your signal handler you must try to acquire the newly
exposed keymutex and do nothing if you were unable to get it.  If you did
acquire it (rare not to), you can walk the keyhead list looking for
autoTLSkey to find the current valid thread state.

I had an intern (hi Elena!) write a signal sampling based low overhead
Python CPU profiler based on that last summer. I believe there are still
bugs to shaken out (if they are even possible to fix... Armin's comments
are true: signal handler code is super limited). I am stating this here
because I want someone to pester me at PyCon if I haven't released our work
as a proof of concept by then. The important take away: From what I could
figure out, you need to modify the CPython interpreter to be more amenable
to such introspection.

A downside of a signal based profiler: *ALL* of the EINTR mishandling bugs
within the Python interpreter, stdlib, and your own code will show up in
your application. So until those are fixed (hooray for Antoine's PEP!), it
may not be practical for use on production processes which is sort of the
entire point of a low overhead sampling profiler...

I'd like to get a buildbot setup that runs the testsuite while a continual
barrage of signals are being generated. We really don't stress test that
stuff (as evidence by the EINTR mishandling issues that are rampant) as
non-fatal signals are so rare for most things... until they aren't.

As a side note and encouragement: I wonder what PyPy could do for
dynamically enabled and disabled low overhead CPU profiling. (take that as
a hint that I want someone else to get extremely creative!)

-gps

On Sat Feb 07 2015 at 1:34:26 PM Greg Ewing 
wrote:

> Maciej Fijalkowski wrote:
> > However, you can't access thread
> > locals from signal handlers (since in some cases it mallocs, thread
> > locals are built lazily if you're inside the .so, e.g. if python is
> > built with --shared)
>
> You might be able to use Py_AddPendingCall to schedule
> what you want done outside the context of the signal
> handler.
>
> The call will be made by the main thread, though,
> so if you need to access the frame of whatever thread
> was running when the signal occured, you will have
> to track down its PyThreadState somehow and get the
> frame from there. Not sure what would be involved
> in doing that.
>
> --
> Greg
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Encoding of PyFrameObject members

[Gregory P. Smith]

On Sun Feb 08 2015 at 12:21:36 AM Victor Stinner 
wrote:

>
> Le 8 févr. 2015 05:39, "Gregory P. Smith"  a écrit :
> > From there, in your signal handler you must try to acquire the newly
> exposed keymutex and (...)
>
> Stop! Acquiring a lock in a signal handler is just a crazy idea. It's very
> far from an async signal-safe function.
>
I'd normally agree... In this case, sem_trywait() is non-blocking. It is
unfortunately not on the official POSIX list of async-signal-safe functions
(of the SEMs, only sem_post() is) but many implementations actually are
safe, documented or not.

https://github.com/lattera/glibc/blob/master/nptl/sysdeps/unix/sysv/linux/x86_64/sem_trywait.S
- appears safe.

So long as the underlying semaphore is implemented using an atomic
instruction on the target platform I wouldn't expect any implementation of
sem_trywait to be async signal unsafe. (nor would I depend on that without
checking)

-gps

> > So until those are fixed (hooray for Antoine's PEP!), ...
>
> I wrote the PEP with Charles François Natali, but Charles wrote the whole
> implementation. Antoine also helped us and approved the PEP. It's the
> french connection! Just to say that Charles did most of the work.
>
> Victor
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Gregory P. Smith]

On Sat Feb 14 2015 at 3:17:51 AM Victor Stinner 
wrote:

> 2015-02-14 11:57 GMT+01:00 Victor Stinner :
> > I propose something else: a DirEntry.inode read-only property (...)
>
> Full DirEntry API:
>
> - name (str) attribute
> - path (str) read-only property, created at the first call
> - inode (int or None) attribute <=== my proposition
>

+1 we need to provide the inode (we shouldn't be throwing anything from the
underlying directory entry away when possible). But...

I think the "or None" semantics are a bad idea.  It'd be better for this to
raise AttributeError on Windows so that someone can't write the most
natural form of code assuming that inode is valid and have it appear to
work on Windows when in fact it'd do the wrong thing.


> - is_dir(*, follow_symlinks=True)
> - is_file(*, follow_symlinks=True)
> - is_symlink(*, follow_symlinks=True)
> - stat(*, follow_symlinks=True)
>
> is_dir(), is_file(), is_symlink() and stat() are method because they
> may all require a syscall (os.stat or os.lstat). They all cache their
> result. In some cases, the result is already known when DirEntry is
> created. In most cases, a single call to os.stat() is required to fill
> the result of all methods.
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Gregory P. Smith]

That suggests the .inode() method approach makes more sense then.

On Sat, Feb 14, 2015, 12:44 PM Antoine Pitrou  wrote:

> On Sat, 14 Feb 2015 15:32:07 -0500
> Ben Hoyt  wrote:
> > > +1 we need to provide the inode (we shouldn't be throwing anything
> from the
> > > underlying directory entry away when possible). But...
> > >
> > > I think the "or None" semantics are a bad idea.  It'd be better for
> this to
> > > raise AttributeError on Windows so that someone can't write the most
> natural
> > > form of code assuming that inode is valid and have it appear to work on
> > > Windows when in fact it'd do the wrong thing.
> >
> > +1 for inode support. I agree with the above -- it should either raise
> > AttributeError on Windows if it's not going to be set ... or it should
> > be more like Victor's original proposal where .inode() is a method
> > that calls stat on Windows. I don't have strong feelings.
>
> The whole point of scandir is to expose low-level system calls in a
> cross-platform way. If you start raising some exceptions on some
> platforms then that quality disappears.
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] getstatusoutput()'s behavior changed in 3.3.4 and 3.4

[Gregory P. Smith]

While porting some code from 2.7 to 3.4 I discovered that
command.getstatusoutput() (renamed to subprocess.getstatusoutput() in 3.x)
had changed. Surprise!

The code was working under an earlier version of 3.3 but broke when I ran
it on 3.4.  Nowhere was this documented that I could find. Tracking down
what changed, I discovered it was unintentional due to the meaning of the
returned status int never being tested. http://bugs.python.org/issue23508
filed...

Given it has shipped in several stable 3.x releases and as part of some
major Linux distros, reverting the behavior change seems wrong. The new
behavior is nicer and more consistent with the rest of the subprocess
module. I suggest just documenting it and moving on. It seems too late to
fix this mistake without causing additional headaches. Anyone disagree?

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] how to inspect if something includes a bound first param

[Gregory P. Smith]

inspect.getargspec(method) and inspect.signature(method) both include the
'self' parameter but how are we to figure out from method itself that it is
actually bound and that its first parameter is expected to be a bound
instance?

So far I've come up with:
 *inspect.ismethod(method) or inspect.ismethoddescriptor(method)*

But that is still not quite right as it remains False in 3.4 for the simple
case of:

>>> class A:
...  def x(self):
...pass
...
>>> inspect.ismethod(A.x)
False
>>> inspect.ismethoddescriptor(A.x)
False
>>> inspect.signature(A.x).parameters
mappingproxy(OrderedDict([('self', )]))
>>> inspect.getargspec(A.x)
ArgSpec(args=['self'], varargs=None, keywords=None, defaults=None)

The above works if use it on object.__init__, but not on a method of an
uninstantiated class.  (it also works on the instantiated A().x)

Checking if *(inspect.isfunction(method) and method.__qualname__ !=
method.__name__)* perhaps but that seems questionably hacky. Is there a
right way to do this via the inspect module that I've missed?

It seems like that'd be nice, but I don't feel like I know enough to write
up a feature request for it yet.  (granted I hope nobody *wants* to write
code that does this...)

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use ptyhon -s as default shbang for system python executables/daemons

[Gregory P. Smith]

On Wed, Mar 18, 2015 at 9:48 AM Barry Warsaw  wrote:

> On Mar 18, 2015, at 05:31 PM, Victor Stinner wrote:
>
> >Does it work to pass command line options to Python in the shebang?
>
> Yes, but only one "word", thus -Es or -I.
>
> We've often mused about whether it makes sense to have two Pythons on the
> system.  One for system scripts and another for users.  System Python
> ('/usr/bin/spython' perhaps) would be locked down and only extensible by
> system packages.  On Debuntu that might mean no /usr/local on sys.path.  It
> would also have a much more limited set of installed packages, i.e. only
> those
> needed to support system functionality.
>

I recommend this. It'd be nice to see a common OS distro actually do it.

For a system-only Python lockdown you should remove distutils and pip to
prevent anyone from easily installing anything that isn't a distro package
into it. Possibly even removing its site-packages directory and sys.path
entry all together (things your distro needs to install could mix directly
with the stdlib packages)


> /usr/bin/python2 and /usr/bin/python3 then would be user tools, with all
> the
> goodness they currently have.
>
> It's never gotten much farther than musings, but protecting the system
> against
> the weird things people install would be a good thing.  OTOH, this feels a
> lot
> like virtual environments so maybe there's something useful to be mined
> there.
>

While people sometimes suggest virtualenv as a solution for this. It isn't
really the same thing. It isn't a hermetic clone of the original
interpreter.  It copies the main binary but symlinks back to much of the
stdlib.  So any existing virtualenv can be an out of date unsupported mix
of both after the original interpreter is updated. This has caused users
problems in the past with some minor version updates where an internal
non-public API used between some binary and a stdlib module was updated as
part of a bugfix. Suddenly they didn't match up for existing virtualenvs.

virtualenv is an amazing hack that I promote to most anyone for their own
applications use with the occasional known caveats (solvable by regurly
rebuilding your virtualenvs)...  But I wouldn't want to see it used for the
core OS itself even though it sounds better at first glance.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword-only parameters

[Gregory P. Smith]

On Tue, Apr 14, 2015 at 1:56 PM Alexander Walters 
wrote:

> Lacking anything anyone else says... the use case for keyword only
> arguments (where they actually make the code better rather than simply
> being different) is rather limited.
>
> I disagree.  For parameters not often passed or beyond 3-4 parameters
being used on any API I actually prefer that we start using keyword only
parameters for those.

I'd say feel free to use keyword only parameters when you believe they make
sense.  Most of the stdlib does not do it because most of the APIs were
defined in the 2.x era and maintaining compatibility with them is desirable
for 2AND3 compatible code. But for new things, feel free.

-gps


> On 4/14/2015 13:40, Eric V. Smith wrote:
> > I'm working on adding a numeric_owner parameter to some tarfile methods
> > (http://bugs.python.org/issue23193),
> >
> > In a review, Berker suggested making the parameter keyword-only. I agree
> > that you'd likely never want to pass just "True", but that
> > "numeric_owner=True" would be a better usage.
> >
> > But, I don't see a lot of keyword-only parameters being added to stdlib
> > code. Is there some position we've taken on this? Barring someone saying
> > "stdlib APIs shouldn't contain keyword-only params", I'm inclined to
> > make numeric_owner keyword-only.
> >
> > Is there anything stopping me from making it keyword-only?
> >
> > Thanks.
> > Eric.
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/tritium-list%40sdamon.com
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Surely "nullable" is a reasonable name?

[Gregory P. Smith]

On Mon, Apr 20, 2015 at 6:55 AM Barry Warsaw  wrote:

> On Apr 19, 2015, at 01:19 AM, Larry Hastings wrote:
>
> >We should rename "types" to "accept".  "accept" should takes a set of
> types;
> >these types specify the types of Python objects the Clinic parameter
> should
> >accept.  For the funny pseudo-types needed in some Clinic declarations
> >("buffer", "robuffer", and "rwbuffer"), Clinic provides empty class
> >declarations so these behave like types too.
>
> Having only followed the AC discussions tangentially, I have to say that
> the
> above suggestion and the given examples make a lot more intuitive sense to
> me.
>

+1 as well: gps(accept={NewlyProposedArgumentClinicSyntax, Cookies})


> I had the same initial thought as Glenn regarding type annotations.  It's
> fine
> that they're separate concepts, but it's also helpful that Larry's
> suggestion
> above seems to align them better.
>
> Cheers,
> -Barry
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Gregory P. Smith]

On Tue, Apr 21, 2015 at 12:50 AM Antoine Pitrou  wrote:

> On Mon, 20 Apr 2015 20:43:38 -0400
> "R. David Murray"  wrote:
> > +1 to this from me too. I'm afraid that means I'm -1 on the PEP.
> >
> > I didn't write this in my earlier email because I wasn't sure about it,
> > but my gut reaction after reading Harry's email was "if type annotations
> > are used in the stdlib, I'll probably stop contributing".  That doesn't
> > mean that's *true*, but that's the first time I've ever had that
> > thought, so it is probably worth sharing.
>
> I think it would be nice to know what the PEP means for daily stdlib
> development. If patches have to carry typing information each time they
> add/enhance an API that's an addition burden. If typing is done
> separately by interested people then it sounds like it wouldn't have
> much of an impact on everyone else's workflow.
>

Separately by interested people.  That won't change until tools appear and
mature that help maintain the types for us.  (if ever)

Nobody wants unreadable code.  Nobody is proposing to make unreadable code
happen or encourage its creation.

One thing I feel is often overlooked in the discussion on this PEP: It is
about creating a unified type expression syntax for everyone working on
Python typing to centralize around. Regardless of if the PEPs version falls
short for some purposes. It allows for sharing work. There are multiple
ongoing projects that are trying to make use of type information with
Python, this allows them to all speak the same language.  (MyPy,
MicroPython, Cython, the static analyzer we are trying to create at Google,
several others not on the top of my head I'm sure, etc.)

We will not be putting type annotations anywhere in the stdlib or expecting
anyone else to maintain them there. That would never happen until tools
that are convincing enough in their utility for developers to _want_ to use
are available and accepted.  That'll be a developer workflow thing we could
address with a later PEP. IF it happens at all.

I view most of this thread as FUD. The fear is understandable, I'm trying
to tell people to stop panicing. This PEP does not mean that Python is
suddenly going to become unreadable. Just that a set of people working on a
common goal have a way to communicate with one another. If that work bears
fruit, great, it'll be shared and provided as tools that people want to
use. If not, it won't matter in the slightest and the typing module and
this PEP will be relegated to history. This is a 100% non-invasive PEP. No
new keywords!

Motivation behind static analysis and type checkers comes directly from the
success of the type annotations and checking done to Javascript in Google's
javascript Closure compiler that has been available for years. Steven
mentioned Facebook's Flow which does a similar thing. These are both opt-in
and by and large we've found that developers love using them in any decent
sized code base. That model is the goal for any of our Python typing
related projects to get to. If developers don't _want_ to use it in the
end, we have failed and they can happily continue not using it because it
was never required.

The reason this PEP exists is for tool developers to be able to do their
thing and prove to everyone that it is (a) possible and (b) genuinely
useful. IF that proves successful, we can consider if we need a saner
syntax for anyone to want to use it. For now we've got this PEP which is a
bit of a hack using the Python 3 annotations and a typing module but at the
same time doesn't involve any language changes we might regret. I call that
a win!

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.x Adoption for PyPI and PyPI Download Numbers

[Gregory P. Smith]

On Tue, Apr 21, 2015 at 10:55 AM Donald Stufft  wrote:

> Just thought I'd share this since it shows how what people are using to
> download things from PyPI have changed over the past year. Of particular
> interest to most people will be the final graphs showing what percentage of
> downloads from PyPI are for Python 3.x or 2.x.
>
> As always it's good to keep in mind, "Lies, Damn Lies, and Statistics".
> I've
> tried not to bias the results too much, but some bias is unavoidable. Of
> particular note is that a lot of these numbers come from pip, and as of
> version
> 6.0 of pip, pip will cache downloads by default. This would mean that older
> versions of pip are more likely to "inflate" the downloads than newer
> versions
> since they don't cache by default. In addition if a project has a file
> which
> is used for both 2.x and 3.x and they do a ``pip install`` on the 2.x
> version
> first then it will show up as counted under 2.x but not 3.x due to caching
> (and
> of course the inverse is true, if they install on 3.x first it won't show
> up
> on 2.x).
>
> Here's the link: https://caremad.io/2015/04/a-year-of-pypi-downloads/
>
> Anyways, I'll have access to the data set for another day or two before I
> shut down the (expensive) server that I have to use to crunch the numbers
> so if
> there's anything anyone else wants to see before I shut it down, speak up
> soon.
>

Thanks!

I like your focus on particular packages of note such as django and
requests.

How do CDNs influence these "lies"?  I thought the download counts on PyPI
were effectively meaningless due to CDN mirrors fetching and hosting things?

Do we have user-agent logs from all PyPI package CDN mirrors or just from
the master?

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 552: deterministic pycs

[Gregory P. Smith]

+1 on this PEP.

The TL;DR summary of this PEP:
  The pyc date+length metadata check was a convenient hack.  It still works
well for many people and use cases, it isn't going away.
  PEP 552 proposes a new alternate hack that relies on file contents
instead of os and filesystem date metadata.
Assumption: The hash function is significantly faster than re-parsing
the source.  (guaranteed to be true)

Questions:

Input from OS package distributors would be interesting.  Would they use
this?  Which way would it impact their startup time (loading the .py file
vs just statting it.  does that even matter?  source files are often
eventually loaded for linecache use in tracebacks anyways)?

Would they benefit from a pyc that can contain _both_ timestamp+length, and
the source_hash?  if both were present, I assume that only one would be
checked at startup.  i'm not sure what would make the decision of what to
check.  one fails, check the other?  i personally do not have a use for
this case so i'd omit the complexity without a demonstrated need.

Something to also state in the PEP:

This is intentionally not a "secure" hash.  Security is explicitly a
non-goal.

Rationale behind my support:

We use a superset of Bazel at Google (unsurprising) and have had to jump
through a lot of messy hoops to deal with timestamp metadata winding up in
output files vs deterministic builds.  What Benjamin describes here sounds
exactly like what we would want.

It allows deterministic builds in distributed build and cached operation
systems where timestamps are never going to be guaranteed.

It allows the check to work on filesystems which do not preserve timestamps.

Also importantly, it allows the check to be disabled via the check_source
bit.  Today we use a modified importer at work that skips checking
timestamps anyways as the way we ship applications where the entire set of
dependencies present is already guaranteed at build time to be correct and
being modified at runtime is not possible or not a concern. This PEP would
avoid the need for an extra importer or modified interpreter logic to make
this happen.

-G

On Thu, Sep 7, 2017 at 3:47 PM Benjamin Peterson 
wrote:

>
>
> On Thu, Sep 7, 2017, at 14:43, Guido van Rossum wrote:
> > On Thu, Sep 7, 2017 at 2:40 PM, Benjamin Peterson 
> > wrote:
> >
> > >
> > >
> > > On Thu, Sep 7, 2017, at 14:19, Guido van Rossum wrote:
> > > > Nice one.
> > > >
> > > > It would be nice to specify the various APIs needed as well.
> > >
> > > The compileall and py_compile ones?
> > >
> >
> > Yes, and the SipHash mod to specify the key you mentioned.
>
> Done.
>
> >
> > >
> > > > Why do you keep the mtime-based format as an option? (Maybe because
> it's
> > > > faster? Did you measure it?)
> > >
> > > I haven't actually measured anything, but stating a file will
> definitely
> > > be faster than reading it completely and hashing it. I suppose if the
> > > speed difference between timestamp-based and hash-based pycs turned out
> > > to be small we could feel good about dropping the timestamp format
> > > completely. However, that difference might be hard to determine
> > > definitely as I expect the speed hit will vary widely based on system
> > > parameters such as disk speed and page cache size.
> > >
> > > My goal in this PEP was to preserve the current pyc invalidation
> > > behavior, which works well today for many use cases, as the default.
> The
> > > hash-based pycs are reserved for distribution and other power use
> cases.
> > >
> >
> > OK, maybe you can clarify that a bit in the PEP.
>
> I've added a paragraph to the Rationale section.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 556: Threaded garbage collection

[Gregory P. Smith]

On Fri, Sep 8, 2017 at 12:52 PM Antoine Pitrou  wrote:

> On Fri, 8 Sep 2017 12:40:34 -0700
> Nathaniel Smith  wrote:
> >
> > PyPy just abandons everything when shutting down, instead of running
> > finalizers. See the last paragraph of :
> >
> http://doc.pypy.org/en/latest/cpython_differences.html#differences-related-to-garbage-collection-strategies
> >
> > So that might be a useful source of experience.
>
> CPython can be embedded in applications, though, and that is why we try
> to be a bit more thorough during the interpreter cleanup phase.
>

Indeed.  My gut feeling is that proposing to not run finalizers on
interpreter shutdown is a non-starter and would get the pep rejected.
We've previously guaranteed that they were run unless the process dies via
an unhandled signal or calls os._exit() in CPython.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup optimization: script vs. service

[Gregory P. Smith]

On Mon, Oct 2, 2017 at 8:03 AM Victor Stinner 
wrote:

> 2017-10-02 16:48 GMT+02:00 Christian Heimes :
> > That approach could work, but I think that it is the wrong approach. I'd
> > rather keep Python optimized for long-running processes and introduce a
> > new mode / option to optimize for short-running scripts.
>
> "Filling caches on demand" is an old pattern. I don't think that we
> are doing anything new here.
>
> If we add an opt-in option, I would prefer to have an option to
> explicitly "fill caches", rather than the opposite.
>

+1 the common case benefits from the laziness.

The much less common piece of code that needs to pre-initialize as much as
possible to avoid work happening at an inopportune future time (prior to
forking, while handling latency sensitive real time requests yet still
being written in CPython, etc.) knows its needs and can ask for it.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can Python guarantee the order of keyword-only parameters?

[Gregory P. Smith]

On Mon, Nov 27, 2017 at 10:13 AM Larry Hastings  wrote:

>
>
> First, a thirty-second refresher, so we're all using the same terminology:
>
> A *parameter* is a declared input variable to a function.
> An *argument* is a value passed into a function.  (*Arguments* are stored
> in *parameters.*)
>
> So in the example "def foo(clonk): pass; foo(3)", clonk is a parameter,
> and 3 is an argument. ++
>
>
> Keyword-only arguments were conceived of as being unordered.  They're
> stored in a dictionary--by convention called **kwargs--and dictionaries
> didn't preserve order.  But knowing the order of arguments is occasionally
> very useful.  PEP 468 proposed that Python preserve the order of
> keyword-only arguments in kwargs.  This became easy with the
> order-preserving dictionaries added to Python 3.6.  I don't recall the
> order of events, but in the end PEP 468 was accepted, and as of 3.6 Python
> guarantees order in **kwargs.
>
> But that's arguments.  What about parameters?
>
> Although this isn't as directly impactful, the order of keyword-only
> parameters *is* visible to the programmer.  The best way to see a
> function's parameters is with inspect.signature, although there's also the
> deprecated inspect.getfullargspec; in CPython you can also directly examine
> fn.__code__.co_varnames.  Two of these methods present their data in a way
> that preserves order for all parameters, including keyword-only
> parameters--and the third one is deprecated.
>
> Python must (and does) guarantee the order of positional and
> positional-or-keyword parameters, because it uses position to map arguments
> to parameters when the function is called.  But conceptually this isn't
> necessary for keyword-only parameters because their position is
> irrelevant.  I only see one place in the language & library that addresses
> the ordering of keyword-only parameters, by way of omission.  The PEP for
> inspect.signature (PEP 362) says that when comparing two signatures for
> equality, their positional and positional-or-keyword parameters must be in
> the same order.  It makes a point of *not* requiring that the two
> functions' keyword-only parameters be in the same order.
>
> For every currently supported version of Python 3, inspect.signature and
> fn.__code__.co_varnames preserve the order of keyword-only parameters.
> This isn't surprising; it's basically the same code path implementing those
> as the two types of positional-relevant parameters, so the most
> straightforward implementation would naturally preserve their order.  It's
> just not guaranteed.
>
> I'd like inspect.signature to guarantee that the order of keyword-only
> parameters always matches the order they were declared in.  Technically
> this isn't a language feature, it's a library feature.  But making this
> guarantee would require that CPython internally cooperate, so it's kind of
> a language feature too.
>
> Does this sound reasonable?  Would it need a PEP?  I'm hoping for "yes"
> and "no", respectively.
>

Seems reasonable to me.  I'm in the "yes" and "no" respectively "just do
it" camp on this if want to see it happen.  The groundwork was already laid
for this by using the order preserving dict in 3.6.  Having the inspect
module behave in a similar manner follows naturally from that.

-gps


>
>
> Three final notes:
>
>- Yes, I do have a use case.  I'm using inspect.signature metadata to
>mechanically map arguments from an external domain (command-line arguments)
>to a Python function.  Relying on the declaration order of keyword-only
>parameters would elegantly solve one small problem.
>- I asked Armin Rigo about PyPy's support for Python 3.  He said it
>should already maintain the order of keyword-only parameters, and if I ever
>catch it not maintaining them in order I should file a bug.  I assert that
>making this guarantee would be nearly zero effort for any Python
>implementation--I bet they all already behave this way, all they need is a
>test case and some documentation.
>- One can extend this concept to functools.partial and
>inspect.Signature.bind: should its transformations of keyword-only
>parameters also maintain order in a consistent way?  I suspect the answer
>there is much the same--there's an obvious way it should behave, it almost
>certainly already behaves that way, but it doesn't guarantee it.  I don't
>think I need this for my use case.
>
>
>
> */arry*
>
> ++ Yes, that means "Argument Clinic" should really have been called
> "Parameter Clinic".  But the "Parameter Clinic" sketch is nowhere near as
> funny.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/lis

[Python-Dev] pep-0557 dataclasses top level module vs part of collections?

[Gregory P. Smith]

It seems a suggested use is "from dataclasses import dataclass"

But people are already familiar with "from collections import namedtuple"
which suggests to me that "from collections import dataclass" would be a
more natural sounding API addition.

But the dataclasses module has additional APIs beyond @dataclass which
clearly do not belong at the top level in collections.

Idea: How about making the current dataclasses.dataclass decorator function
instead be a callable class instance (ie: it still functions as property,
todays dataclasses.dataclass becomes collections.dataclass.__call__) with
all of the current contents of the dataclasses module as attributes of a
collections.dataclass class/instance singleton?

It feels like a more natural API to me:

from collections import dataclass
@dataclass
class ...

and the following APIs show up on dataclass itself:

dataclass.Field, dataclass.field, dataclass.fields, dataclass.make,
dataclass.astuple, dataclass.replace, dataclass.asdict,
dataclass.FrozenInstanceError, dataclass.InitVar

instead of being in a separate dataclasses module and being a different
style of thing to import than namedtuple.

[ if this was discussed earlier for this pep and rejected and I missed it,
my apologies, just drop me a reference to that thread if you've got one ]

This isn't a blocker for me.  I like having a dataclass implementation no
matter how we arrange it.  If we go with what's checked in today, a top
level dataclasses module, so be it.  I'm not going to bikeshed this to
death it just feels odd to have such an API outside of collections but
figured it was worth suggesting.

Part of me just doesn't like the plural dataclasses module name.  I can get
over that.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Gregory P. Smith]

On Thu, Dec 21, 2017 at 3:10 PM MRAB  wrote:

> On 2017-12-21 22:45, Chris Barker wrote:
> > On Thu, Dec 21, 2017 at 11:55 AM, Terry Reedy  > > wrote:
> >
> > I think the understanding problem with this feature arises from two
> > factors: using annotations to define possibly un-initialized slots
> > is non-obvious; a new use of annotations for something other than
> > static typing is a bit of a reversal of the recent pronouncement
> > 'annotations should only be used for static typing'.
> >
> >
> > you know, that may be where part of my confusion came from -- all the
> > talk lately has been about "type hints" and "type annotations" -- the
> > idea of "arbitrary annotations" has been lost.
> >
> > Therefore, getting the permanent doc 'right' is important.
> >
> >
> > yup.
> >
> > @dataclass
> > class C:
> >  x
> >  y = 0
> >
> > I think the doc should explicitly say that uninitialized fields
> > require annotation with something (anything, not necessarily a type)
> > simply to avoid NameError during class creation.
> >
> >
> > would this be possible?
> >
> > @dataclass
> > class C:
> >  x:
> >  y: = 0
> >
> > That is -- the colon indicates an annotation, but in this case, it's a
> > "nothing" annotation.
> >
> >
> "..." or "pass", perhaps?
>
> @dataclass
> class C:
>   x: ...
>   y: ... = 0
>
> or:
>
> @dataclass
> class C:
>   x: pass
>   y: pass = 0
>

pass does not currently parse in that context.  Otherwise I was thinking
the same thing.  But we already have ... which does - so I'd suggest that
for people who are averse to importing anything from typing and using the
also quite readable Any.  (ie: document this as the expected practice with
both having the same meaning)

While I consider the annotation to be a good feature of data classes, it
seems worth documenting that people not running a type analyzer should
avoid declaring a type. A worse thing than no-type being specified is a
wrong type being specified. That appearing in a library will break people
who need their code to pass the analyzer and pytype, mypy, et. al. could be
forced to implement a typeshed.pypi of sorts containing blacklists of known
bad annotations in public libraries and/or actually correct type
specification overrides for them.

As for problems with order, if we were to accept

@dataclass
class Spam:
beans = True
ham: bool

style instead, would it be objectionable to require keyword arguments only
for dataclass __init__ methods?  That'd get rid of the need to care about
order.  (but would annoy people with small 2-3 element data classes... so
I'm assuming this idea is already rejected)

-gps


>
> > It's a syntax error now, but would it be possible to change that? Or
> > would the parsing be ambiguous? particularly in other contexts.
> >
> > of course, then we'd need something to store in as a "nothing"
> > annotation -- empty string? None? (but None might mean something) create
> > yet anther type for "nothing_annotation"
> >
> > Hmm, I may have talked myself out of it
> >
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] is typing optional in dataclasses?

[Gregory P. Smith]

(subject for this sub-thread updated)

On Thu, Dec 21, 2017 at 4:08 PM Chris Barker  wrote:

> On Thu, Dec 21, 2017 at 3:36 PM, Gregory P. Smith  wrote:
>
>
>>  But we already have ... which does - so I'd suggest that for people who
>> are averse to importing anything from typing and using the also quite
>> readable Any.  (ie: document this as the expected practice with both having
>> the same meaning)
>>
>
> I don't think they do, actually - I haven't been following the typing
> discussions, but someone in this thread said that ... means "use the type
> of teh default" or something like that.
>

indeed, they may not.  though if that is the definition is it reasonable to
say that type analyzers recognize the potential recursive meaning when the
_default_ is ... and treat that as Any?

another option that crossed my mind was "a: 10" without using =.  but that
really abuses __attributes__ by sticking the default value in there which
the @dataclass decorator would presumably immediately need to undo and fix
up before returning the class.  but I don't find assigning a value without
an = sign to be pythonic so please lets not do that! :)


>
>> While I consider the annotation to be a good feature of data classes, it
>> seems worth documenting that people not running a type analyzer should
>> avoid declaring a type.
>>
>
> +1 !
>
>
>> A worse thing than no-type being specified is a wrong type being
>> specified. That appearing in a library will break people who need their
>> code to pass the analyzer and pytype, mypy, et. al. could be forced to
>> implement a typeshed.pypi of sorts containing blacklists of known bad
>> annotations in public libraries and/or actually correct type specification
>> overrides for them.
>>
>
> and the wrong type could be very common -- folks using "int", when float
> would do just fine, or "list" when any iterable would do, the list goes on
> and on. Typing is actually pretty complex in Python -- it's hard to get
> right, and if you aren't actually running a type checker, you'd never know.
>

Yeah, that is true.  int vs float vs Number, etc.  It suggests means we
shouldn't worry about this problem at all for the pep 557 dataclasses
implementation.  Type analyzers by that definition are going to need to
deal with incorrect annotations in data classes as a result no matter what
so they'll deal with that regardless of how we say dataclasses should work.

-gps


>
> One challenge here is that annotations, per se, aren't only for typing. Bu
> tit would be nice if a type checker could see whatever "non-type" is
> recommended for dataclasses as "type not specified". Does an ellipses spell
> that? or None? or anything that doesn't have to be imported from typing :-)
>
> As for problems with order, if we were to accept
>>
>> @dataclass
>> class Spam:
>> beans = True
>> ham: bool
>>
>> style instead, would it be objectionable to require keyword arguments
>> only for dataclass __init__ methods?  That'd get rid of the need to care
>> about order.
>>
>
> wouldn't  that make the "ham: bool" legal -- i.e. no default?
>
> -CHB
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE
> <https://maps.google.com/?q=7600+Sand+Point+Way+NE&entry=gmail&source=g>
>   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Gregory P. Smith]

On Fri, Dec 22, 2017 at 11:40 AM Chris Barker  wrote:

> On Fri, Dec 22, 2017 at 8:49 AM, Brett Cannon 
>
But I think the key point I want to make is Guido chose dataclasses to
>> support using the type hints syntax specifically over how attrs does
>> things, so I don't see this thread trying to work around that going
>> anywhere at this point since I haven't seen a solid alternative be proposed
>> after all of this debating.
>>
>
> And the PEP has been approved.
>
> So the actionable things are:
>
> Writing good docs
>
> Converging on a "recommended" way to do non-typed dataclass fields.
>

My preference for this is "just use Any" for anyone not concerned about the
type.  But if we wanted to make it more opaque so that people need not
realizing that they are actually type annotations, I suggest adding an
alias for Any in the dataclasses module (dataclasses.Data = typing.Any)

from dataclasses import dataclass, Data

@dataclass
class Swallow:
weight_in_oz: Data = 5
laden: Data = False
species: Data = SwallowSpecies.AFRICAN

the word "Data" is friendlier than "Any" in this context for people who
don't need to care about the typing module.

We could go further and have Data not be an alias for Any if desired (so
that its repr wouldn't be confusing, not that anyone should be looking at
its repr ever).

-gps


>
> And that should be decided in order to write the docs, (and probably
> should be in the PEP).
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pep-0557 dataclasses top level module vs part of collections?

[Gregory P. Smith]

On Thu, Dec 21, 2017 at 10:47 PM Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

>
>
> > On Dec 21, 2017, at 3:21 PM, Gregory P. Smith  wrote:
> >
> > It seems a suggested use is "from dataclasses import dataclass"
> >
> > But people are already familiar with "from collections import
> namedtuple" which suggests to me that "from collections import dataclass"
> would be a more natural sounding API addition.
>
> This might make sense if it were a single self contained function.  But
> dataclasses are their own little ecosystem that warrants its own module
> namespace:
>
> >>> import dataclasses
> >>> dataclasses.__all__
> ['dataclass', 'field', 'FrozenInstanceError', 'InitVar', 'fields',
> 'asdict', 'astuple', 'make_dataclass', 'replace']
>
> Also, remember that dataclasses have a dual role as a data holder (which
> is collection-like) and as a generator of boilerplate code (which is more
> like functools.total_ordering).
>
> I support Eric's decision to make this a separate module.
>

sounds good.  lets leave it that way.  dataclasses it is.

if we were further along in figuring out how to remove the distinction
between a class and a module as a namespace I'd suggest the module name
itself be dataclass with a __call__ method so that the module could be the
decorator so we could avoid the antipattern of importing a name from a
module into your local namespace.   but we're not, so we can't. :)

-gps

>
>
> Raymond
>
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Possible bug in base64.decode: linebreaks are not ignored

[Gregory P. Smith]

I opened https://bugs.python.org/issue32491 so this wouldn't be lost under
the assumption that you'll eventually resolve your bugtracker account
situation. :)

I don't know what the correct behavior *should* be but agree that it seems
odd for decode to behave different than decodebytes.

-gps

On Wed, Jan 3, 2018 at 8:00 AM Oleg Sivokon  wrote:

> Hello,
>
> I've tried reading various RFCs around Base64 encoding, but I couldn't
> make the ends meet.  Yet there is an inconsistency between
> base64.decodebytes() and base64.decode() in that how they handle linebreaks
> that were used to collate the encoded text.  Below is an example of what
> I'm talking about:
>
> >>> import base64
> >>> foo = base64.encodebytes(b'123456789')
> >>> foo
> b'MTIzNDU2Nzg5\n'
> >>> foo = b'MTIzND\n' + b'U2Nzg5\n'
> >>> foo
> b'MTIzND\nU2Nzg5\n'
> >>> base64.decodebytes(foo)
> b'123456789'
> >>> from io import BytesIO
> >>> bytes_in = BytesIO(foo)
> >>> bytes_out = BytesIO()
> >>> bytes_in.seek(0)
> 0
> >>> base64.decode(bytes_in, bytes_out)
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "/somewhere/lib/python3.6/base64.py", line 512, in decode
> s = binascii.a2b_base64(line)
> binascii.Error: Incorrect padding
> >>> bytes_in = BytesIO(base64.encodebytes(b'123456789'))
> >>> bytes_in.seek(0)
> 0
> >>> base64.decode(bytes_in, bytes_out)
> >>> bytes_out.getvalue()
> b'123456789'
>
> Obviously, I'd expect encodebytes() and encode both to either accept or to
> reject the same input.
>
> Thanks.
>
> Oleg
>
> PS. I couldn't register to the bug-tracker (never received an email
> confirmation, not even in a spam folder), this is why I'm sending it here.
> This communication and all information contained in or attached to it is
> confidential, intended solely for the addressee, may be legally privileged
> and is the intellectual property of one of the companies of NEX Group plc
> ("NEX") or third parties. If you are not the intended addressee or receive
> this message in error, please immediately delete all copies of it and
> notify the sender. We have taken precautions to minimise the risk of
> transmitting software viruses, but we advise you to carry out your own
> virus checks on any attachments. We do not accept liability for any loss or
> damage caused by software viruses. NEX reserves the right to monitor all
> communications. We do not accept any legal responsibility for the content
> of communications, and no communication shall be considered legally
> binding. Furthermore, if the content of this communication is personal or
> unconnected with our business, we accept no liability or responsibility for
> it. NEX Group plc is a public limited company regi
>  stered in England and Wales under number 10013770 and certain of its
> affiliates are authorised and regulated by regulatory authorities. For
> further regulatory information please see www.NEX.com.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess not escaping "^" on Windows

[Gregory P. Smith]

the best way to improve shell escaping on windows is to send a PR against
the list2cmdline code that escapes everything you believe it should when
running on windows. With hyperlinks to the relevant msdn info about what
might need escaping.

On Sun, Jan 7, 2018 at 11:38 AM Christian Tismer 
wrote:

> Ok, I thought only about Windows where people often use shell=True.
> I did not see that as a Linux problem, too.
>
> Not meant as a proposal, just loud thinking... :-)
>
> But as said, the incomplete escaping is a complete mess.
>
> Ciao -- Chris
>
> On 07.01.18 19:54, Christian Tismer wrote:
> > By "normal user expectations" I meant the behavior when the builtin
> commands
> > were normal programs.
> >
> > Using "shell=True" is everywhere recommended to avoid, and I believe
> > we could avoid it by giving them replacements for build-ins.
> >
> > But I don't care if the shell escaping is correct. And that is not
> > trivial, either.
> >
> > On 07.01.18 18:22, Guido van Rossum wrote:
> >> On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer  >> > wrote:
> >>
> >> As a side note: In most cases where shell=True is found, people
> >> seem to need evaluation of the PATH variable. To my understanding,
> >>
> >> >>> from subprocess import call
> >> >>> call(("ls",))
> >>
> >> works in Linux, but (with dir) not in Windows. But that is
> misleading
> >> because "dir" is a builtin command but "ls" is not. The same holds
> for
> >> "del" (Windows) and "rm" (Linux).
> >>
> >> So I thought that using shell=True was a good Thing on windows,
> >> but actually it is the start of all evil.
> >> Using regular commands like "git" works fine on Windows and Linux
> >> without the shell=True parameter.
> >>
> >> Perhaps it would be a good thing to emulate the builtin programs
> >> in python by some shell=True replacement (emulate_shell=True?)
> >> to match the normal user expectations without using the shell?
> >>
> >>
> >> That feels like a terrible idea to me. How do you define "normal user
> >> expectations" here? If people want shell builtins they should just use
> >> shell=True. (Also note IIUC there are several quite different shells
> >> commonly used on Windows, e.g. PowerShell.)
> >>
> >> --
> >> --Guido van Rossum (python.org/~guido )
> >
> >
>
>
> --
> Christian Tismer-Sperling:^)   tis...@stackless.com
> Software Consulting  : http://www.stackless.com/
> Karl-Liebknecht-Str. 121 : https://github.com/PySide
> 14482 Potsdam: GPG key -> 0xFB7BEE0E
> phone +49 173 24 18 776 <+49%20173%202418776>  fax +49 (30) 700143-0023
> <+49%2030%207001430023>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best Python API for exposing posix_spawn

[Gregory P. Smith]

On Mon, Jan 8, 2018 at 12:36 PM Serhiy Storchaka 
wrote:

> 08.01.18 11:11, Pablo Galindo Salgado пише:
> > Following Gregory's comment on the PR I understand that he is proposing
> > to have three objects in the os module representing each action and pass
> > a sequence of these objects to the Python API. What I am not sure about
> > this is that there is no previous example of such classes in the os
> > module for other similar APIs and therefore I am not sure if there is a
> > better approach.
>
> I would pass a sequence like:
>
> [(os.close, 0),
>   (os.open, 1, '/tmp/mylog', os.O_WRONLY, 0o700),
>   (os.dup2, 1, 2),
> ]
>

i agree with just a list of tuples, but i suggest creating namedtuple
instances in the posix module for the purpose (one each for close, dup2,
open) .  Don't put a reference to a function in the tuple as Serhiy
suggested as, while obvious what it means, it gives the wrong impression to
the user: nothing is calling the Python functions.  This is a posix API
that takes a list of arguments for a specific set of system calls for _it_
to make for us in a specific order.

-gps


>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best Python API for exposing posix_spawn

[Gregory P. Smith]

On Mon, Jan 8, 2018 at 4:03 PM Random832  wrote:

> On Mon, Jan 8, 2018, at 18:05, Gregory P. Smith wrote:
> > i agree with just a list of tuples, but i suggest creating namedtuple
> > instances in the posix module for the purpose (one each for close, dup2,
> > open) .  Don't put a reference to a function in the tuple as Serhiy
> > suggested as, while obvious what it means, it gives the wrong impression
> to
> > the user: nothing is calling the Python functions.  This is a posix API
> > that takes a list of arguments for a specific set of system calls for
> _it_
> > to make for us in a specific order.
>
> Instead of a sequence of functions to call, it'd be nice if a higher-level
> API could allow just passing in a mapping of file descriptor numbers to
> what they should point to in the new process, and the implementation
> figures out what sequence is necessary to get that result.
>
> And at that point we could just extend the subprocess API to allow
> redirection of file descriptors other than 0/1/2, and have an
> implementation of it in terms of posix_spawn.
>

sure, but high level APIs don't belong in the os/posix module.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3

[Gregory P. Smith]

On Sat, Jan 13, 2018 at 4:34 PM Steven D'Aprano  wrote:

> On Sat, Jan 13, 2018 at 02:23:19PM +0100, Antoine Pitrou wrote:
> > On Sat, 13 Jan 2018 13:54:33 +0100
> > Christian Heimes  wrote:
> > >
> > > If we agree to drop support for OpenSSL 0.9.8 and 1.0.1, then I can
> land
> > > bunch of useful goodies like proper hostname verification [2], proper
> > > fix for IP address in SNI TLS header [3], PEP 543 compatible
> Certificate
> > > and PrivateKey types (support loading certs and keys from file and
> > > memory) [4], and simplified cipher suite configuration [5]. I can
> > > finally clean up _ssl.c during the beta phase, too.
> >
> > Given the annoyance of supporting old OpenSSL versions, I'd say +1 to
> > this.
> >
> > We'll have to deal with the complaints of users of Debian oldstable,
> > CentOS 6 and RHEL 6, though.
>
> It will probably be more work for Christian, but is it reasonable to
> keep support for the older versions of OpenSSL, but make the useful
> goodies conditional on a newer version?
>

I don't think it is worth spending our limited engineering time supporting
an unsupported library version.  Leave that burden to stale distro
maintainers who continue to choose dangerously stale software versions if
they ironically want to use something as modern as 3.7 on top of an ancient
set of libraries.

+1 from me for requiring OpenSSL >= 1.0.2 in Python 3.7.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7, long double vs allocator alignment, GCC 8 on x86-64

[Gregory P. Smith]

The proper fix for this in the code would likely break ABI compatibility
(ie: not possible in python 2.7 or any other stable release).

Clang's UBSAN (undefined behavior sanitizer) has been flagging this one for
a long time.

In Python 3 a double is used instead of long double since 2012 as I did
some digging at the time:
https://github.com/python/cpython/commit/e348c8d154cf6342c79d627ebfe89dfe9de23817

-gps

On Tue, Jan 30, 2018 at 10:59 AM Florian Weimer  wrote:

> I hope this is the right list for this kind of question.  We recently
> tried to build Python 2.6 with GCC 8, and ran into this issue:
>
>   
>
> Also quoting for context:
>
> | PyInstance_NewRaw contains this code:
> |
> | inst = PyObject_GC_New(PyInstanceObject, &PyInstance_Type);
> | if (inst == NULL) {
> | Py_DECREF(dict);
> | return NULL;
> | }
> | inst->in_weakreflist = NULL;
> | Py_INCREF(klass);
> | inst->in_class = (PyClassObject *)klass;
> | inst->in_dict = dict;
> | _PyObject_GC_TRACK(inst);
> |
> | _PyObject_GC_TRACK expands to:
> |
> | #define _PyObject_GC_TRACK(o) do { \
> | PyGC_Head *g = _Py_AS_GC(o); \
> | if (g->gc.gc_refs != _PyGC_REFS_UNTRACKED) \
> | Py_FatalError("GC object already tracked"); \
> | …
> |
> | Via:
> |
> | #define _Py_AS_GC(o) ((PyGC_Head *)(o)-1)
> |
> | We get to this:
> |
> | /* GC information is stored BEFORE the object structure. */
> | typedef union _gc_head {
> | struct {
> | union _gc_head *gc_next;
> | union _gc_head *gc_prev;
> | Py_ssize_t gc_refs;
> | } gc;
> | long double dummy;  /* force worst-case alignment */
> | } PyGC_Head;
> |
> | PyGC_Head has 16-byte alignment.  The net result is that
> |
> | _PyObject_GC_TRACK(inst);
> |
> | promises to the compiler that inst is properly aligned for the
> | PyGC_Head type, but it is not: PyObject_GC_New returns a pointer which
> | is only 8-byte-aligned.
> |
> | Objects/obmalloc.c contains this:
> |
> | /*
> |  * Alignment of addresses returned to the user. 8-bytes alignment works
> |  * on most current architectures (with 32-bit or 64-bit address busses).
> |  * The alignment value is also used for grouping small requests in size
> |  * classes spaced ALIGNMENT bytes apart.
> |  *
> |  * You shouldn't change this unless you know what you are doing.
> |  */
> | #define ALIGNMENT   8   /* must be 2^N */
> | #define ALIGNMENT_SHIFT 3
> | #define ALIGNMENT_MASK  (ALIGNMENT - 1)
> |
> | So either the allocator alignment needs to be increased, or the
> | PyGC_Head alignment needs to be decreased.
>
> Is this a known issue?  As far as I can see, it has not been fixed on
> the 2.7 branch.
>
> (Store merging is a relatively new GCC feature.  Among other things,
> this means that on x86-64, for sufficiently aligned pointers, vector
> instructions are used to update multiple struct fields at once.  These
> vector instructions can trigger alignment traps, similar to what
> happens on some other architectures for scalars.)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7, long double vs allocator alignment, GCC 8 on x86-64

[Gregory P. Smith]

I'm curious if changing the obmalloc.c ALIGNMENT and ALIGNMENT_SHIFT
defines is sufficient to avoid ABI breakage.

-gps

On Tue, Jan 30, 2018 at 1:20 PM Gregory P. Smith  wrote:

> The proper fix for this in the code would likely break ABI compatibility
> (ie: not possible in python 2.7 or any other stable release).
>
> Clang's UBSAN (undefined behavior sanitizer) has been flagging this one
> for a long time.
>
> In Python 3 a double is used instead of long double since 2012 as I did
> some digging at the time:
> https://github.com/python/cpython/commit/e348c8d154cf6342c79d627ebfe89dfe9de23817
>
> -gps
>
> On Tue, Jan 30, 2018 at 10:59 AM Florian Weimer  wrote:
>
>> I hope this is the right list for this kind of question.  We recently
>> tried to build Python 2.6 with GCC 8, and ran into this issue:
>>
>>   <https://bugzilla.redhat.com/show_bug.cgi?id=1540316>
>>
>> Also quoting for context:
>>
>> | PyInstance_NewRaw contains this code:
>> |
>> | inst = PyObject_GC_New(PyInstanceObject, &PyInstance_Type);
>> | if (inst == NULL) {
>> | Py_DECREF(dict);
>> | return NULL;
>> | }
>> | inst->in_weakreflist = NULL;
>> | Py_INCREF(klass);
>> | inst->in_class = (PyClassObject *)klass;
>> | inst->in_dict = dict;
>> | _PyObject_GC_TRACK(inst);
>> |
>> | _PyObject_GC_TRACK expands to:
>> |
>> | #define _PyObject_GC_TRACK(o) do { \
>> | PyGC_Head *g = _Py_AS_GC(o); \
>> | if (g->gc.gc_refs != _PyGC_REFS_UNTRACKED) \
>> | Py_FatalError("GC object already tracked"); \
>> | …
>> |
>> | Via:
>> |
>> | #define _Py_AS_GC(o) ((PyGC_Head *)(o)-1)
>> |
>> | We get to this:
>> |
>> | /* GC information is stored BEFORE the object structure. */
>> | typedef union _gc_head {
>> | struct {
>> | union _gc_head *gc_next;
>> | union _gc_head *gc_prev;
>> | Py_ssize_t gc_refs;
>> | } gc;
>> | long double dummy;  /* force worst-case alignment */
>> | } PyGC_Head;
>> |
>> | PyGC_Head has 16-byte alignment.  The net result is that
>> |
>> | _PyObject_GC_TRACK(inst);
>> |
>> | promises to the compiler that inst is properly aligned for the
>> | PyGC_Head type, but it is not: PyObject_GC_New returns a pointer which
>> | is only 8-byte-aligned.
>> |
>> | Objects/obmalloc.c contains this:
>> |
>> | /*
>> |  * Alignment of addresses returned to the user. 8-bytes alignment works
>> |  * on most current architectures (with 32-bit or 64-bit address busses).
>> |  * The alignment value is also used for grouping small requests in size
>> |  * classes spaced ALIGNMENT bytes apart.
>> |  *
>> |  * You shouldn't change this unless you know what you are doing.
>> |  */
>> | #define ALIGNMENT   8   /* must be 2^N */
>> | #define ALIGNMENT_SHIFT 3
>> | #define ALIGNMENT_MASK  (ALIGNMENT - 1)
>> |
>> | So either the allocator alignment needs to be increased, or the
>> | PyGC_Head alignment needs to be decreased.
>>
>> Is this a known issue?  As far as I can see, it has not been fixed on
>> the 2.7 branch.
>>
>> (Store merging is a relatively new GCC feature.  Among other things,
>> this means that on x86-64, for sufficiently aligned pointers, vector
>> instructions are used to update multiple struct fields at once.  These
>> vector instructions can trigger alignment traps, similar to what
>> happens on some other architectures for scalars.)
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why is Python for Windows compiled with MSVC?

[Gregory P. Smith]

TL;DR of Steve's post - MSVC is the compiler of choice for most serious
software on Windows. So we use it to best integrate with the world. There
is no compelling reason to change that.

The free-as-in-beer MSVC community edition is finally non-sucky (their
earlier efforts were crippled, they seem to have learned the lesson)

There are other viable Windows compilers.  If we want to support those in
CPython someone needs to contribute the work to do so, ongoing maintenance,
and buildbots.  I'd love to see a Clang based Windows build (Google Chrome
is built using that).  But I have no motivating reason to do the work.  I
*believe* such a build could be made to integrate and inter-operate fully
with MSVC builds and ABIs.  We could *probably* even make cross-compilation
of extensions from Linux -> Windows work that way.

We're highly unlikely to ever stop shipping python.org Windows binaries
built with anything other than MSVC unless Microsoft takes a turn toward
the dark side again.

-gps


On Wed, Jan 31, 2018 at 3:07 PM Steve Dower  wrote:

> Because every other supported platform builds using the native tools, so
> why shouldn’t the one with the most users?
>
>
>
> I’m likely biased because I work there and I’m the main intermediary with
> python-dev, but these days Microsoft is one of the strongest supporters of
> CPython. We employ the most core developers of any private company and we
> all are allowed work time to contribute, we provide full access to our
> development tools and platforms to all core developers and some prominent
> projects, we’ve made fixes, enhancements and releases or core products such
> as the CRT, MSVC, Visual Studio, Visual Studio Code, and Azure SPECIFICALLY
> to support CPython development and users. As far as I know, ALL the Windows
> buildbots are running on Azure subscriptions that Microsoft provides
> (though managed by some awesome volunteers). You’ll see us at PyCon US
> under the biggest banner and we’ll have a booth filled with engineers and
> not recruiters. Crash reports from thousands of opted-in users come into
> our systems and have directly lead to both CPython and Windows bug fixes.
>
>
>
> Meanwhile, most of the MinGW contributions have been complaints and
> drive-by patches. We (python-dev) are not opposed to supporting a second
> compiler for Windows, and honestly I’d love for extensions built with other
> compilers to be fully compatible with our main binary release, but the
> sacrifice involved in switching is significant and there’s no apparent
> commitment from the alternative options.
>
>
>
> (Note that I’m not saying Microsoft’s support is conditional on our
> compiler being used. But our ability to contribute technically would be
> greatly reduced if we didn’t have the inside access that we do.)
>
>
>
> And as has been mentioned, MSVC was selected before the other options were
> feasible. Python is a much older tool than those others, and so uses the
> tools that were best at the time.
>
>
>
> So in my opinion at least, the reasoning for selecting MSVC was perfectly
> sound, and the reasoning for continuing with it is perfectly sound.
> Unwillingness on the part of package developers to not even test on Windows
> before releasing a wheel for it is not a compelling reason to change
> anything.
>
>
>
> Cheers,
>
> Steve
>
>
>
> Top-posted from my Windows phone
>
>
>
> *From: *Oleg Sivokon 
> *Sent: *Thursday, February 1, 2018 5:40
> *To: *python-dev@python.org
> *Subject: *[Python-Dev] Why is Python for Windows compiled with MSVC?
>
>
>
> Hello list.
>
>
>
> I'll give some background before asking my question in more detail.
>
>
>
> I've been tasked with writing some infrastructure code that needs to talk
> to Kubernetes.  (Kubernetes is a popular software for managing and
> automating virtualization / containerization of cloud services).  One of
> the requirements was that the code be written in Python 3.X.
>
>
>
> The tasks my code was supposed to perform on Kubernetes would be something
> like cluster creation from specification, deletion of all or parts of the
> cluster, providing realtime statistics of cluster usage etc.  There were
> few prototype scripts written in Bash using kubectl (official client
> written in Go).
>
>
>
> My first reaction was to look for an official client for Kubernetes
> written in Python. There is one official client for Kubernetes, with a
> single maintainer, impossible to parse documentation, containing mostly
> generated code.  It is nigh impossible to use.  Here I need to explain that
> for whatever reason Kubernetes team decided to write their HTTP API in such
> a way that the server is "dumb" and the client must be "smart" in order to
> do anything useful.  For instance, if you have a description of your
> cluster, you cannot just send this description to the server and hope that
> it will know how to create the cluster from description.  You need to make
> multiple API calls (perhaps hundreds of them) to arrange for

Re: [Python-Dev] "threading.Lock().locked()" is not documented

[Gregory P. Smith]

On Wed, Jan 31, 2018 at 4:46 PM Jesus Cea  wrote:

> https://docs.python.org/3.6/library/threading.html doesn't document
> "threading.Lock().locked()", and it is something quite useful.
>
> In fact, it is used in "threading.py" itself. For instance, lines 109,
> 985, 1289.
>
> Is there any reason to not document it?.
>

No good reason.  It is a public method and part of the public API
regardless of being documented or not.  Always feel free to make a
documentation update PR.

The method does have a docstring.
  https://github.com/python/cpython/blob/master/Modules/_threadmodule.c#L193

-gps


>
> (I didn't investigate other objects in the module).
>
> --
> Jesús Cea Avión _/_/  _/_/_/_/_/_/
> j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
> Twitter: @jcea_/_/_/_/  _/_/_/_/_/
> jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
> "Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
> "My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
> "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dataclasses and correct hashability

[Gregory P. Smith]

On Fri, Feb 2, 2018 at 10:25 PM Nick Coghlan  wrote:

>
>
> On 3 Feb. 2018 1:09 am, "Eric V. Smith"  wrote:
>
>
> The problem with dropping hash=True is: how would you write __hash__
> yourself? It seems like a bug magnet if you're adding fields to the class
> and forget to update __hash__, especially in the presence of per-field
> hash=False and eq=False settings. And you'd need to make sure it matches
> the generated __eq__ (if 2 objects are equal, they need to have the same
> hash value).
>
>
> I think anyone that does this needs to think *very* carefully about how
> they do it, and offering both "hash=True" and "frozen=True" is an
> attractive nuisance that means people will write "hash=True" when what they
> wanted was "frozen=True".
>

> In particular, having to work out how write a maintainable "__hash__" will
> encourage folks to separate out the hashed fields as a separate frozen
> subrecord or base class.
>
> If this proves to be an intolerable burden then the short hand spelling
> could be added back in 3.8, but once we ship it we're going to be stuck
> with explaining the interactions indefinitely.
>

+1 Nick put words to my chief concerns.

It is easy for an author see hash=True in existing code somewhere (cargo
culting) and assume it does what they want, or quickly glance at the the
API and see "hash=True" without actually taking the time to understand the
implications of that to see that the parameter named "frozen" is the one
they are supposed to want that _safely_ makes their dataclass properly
hashable, not the more attractive parameter named "hash" that enables
dangerous behavior.

Forcing people who need a __hash__ method to write it explicitly sounds
like a good thing to me.  I am not at all worried about someone forgetting
to add a new field to an implementation of the __hash__ method when adding
a new field, the fields and __hash__ method are all defined in the same
place in the code.

I expect someone with a common need for always having a __hash__ method
will produce a library on top of dataclasses that implements something like
our current hash=True behavior.  If that kind of thing turns out to be
widely used, we can reintroduce the feature in dataclasses in 3.8 or later,
informed by what we see practical uses actually doing.

In my practical experience, people writing Python code do not need to learn
and understand the intricacies of what it means to have a __hash__ method,
be hashable, or "frozen".  We intentionally warn people against writing
dunder methods other than __init__ in their code as they are often power
features with less obvious semantics than it may seem at first glance
making such code harder to maintain.

Even calling the parameter "hash=" and saying it adds a __hash__ method as
the PEP currently does seems to launder the danger, washing away the
"dunder smell" that adding a special considerations __hash__ method carries.

The PEP (and presumably forthcoming dataclasses module documentation) says
"This is a specialized use case and should be considered carefully" which I
agree with.  But any time we suggest that in an API, how about having the
API name make it clear that this is special and not to be done lightly?  I
guess i'm arguing against using "hash=" as the arg name in favor of
"danger_there_be_vorpal_rabbits_hash_me_maybe=" or something more usefully
similar if we're going to have it.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dataclasses and correct hashability

[Gregory P. Smith]

+1 using unsafe_hash as a name addresses my concern. It's a good signal
that there are caveats worth considering.

-gps

On Sun, Feb 4, 2018, 9:50 PM Guido van Rossum  wrote:

> Looks like this is turning into a major flamewar regardless of what I say.
> :-(
>
> I really don't want to lose the ability to add a hash function to a
> mutable dataclass by flipping a flag in the decorator. I'll explain below.
> But I am fine if this flag has a name that clearly signals it's an unsafe
> thing to do.
>
> I propose to replace the existing (as of 3.7.0b1) hash= keyword for the
> @dataclass decorator with a simpler flag named unsafe_hash=. This would be
> a simple bool (not a tri-state flag like the current hash=None|False|True).
> The default would be False, and the behavior then would be to add a hash
> function automatically only if it's safe (using the same rules as for
> hash=None currently). With unsafe_hash=True, a hash function would always
> be generated that takes all fields into account except those declared using
> field(hash=False). If there's already a `def __hash__` in the function I
> don't care what it does, maybe it should raise rather than quietly doing
> nothing or quietly overwriting it.
>
> Here's my use case.
>
> A frozen class requires a lot of discipline, since you have to compute the
> values of all fields before calling the constructor. A mutable class allows
> other initialization patterns, e.g. manually setting some fields after the
> instance has been constructed, or having a separate non-dunder init()
> method. There may be good reasons for using these patterns, e.g. the object
> may be part of a cycle (e.g. parent/child links in a tree). Or you may just
> use one of these patterns because you're a pretty casual coder. Or you're
> modeling something external.
>
> My point is that once you have one of those patterns in place, changing
> your code to avoid them may be difficult. And yet your code may treat the
> objects as essentially immutable after the initialization phase (e.g. a
> parse tree). So if you create a dataclass and start coding like that for a
> while, and much later you need to put one of these into a set or use it as
> a dict key, switching to frozen=True may not be a quick option. And writing
> a __hash__ method by hand may feel like a lot of busywork. So this is where
> [unsafe_]hash=True would come in handy.
>
> I think naming the flag unsafe_hash should take away most objections,
> since it will be clear that this is not a safe thing to do. People who
> don't understand the danger are likely to copy a worse solution from
> StackOverflow anyway. The docs can point to frozen=True and explain the
> danger.
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How is the GitHub workflow working for people?

[Gregory P. Smith]

How often do we find ourselves grumbling over .py file style in PRs on
github?  If the answer to that isn't very often, the rest of my response
below seems moot. :)

On Wed, Feb 21, 2018 at 7:30 PM Guido van Rossum  wrote:

> Where I work we have some teams using flake8 and some teams that use
> pylint, and while pylint is more thorough, it is also slower and pickier,
> and the general sense is to strongly prefer flake8.
>
> I honestly expect that running either with close-to-default flags on
> stdlib code would be a nightmare, and I wouldn't want *any* directives for
> either one to appear in stdlib code, ever.
>
> In some ideal future all code would just be reformatted before it's
> checked in -- we're very far from that, and I used to be horrified by the
> very idea, but in the Go world this is pretty much standard practice, and
> the people at work who are using it are loving it. So I'm trying to have an
> open mind about this. But there simply isn't a tool that does a good enough
> job of this.
>

I don't know the specifics of your idea of "a good enough job of this" for
an auto-formatter is (we'd need to settle on that definition in a PEP)..
but there is for my definition: yapf .

Many teams require it pre-check-in for their code at Google these days. We
also use it for auto-reformatting of surrounding lines of code during all
sorts of mass-refactorings.  IIRC Lukas said Instagram/Facebook adopted it
as standard practice as well.

Some teams go all the way to enforce a "yapf must suggest no changes to the
edited areas of .py files" pre-submit error on their projects.

As Eric (and I believe Lukas in the past) has mentioned: auto formatters
don't have to produce mythical(*) "perfect" style the way individuals might
choose - they need to be good enough to keep people from arguing about
style with one another.  That's the productivity and consistency win.

What we need now is not more opinions on which formatter or linter is best.
> We need someone to actually do some work and estimate how much code would
> be changed if we ran e.g. tabnanny.py (or something more advanced!) over
> the entire stdlib, how much code would break (even the most conservative
> formatter sometimes breaks code that wasn't expecting to be reformatted --
> e.g. we used to have tests with significant trailing whitespace), and how
> often the result would be just too ugly to look at. If you're not willing
> to work on that, please don't respond to this thread.
>

Indeed.  There are at *least* four Python style and gotcha checkers in wide
use for Python code out there today.  Prematurely picking one up front
rather than coming up for criteria of which limited set of things we would
want and could practically use on the stdlib + testsuite seems wrong.

I've added "run yapf on all of CPython's .py files" to my list of things to
explore (or encourage someone else to) some day...

-gps

(*) caveat: Given that Guido has obtained History status
 and is BDFL he
can define mythical perfect style
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How is the GitHub workflow working for people?

[Gregory P. Smith]

On Tue, Feb 20, 2018 at 6:50 PM Brett Cannon  wrote:

> It's been a year and 10 days since we moved to GitHub, so I figured now is
> as good a time as any to ask people if they are generally happy with the
> workflow and if there is a particular sticking point to please bring it up
> on the core-workflow mailing list so we can potentially address it.
>

+1 happy!  Especially with the amazing automation piled on top.

It makes it sooo much easier to deal with changes coming from people than
anything involving manual patch files and clients.  Even within github's
quite limited concept of a code review tool (from a Googler perspective).

I do feel like we need more CI resources during sprints.  But we always
need more everything resources during sprints so that is nothing new and
not related to github itself.

The move to our github workflow is a win for all Python users in the world.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Symmetry arguments for API expansion

[Gregory P. Smith]

On Mon, Mar 12, 2018 at 9:51 AM Raymond Hettinger <
raymond.hettin...@gmail.com> wrote:

> There is a feature request and patch to propagate the float.is_integer()
> API through rest of the numeric types ( https://bugs.python.org/issue26680
> ).
>
> While I don't think it is a good idea, the OP has been persistent and
> wants his patch to go forward.
>
> It may be worthwhile to discuss on this list to help resolve this
> particular request and to address the more general, recurring design
> questions. Once a feature with a marginally valid use case is added to an
> API, it is common for us to get downstream requests to propagate that API
> to other places where it makes less sense but does restore a sense of
> symmetry or consistency.  In cases where an abstract base class is
> involved, acceptance of the request is usually automatic (i.e. range() and
> tuple() objects growing index() and count() methods).  However, when our
> hand hasn't been forced, there is still an opportunity to decline.  That
> said, proponents of symmetry requests tend to feel strongly about it and
> tend to never fully accept such a request being declined (it leaves them
> with a sense that Python is disordered and unbalanced).
>
>
> Raymond
>
>
>  My thoughts on the feature request -
>
> What is the proposal?
> * Add an is_integer() method to int(), Decimal(), Fraction(), and Real().
> Modify Rational() to provide a default implementation.
>
> Starting point: Do we need this?
> * We already have a simple, traditional, portable, and readable way to
> make the test:  int(x) == x
>

Mark Dickerson left a comment on the bug pointing out that such a test is
not great as it can lead to an excessive amount of computation to create
the int from some numeric types such as Decimal when all that is desired is
something the type itself may be able to answer without that.


> * In the context of ints, the test x.is_integer() always returns True.
> This isn't very useful.
> * Aside from the OP, this behavior has never been requested in Python's 27
> year history.
>
> Does it cost us anything?
> * Yes, adding a method to the numeric tower makes it a requirement for
> every class that ever has or ever will register or inherit from the tower
> ABCs.
> * Adding methods to a core object such as int() increases the cognitive
> load for everyday users who look at dir(), call help(), or read the main
> docs.
> * It conflicts with a design goal for the decimal module to not invent new
> functionality beyond the spec unless essential for integration with the
> rest of the language.  The reasons included portability with other
> implementations and not trying to guess what the committee would have
> decided in the face of tricky questions such as whether
> Decimal('1.01').is_integer()
> should return True when the context precision is only three decimal places
> (i.e. whether context precision and rounding traps should be applied before
> the test and whether context flags should change after the test).
>
> Shouldn't everything in a concrete class also be in an ABC and all its
> subclasses?
> * In general, the answer is no.  The ABCs are intended to span only basic
> functionality.  For example, GvR intentionally omitted update() from the
> Set() ABC because the need was fulfilled by __ior__().
>
> But int() already has real, imag, numerator, and denominator, why is this
> different?
> * Those attributes are central to the functioning of the numeric tower.
> * In contrast, the is_integer() method is a peripheral and incidental
> concept.
>
> What does "API Parsimony" mean?
> * Avoidance of feature creep.
> * Preference for only one obvious way to do things.
> * Practicality (not craving things you don't really need) beats purity
> (symmetry and foolish consistency).
> * YAGNI suggests holding off in the absence of clear need.
> * Recognition that smaller APIs are generally better for users.
>
> Are there problems with symmetry/consistency arguments?
> * The need for guard rails on an overpass doesn't imply the same need on a
> underpass even though both are in the category of grade changing byways.
> * "In for a penny, in for a pound" isn't a principle of good design;
> rather, it is a slippery slope whereby the acceptance of a questionable
> feature in one place seems to compel later decisions to propagate the
> feature to other places where the cost / benefit trade-offs are less
> favorable.
>
> Should float.as_integer() have ever been added in the first place?
> * Likely, it should have been a math module function like isclose() and
> isinf() so that it would not have been type specific.
> * However, that ship has sailed; instead, the question is whether we now
> have to double down and have to dispatch other ships as well.
> * There is some question as to whether it is even a good idea to be
> testing the results of floating point calculations for exact values. It may
> be useful for testing inputs, but is likely a trap for people using it

Re: [Python-Dev] Why is pickle.DEFAULT_PROTOCOL still 3?

[Gregory P. Smith]

Given that, go ahead and change in master (3.8).

On Mon, Apr 2, 2018 at 3:13 PM Christian Heimes 
wrote:

> On 2018-04-02 22:48, Lukasz Langa wrote:
> > Pickle protocol version 4.0 was originally defined back in PEP 3154 and
> shipped as part of Python 3.4 back in 2011. Yet it's still not the default.
> There's a number of things that would run faster with it like
> multiprocessing.
> >
> > This is too late for 3.7 which is a shame but can we at least bump it
> for 3.8?
>
> It sounds like a reasonable request. Python 3.4 is out of commission by
> then. I'm sure the release manager for 3.8 is going to agree with you,
> too. :)
>
> Christian
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why is pickle.DEFAULT_PROTOCOL still 3?

[Gregory P. Smith]

On Mon, Apr 2, 2018 at 3:57 PM Lukasz Langa  wrote:

>
> > On Apr 2, 2018, at 2:13 PM, Antoine Pitrou  wrote:
> >
> > On Mon, 2 Apr 2018 13:48:46 -0700
> > Lukasz Langa  wrote:
> >> Pickle protocol version 4.0 was originally defined back in PEP 3154 and
> shipped as part of Python 3.4 back in 2011. Yet it's still not the default.
> >
> > Because we want pickles produced with the default to be readable by
> > earlier Python 3 versions.
> > (the same reason protocol 0 stayed the default throughout the Python 2
> > lifetime)
>
> Alright, so that means we can easily do this for Python 3.8, right? I
> mean, following Christian's logic, Python 3.3 is already dead, with its
> final release done in February 2016 and support dropped in September 2017
> per PEP 398.
>
> I think we need to get past thinking about "Python 2" vs. "Python 3". This
> frame of mind creates space for another mythical release of Python that
> will break all the compatibilities, something we promised not to do. A
> moving backward compatibility window that includes the last release still
> under security fixes seems like a good new framework for this.
>
> What do you think?
>

+1 - That puts the words to the reason I suggest just running with the
change for 3.8.  If we had noticed in time I'd've suggested doing it in
3.7, too late and not a huge deal so just wait for 3.8.

Never changing it in the past during 1 and 2.x led a lot of code to always
specify HIGHEST as the protocol because the default was unreasonable.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Assignment Expressions

[Gregory P. Smith]

On Wed, Apr 18, 2018 at 11:04 AM Barry Warsaw  wrote:

> On Apr 18, 2018, at 10:43, MRAB  wrote:
>
> > Some languages use '=' for assignment, others for equality, but do you
> know of a language that uses ':=' for equality' or '==' for assignment?
>
> Clearly we should take a page from the ternary operator and make the
> assignment expression operator just ugly enough that people won’t overuse
> it.  Since I can’t have ‘>>’ or ‘<>’ back, I propose ‘===‘.
>
go-ahead-count-‘em-every-time-ly y’rs,
>

8 of course.  to "match" what merge conflict markers look like. ;)

php already uses === for something, we should just use === so we
can say "it goes to eleven", ending the operator war once and for all. :P

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Gregory P. Smith]

On Wed, May 2, 2018 at 2:13 PM, Barry Warsaw  wrote:

> Thanks for bringing this topic up again.  At $day_job, this is a highly
> visible and important topic, since the majority of our command line tools
> are written in Python (of varying versions from 2.7 to 3.6).  Some of those
> tools can take upwards of 5 seconds or more just to respond to —help, which
> causes lots of pain for developers, who complain (rightly so) up the
> management chain. ;)
>
> We’ve done a fair bit of work to bring those numbers down without super
> radical workarounds.  Often there are problems not strictly related to the
> Python interpreter that contribute to this.  Python gets blamed, but it’s
> not always the interpreter’s fault.  Common issues include:
>
> * Modules that have import-time side effects, such as network access or
> expensive creation of data structures.  Python 3.7’s `-X importtime` switch
> is a really wonderful way to identify the worst offenders.  Once 3.7 is
> released, I do plan to spend some time using this to collect data
> internally so we can attack our own libraries, and perhaps put automated
> performance testing into our build stack, to identify start up time
> regressions.
>
> * pkg_resources.  When you have tons of entries on sys.path, pkg_resources
> does a lot of work at import time, and because of common patterns which
> tend to use pkg_resources namespace package support in __init__.py files,
> this just kills start up times.  Of course, pkg_resources has other uses
> too, so even in a purely Python 3 world (where your namespace packages can
> omit the __init__.py), you’ll often get clobbered as soon as you want to
> use the Basic Resource Access API.  This is also pretty common, and it’s
> the main reason why Brett and I created importlib.resources for 3.7 (with a
> standalone API-compatible library for older Pythons).  That’s one less
> reason to use pkg_resources, but it doesn’t address the __init__.py use.
> Brett and I have been talking about addressing that for 3.8.
>
> * pex - which we use as our single file zipapp tool.  Especially the
> interaction between pex and pkg_resources introduces pretty significant
> overhead.  My colleague Loren Carvalho created a tool called shiv which
> requires at least Python 3.6, avoids the use of pkg_resources, and
> implements other tricks to be much more performant than pex.   Shiv is now
> open source and you can find it on RTD and GitHub.
>
> The switch to shiv and importlib.resources can shave 25-50% off of warm
> cache start up times for zipapp style executables.
>
> Another thing we’ve done, although I’m much less sanguine about them as a
> general approach, is to move imports into functions, but we’re trying to
> only use that trick on the most critical cases.
>
> Some import time effects can’t be changed.  Decorators come to mind, and
> click is a popular library for CLIs that provides some great features, but
> decorators do prevent a lazy loading approach.
>
> > On May 1, 2018, at 20:26, Gregory Szorc  wrote:
>
> >> You might think "what's a few milliseconds matter".  But if you run
> >> hundreds of commands in a shell script it adds up.  git's speed is one
> >> of the few bright spots in its UX, and hg's comparative slowness here is
> >> a palpable disadvantage.
>
> Oh, for command line tools, milliseconds absolutely matter.
>
> > As a concrete example, I recently landed a Mercurial patch [2] that
> > stubs out zope.interface to prevent the import of 9 modules on every
> > `hg` invocation.
>
> I have a similar dastardly plan to provide a pkg_resources stub :).
>
> > Mercurial provides a `chg` program that essentially spins up a daemon
> > `hg` process running a "command server" so the `chg` program [written in
> > C - no startup overhead] can dispatch commands to an already-running
> > Python/`hg` process and avoid paying the startup overhead cost. When you
> > run Mercurial's test suite using `chg`, it completes *minutes* faster.
> > `chg` exists mainly as a workaround for slow startup overhead.
>
> A couple of our developers demoed a similar approach for one of our CLIs
> that almost everyone uses.  It’s a big application with lots of
> dependencies, so particularly vulnerable to pex and pkg_resources
> overhead.  While it was just a prototype, it was darn impressive to see
> subsequent invocations produce output almost immediately.  It’s unfortunate
> that we have to utilize all these tricks to get even moderately performant
> Python CLIs.
>
>
Note that this kind of "trick" is not unique to Python.  I see it used by
large Java tools at work.  In effect emacs has done similar things for many
decades with its saved core-dump at build time. It saves a snapshot of
initialized elisp interpreter state and loads that into memory instead of
rerunning initialization to reproduce the state.

I don't know if anyone has looked at making a similar concept of saved
post-startup interpreter state for rapid loading as a memory image possible
in Python.

Re: [Python-Dev] A fast startup patch (was: Python startup time)

[Gregory P. Smith]

+1 to the concept!

On Thu, May 3, 2018 at 1:13 PM, Carl Shapiro  wrote:

> Hello,
>
> Yesterday Neil Schemenauer mentioned some work that a colleague of mine
> (CCed) and I have done to improve CPython start-up time.  Given the recent
> discussion, it seems timely to discuss what we are doing and whether it is
> of interest to other people hacking on the CPython runtime.
>
> There are many ways to reduce the start-up time overhead.  For this
> experiment, we are specifically targeting the cost of unmarshaling heap
> objects from compiled Python bytecode.  Our measurements show this specific
> cost to represent 10% to 25% of the start-up time among the applications we
> have examined.
>
> Our approach to eliminating this overhead is to store unmarshaled objects
> into the data segment of the python executable.  We do this by processing
> the compiled python bytecode for a module, creating native object code with
> the unmarshaled objects in their in-memory representation, and linking this
> into the python executable.
>
> When a module is imported, we simply return a pointer to the top-level
> code object in the data segment directly without invoking the unmarshaling
> code or touching the file system.  What we are doing is conceptually
> similar to the existing capability to freeze a module, but we avoid
> non-trivial unmarshaling costs.
>
> The patch is still under development and there is still a little bit more
> work to do.  With that in mind, the numbers look good but please take these
> with a grain of salt
>
> Baseline
>
> $ bench "./python.exe -c ''"
> benchmarking ./python.exe -c ''
> time 31.46 ms   (31.24 ms .. 31.78 ms)
>  1.000 R²   (0.999 R² .. 1.000 R²)
> mean 32.08 ms   (31.82 ms .. 32.63 ms)
> std dev  778.1 μs   (365.6 μs .. 1.389 ms)
>
> $ bench "./python.exe -c 'import difflib'"
> benchmarking ./python.exe -c 'import difflib'
> time 32.82 ms   (32.64 ms .. 33.02 ms)
>  1.000 R²   (1.000 R² .. 1.000 R²)
> mean 33.17 ms   (33.01 ms .. 33.44 ms)
> std dev  430.7 μs   (233.8 μs .. 675.4 μs)
>
>
> With our patch
>
> $ bench "./python.exe -c ''"
> benchmarking ./python.exe -c ''
> time 24.86 ms   (24.62 ms .. 25.08 ms)
>  0.999 R²   (0.999 R² .. 1.000 R²)
> mean 25.58 ms   (25.36 ms .. 25.94 ms)
> std dev  592.8 μs   (376.2 μs .. 907.8 μs)
>
> $ bench "./python.exe -c 'import difflib'"
> benchmarking ./python.exe -c 'import difflib'
> time 25.30 ms   (25.00 ms .. 25.55 ms)
>  0.999 R²   (0.998 R² .. 1.000 R²)
> mean 26.78 ms   (26.30 ms .. 27.64 ms)
> std dev  1.413 ms   (747.5 μs .. 2.250 ms)
> variance introduced by outliers: 20% (moderately inflated)
>
>
> Here are some numbers with the patch but with the stat calls preserved to
> isolate just the marshaling effects
>
> Baseline
>
> $ bench "./python.exe -c 'import difflib'"
> benchmarking ./python.exe -c 'import difflib'
> time 34.67 ms   (33.17 ms .. 36.52 ms)
>  0.995 R²   (0.990 R² .. 1.000 R²)
> mean 35.36 ms   (34.81 ms .. 36.25 ms)
> std dev  1.450 ms   (1.045 ms .. 2.133 ms)
> variance introduced by outliers: 12% (moderately inflated)
>
>
> With our patch (and calls to stat)
>
> $ bench "./python.exe -c 'import difflib'"
> benchmarking ./python.exe -c 'import difflib'
> time 30.24 ms   (29.02 ms .. 32.66 ms)
>  0.988 R²   (0.968 R² .. 1.000 R²)
> mean 31.86 ms   (31.13 ms .. 32.75 ms)
> std dev  1.789 ms   (1.329 ms .. 2.437 ms)
> variance introduced by outliers: 17% (moderately inflated)
>
>
> (This work was done in CPython 3.6 and we are exploring back-porting to
> 2.7 so we can run the hg startup benchmarks in the performance test suite.)
>
> This is effectively a drop-in replacement for the frozen module capability
> and (so far) required only minimal changes to the runtime.  To us, it seems
> like a very nice win without compromising on compatibility or complexity.
> I am happy to discuss more of the technical details until we have a public
> patch available.
>
> I hope this provides some optimism around the possibility of improving the
> start-up time of CPython.  What do you all think?
>
> Kindly,
>
> Carl
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> greg%40krypto.org
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Gregory P. Smith]

On Thu, May 3, 2018 at 5:22 PM, Lukasz Langa  wrote:

>
> > On May 2, 2018, at 8:57 PM, INADA Naoki  wrote:
> >
> > Recently, I reported how stdlib slows down `import requests`.
> > https://github.com/requests/requests/issues/4315#issuecomment-385584974
> >
> > For Python 3.8, my ideas for faster startup time are:
> >
> > * Add lazy compiling API or flag in `re` module.  The pattern is compiled
> > when first used.
>
> How about go the other way and allow compiling at Python *compile*-time?
> That would actually make things faster instead of just moving the time
> spent around.
>
> I do see value in being less eager in Python but I think the real wins are
> hiding behind ahead-of-time compilation.
>

Agreed in concept.  We've got a lot of unused letters that could be new
string prefixes... (ugh)

I'd also like to see this concept somehow extended to decorators so that
the results of the decoration can be captured in the compiled pyc rather
than requiring execution at import time.  I realize that limits what
decorators can do, but the evil things they could do that this would
eliminate are things they just shouldn't be doing in most situations.
meaning: there would probably be two types of decorators... colons seem to
be all the rage these days so we could add an @: operator for that. :P ...
Along with a from __future__ import to change the behavior or all
decorators in a file from runtime to compile time by default.

from __future__ import compile_time_decorators  # we'd be unlikely to ever
change the default and break things, __future__ seems wrong

@this_happens_at_compile_time(3)
def ...

@:this_waits_until_runtime(5)
def ...

Just a not-so-wild idea, no idea if this should become a PEP for 3.8.  (the
: syntax is a joke - i'd prefer @@ so it looks like eyeballs)

If this were done to decorators, you can imagine extending that concept to
something similar to allow compile time re.compile calls as some form of
assignment decorator:

GREGS_RE = @re.compile(r'A regex compiled at compile time\. number = \d+')

-gps


> - Ł
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] bpo-33522: Enable CI builds on Visual Studio Team Services (GH-6865) (GH-6925)

[Gregory P. Smith]

Why did this commit modify .py files, unittests, and test.support?

That is inappropriate for something claiming to merely enable a CI platform.

-gps

On Thu, May 17, 2018 at 6:50 AM Steve Dower 
wrote:

>
> https://github.com/python/cpython/commit/0d8f83f59c8f4cc7fe125434ca4ecdcac111810f
> commit: 0d8f83f59c8f4cc7fe125434ca4ecdcac111810f
> branch: 3.6
> author: Steve Dower 
> committer: GitHub 
> date: 2018-05-17T09:46:00-04:00
> summary:
>
> bpo-33522: Enable CI builds on Visual Studio Team Services (GH-6865)
> (GH-6925)
>
> files:
> A .vsts/docs-release.yml
> A .vsts/docs.yml
> A .vsts/linux-buildbot.yml
> A .vsts/linux-coverage.yml
> A .vsts/linux-deps.yml
> A .vsts/linux-pr.yml
> A .vsts/macos-buildbot.yml
> A .vsts/macos-pr.yml
> A .vsts/windows-buildbot.yml
> A .vsts/windows-pr.yml
> A Misc/NEWS.d/next/Build/2018-05-15-12-44-50.bpo-33522.mJoNcA.rst
> A Misc/NEWS.d/next/Library/2018-05-16-17-05-48.bpo-33548.xWslmx.rst
> M Doc/make.bat
> M Lib/tempfile.py
> M Lib/test/support/__init__.py
> M Lib/test/test_asyncio/test_base_events.py
> M Lib/test/test_bdb.py
> M Lib/test/test_pathlib.py
> M Lib/test/test_poplib.py
> M Lib/test/test_selectors.py
> M PCbuild/rt.bat
> M Tools/ssl/multissltests.py
>
> diff --git a/.vsts/docs-release.yml b/.vsts/docs-release.yml
> new file mode 100644
> index ..e90428a42494
> --- /dev/null
> +++ b/.vsts/docs-release.yml
> @@ -0,0 +1,43 @@
> +# Current docs for the syntax of this file are at:
> +#
> https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted.md
> +
> +name: $(BuildDefinitionName)_$(Date:MMdd)$(Rev:.rr)
> +
> +queue:
> +  name: Hosted Linux Preview
> +
> +#variables:
> +
> +steps:
> +- checkout: self
> +  clean: true
> +  fetchDepth: 5
> +
> +- script: sudo apt-get update && sudo apt-get install -qy --force-yes
> texlive-full
> +  displayName: 'Install LaTeX'
> +
> +- task: UsePythonVersion@0
> +  displayName: 'Use Python 3.6 or later'
> +  inputs:
> +versionSpec: '>=3.6'
> +
> +- script: python -m pip install sphinx blurb python-docs-theme
> +  displayName: 'Install build dependencies'
> +
> +- script: make dist PYTHON=python SPHINXBUILD='python -m sphinx'
> BLURB='python -m blurb'
> +  workingDirectory: '$(build.sourcesDirectory)/Doc'
> +  displayName: 'Build documentation'
> +
> +- task: PublishBuildArtifacts@1
> +  displayName: 'Publish build'
> +  inputs:
> +PathToPublish: '$(build.sourcesDirectory)/Doc/build'
> +ArtifactName: build
> +publishLocation: Container
> +
> +- task: PublishBuildArtifacts@1
> +  displayName: 'Publish dist'
> +  inputs:
> +PathToPublish: '$(build.sourcesDirectory)/Doc/dist'
> +ArtifactName: dist
> +publishLocation: Container
> diff --git a/.vsts/docs.yml b/.vsts/docs.yml
> new file mode 100644
> index ..efa1e871656d
> --- /dev/null
> +++ b/.vsts/docs.yml
> @@ -0,0 +1,43 @@
> +# Current docs for the syntax of this file are at:
> +#
> https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted.md
> +
> +name: $(BuildDefinitionName)_$(Date:MMdd)$(Rev:.rr)
> +
> +queue:
> +  name: Hosted Linux Preview
> +
> +trigger:
> +  branches:
> +include:
> +- master
> +- 3.7
> +- 3.6
> +  paths:
> +include:
> +- Doc/*
> +
> +#variables:
> +
> +steps:
> +- checkout: self
> +  clean: true
> +  fetchDepth: 5
> +
> +- task: UsePythonVersion@0
> +  displayName: 'Use Python 3.6 or later'
> +  inputs:
> +versionSpec: '>=3.6'
> +
> +- script: python -m pip install sphinx~=1.6.1 blurb python-docs-theme
> +  displayName: 'Install build dependencies'
> +
> +- script: make check suspicious html PYTHON=python
> +  workingDirectory: '$(build.sourcesDirectory)/Doc'
> +  displayName: 'Build documentation'
> +
> +- task: PublishBuildArtifacts@1
> +  displayName: 'Publish build'
> +  inputs:
> +PathToPublish: '$(build.sourcesDirectory)/Doc/build'
> +ArtifactName: build
> +publishLocation: Container
> diff --git a/.vsts/linux-buildbot.yml b/.vsts/linux-buildbot.yml
> new file mode 100644
> index ..d75d7f57650e
> --- /dev/null
> +++ b/.vsts/linux-buildbot.yml
> @@ -0,0 +1,71 @@
> +# Current docs for the syntax of this file are at:
> +#
> https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted.md
> +
> +name: $(BuildDefinitionName)_$(Date:MMdd)$(Rev:.rr)
> +
> +queue:
> +  name: Hosted Linux Preview
> +
> +trigger:
> +  branches:
> +include:
> +- master
> +- 3.7
> +- 3.6
> +  paths:
> +exclude:
> +- Doc/*
> +- Tools/*
> +
> +variables:
> +  # Copy-pasted from linux-deps.yml until template support arrives
> +  OPENSSL: 1.1.0g
> +  OPENSSL_DIR: "$(build.sourcesDirectory)/multissl/openssl/$(OPENSSL)"
> +
> +
> +steps:
> +- checkout: self
> +  clean: true
> +  fetchDepth: 5
> +
> +#- template: linux-deps.yml
> +
> +# See
> https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted-templates.md
> +# For now, we copy/paste the s

[Python-Dev] please help triage VSTS failures

[Gregory P. Smith]

These both look like VSTS infrastructure falling over on PRs:

 https://python.visualstudio.com/cpython/_build?buildId=522

 https://python.visualstudio.com/cpython/_build?buildId=523

I don't see anywhere that gives information about the failures. (*)

These CI failures on different platforms are both for the same change, on
different branches, for a documentation only change.  That passed on the
other branches and platforms for the same change.

-gps

(*) I refuse to "Download logs as a zip file".  I'm in a web browser, if
the information I might need is potentially buried somewhere in a zip file
of logs, that is a waste of my time. I'm not going to do it. The web UI
*needs* to find and display the relevant failure info from any logs
directly.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] please help triage VSTS failures

[Gregory P. Smith]

Hah, yes, that was un-intuitive for me.  I was looking for something
labelled "logs" to click on.  thanks.  (I hope MS is taking notes on UX
issues here)

So these failures were in fact the known flakes and not infrastructure.
good!

On the high level view of VSTS output on a failure:

"Issues:
phase1
Cmd.exe exited with code '2'."

is not useful.  Can we make that display the tail of the failing phase's
log?  that'd avoid needing to find the inner most red X and click on it
which is what I'll always need to do otherwise as there doesn't appear to
be a direct hyperlink to that.

It looks like VSTS also had an API for surfacing individual test results
into their Tests / Test Results summary pane?  Doing something to integrate
with that would likely be a nicer UI.  We have something similar at work,
here's how we make unittest.TestCase emit the JUnit XML files of test
results for CI systems to display:

https://github.com/abseil/abseil-py/blob/master/absl/testing/xml_reporter.py


If there are industry standard format(s) that make Python test suites play
nicer for reporting on CI systems such as that, they'd be worthy of stdlib
inclusion.

-gps


On Fri, May 18, 2018 at 5:35 PM Nathaniel Smith  wrote:

> On Fri, May 18, 2018 at 1:13 PM, Steve Dower 
> wrote:
> > According to the VSTS dev team, an easy “rerun this build” button and
> > filtering by changed paths are coming soon, which should clean things up.
>
> If you're talking to them, please ask them to make sure that the
> "rerun this build" button doesn't erase the old log. (That's what it
> does on Travis. Appveyor is better.) The problem is that when you have
> a flaky/intermittent failure, your todo list is always (a) rerun the
> build so at least it's not blocking whatever this unrelated change is,
> (b) file some sort of bug, or comment on some existing bug, and link
> to the log to help track down the intermittent failure. If you click
> the "rebuild" button on Travis, then it solves (a), while deleting the
> information you need for (b) – and for rare intermittent bugs you
> might not have much information to go on besides that build log.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Gregory P. Smith]

Why now?  We intentionally decided not to do this for 2.7 in the past
because it was too late for the release cutoff.

Has anyone benchmarked compiling python in profile-opt mode vs having the
computed goto patch?  I'd *expect* the benefits to be the roughly the same.
Has this been compared to that?  (Anyone not compiling their Python
interpreter in profile-opt mode is doing themselves a major disservice.)

Does it shows noteworthy improvements even when used with a profile-opt
build using a PROFILE_TASK of regrtest.py with the same test exclusion list
as the debian python2.7 package?

Could you please rerun benchmarks including the profile-opt build with and
without the patch for comparsion.

-gps

PS I recommend attaching the up to date patch against 2.7.10 to issue4753.
That is where anyone will go looking for it, not buried in a mailing list
archive.

On Wed, May 27, 2015 at 10:01 PM Terry Reedy  wrote:

> On 5/27/2015 9:31 PM, Nick Coghlan wrote:
>
> > +1 from me, for basically the same reasons Guido gives: Python 2.7 is
> > going to be with us for a long time, and this particular change
> > shouldn't have any externally visible impacts at either an ABI or API
> level.
>
> Immediately after a release, giving the patch plenty of time to be be
> tested, seems like a good time.
>
> --
> Terry Jan Reedy
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Gregory P. Smith]

On Thu, May 28, 2015 at 9:08 AM Guido van Rossum  wrote:

> Wow. Such thread. :-)
>
> This patch could save companies like Dropbox a lot of money. We run a ton
> of Python code in large datacenters, and while we are slow in moving to
> Python 3, we're good at updating to the latest 2.7.
>

Dropbox should be compiling its own interpreter with whatever patches it
deems appropriate. The people it'll save resources for are companies not
enlightened enough to do that: thousands of them, generally small or
non-tech focused :)

The patch is forward and backward compatible.I'm strongly in favor.
>

+1 I'm in favor as well.  I mostly wanted to make sure that people were
aware of profile-opt builds and that it was being compared.  Sounds like
both benefit, even used together.  Win win.

This is a 100% API compatible change.  It just rearranges the interpreter
loop on compilers enlightened enough to allow it.  I was always bummed that
it didn't make it into 2.7 itself.  But given the world+dog is going to
have 2.7 around and kicking for a long time, lets save the world some CPU
cycles (read: carbon) for little effort.  Very practical.  Good for the
world.

People who need to save orders of magnitude more cycles shouldn't use an
interpreter. ie: PyPy. Or consider the costs of moving to a compiled
language.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Gregory P. Smith]

On Thu, May 28, 2015 at 6:10 PM Larry Hastings  wrote:

> On 05/28/2015 05:58 PM, Victor Stinner wrote:
>
> Why not continue to enhance Python 3 instead of wasting our time with
> Python 2? We have limited resources in term of developers to maintain
> Python.
>
>
> Uh, guys, you might as well call off the debate.  Benjamin already checked
> it in.
>
> https://hg.python.org/cpython/rev/17d3bbde60d2
>
>
Thanks Benjamin! :)


>
>
> */arry*
>  ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 is here until 2020, please don't call it a waste.

[Gregory P. Smith]

On Fri, May 29, 2015 at 12:24 AM Nick Coghlan  wrote:

>
> On 29 May 2015 11:01 am, "Victor Stinner" 
> wrote:
> >
> > Why not continue to enhance Python 3 instead of wasting our time with
> > Python 2? We have limited resources in term of developers to maintain
> > Python.
> >
> > (I'm not talking about fixing *bugs* in Python 2 which is fine with me.)
>
> I'm actually OK with volunteers deciding that even fixing bugs in 2.7
> isn't inherently rewarding enough for them to be willing to do it for free
> on their own time.
>

That is 100% okay.

What is not okay is for python-dev representatives to respond to users (in
any list/forum/channel) reporting bugs in 2.7 or asking if a fix in 3 can
be backported to 2.7 with things akin to "just use Python 3" or "sorry, 2.7
is critical fixes only. move to python 3 already." This is actively driving
our largest users away.  I bring this up because a user was bemoaning how
useless they feel python core devs are because of this attitude recently.
Leading to feelings of wishing to just abandon CPython if not Python all
together.

I'm sure I have even made some of those responses myself (sorry!). My point
here is: know it. recognize it. don't do it anymore. It harms the community.

A correct and accurate response to desires to make non-api-breaking changes
in 2.7 is "Patches that do not change any APIs for 2.7 are welcome in the
issue tracker." possibly including "I don't have the bandwidth to review
2.7 changes, find someone on python-dev to review and champion this for you
if you need it."  Finding someone may not always be easy. But at least is
still the "patches welcome" attitude and suggests that the work can be done
if someone is willing to do it. Lets make a concerted effort to not be
hostile and against it by default.

Ex: Is someone with a python application that is a million of lines
supposed to have everyone involved in that drop the productive work they
are doing and spend that porting their existing application to python 3
because we have so far failed to provide the tools to make that migration
easy?  No.  Empathize with our community.  Feel their pain.  (and everyone
who is working on tools to aid the transition: keep doing that! Our users
are gonna need it unless we don't want them as users anymore.)

We committed to supporting 2.7 until 2020 in 2014 per
https://hg.python.org/peps/rev/76d43e52d978.  That means backports of
important bug or performance fixes should at least be allowed on the table,
even if hairy, even if you won't work on them yourselves on a volunteer
basis. This is the first long term support release of Python ever. This is
what LTS means.  LTS could *also* stand for Learn To Support...

-gps

> Stepping up to extrinsically reward activities that are beneficial for
> customers but aren't intrinsically interesting enough for people to be
> willing to do for free is one of the key reasons commercial open source
> redistributors get paid.
>
> That more explicitly commercial presence is a dynamic we haven't
> historically had to deal with in core development, so there are going to be
> some growing pains as we find an arrangement that everyone is comfortable
> with (or is at least willing to tolerate, but I'm optimistic we can do
> better than that).
>
> Cheers,
> Nick.
>
> >
> > --
> >
> > By the way, I just wrote sixer, a new tool to generate patches to port
> > OpenStack to Python 3 :-)
> > https://pypi.python.org/pypi/sixer
> >
> > It's based on regex, so it's less reliable than 2to3, 2to6 or
> > modernize, but it's just enough for my specific use case. On
> > OpenStack, it's not possible to send one giant patch "hello, this is
> > python 3". Code is modified by small and incremental changes.
> >
> > Come on in the Python 3 world and... always look on the bright side of
> > life ( https://www.youtube.com/watch?v=VOAtCOsNuVM )!
> >
> > Victor
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
>
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>  ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tracker reviews look like spam

[Gregory P. Smith]

On Tue, May 12, 2015 at 3:09 PM Terry Reedy  wrote:

> Gmail dumps patch review email in my junk box. The problem seems to be
> the spoofed From: header.
>
> Received: from psf.upfronthosting.co.za ([2a01:4f8:131:2480::3])
>  by mx.google.com with ESMTP id
> m1si26039166wjy.52.2015.05.12.00.20.38
>  for ;
>  Tue, 12 May 2015 00:20:38 -0700 (PDT)
> Received-SPF: softfail (google.com: domain of transitioning
> storch...@gmail.com does not designate 2a01:4f8:131:2480::3 as permitted
> sender) client-ip=2a01:4f8:131:2480::3;
>
> Tracker reviews are the only false positives in my junk list. Otherwise,
> I might stop reviewing. Verizon does not even deliver mail that fails
> its junk test, so I would not be surprised if there are people who
> simply do not get emailed reviews.
>
> Tracker posts are sent from Person Name 
> Perhaps reviews could come 'from' Person Name 
>
> Even direct tracker posts just get a neutral score.
> Received-SPF: neutral (google.com: 2a01:4f8:131:2480::3 is neither
> permitted nor denied by best guess record for domain of
> roundup-ad...@psf.upfronthosting.co.za) client-ip=2a01:4f8:131:2480::3;
>
> SPF is Sender Policy Framework
> https://en.wikipedia.org/wiki/Sender_Policy_Framework
>
> Checkins mail, for instance, gets an SPF 'pass' because python.org
> designates mail.python.org as a permitted sender.
>
> --
> Terry Jan Reedy
>

FWIW while the email signing issue might be interesting to fix...  I really
wish code reviews weren't emailed separately, I'd like to see them simply
appended to the related issue as a block comment on the issue. Then all
email notification would be handled by the bug tracker itself (which
happens to already work).  It keeps related conversation in a single place
which will continue to persist even if we switch review tools in the future.

I *believe* you can get this to happen today in a review if you add the
rep...@bugs.python.org email address to the code review as the issueX
in the subject line will make the tracker turn it into a bug comment.  If
so, having that be the default cc for all reviews created would be a great
feature (and modify it not to send mail to anyone else).

My 2 cents.  (literally; as I'm only suggesting, not implementing)

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's New editing

[Gregory P. Smith]

On Mon, Jul 6, 2015 at 8:38 AM David Mertz  wrote:

> Hi Folks,
>
> I hereby volunteer to write "What's New for Python 3.5?" if folks on
> python-dev are fine with me taking the job (i.e. I ran it by Travis, my
> boss at Continuum, and he's happy to allow me to do that work within my
> salaried hours... so having time isn't a problem).
>
> If this is OK with the powers-that-be, I'll coordinate with David Murray
> on how best to take over this task from him.
>

+1


>
> Thanks, David...
>
> On Sun, Jul 5, 2015 at 8:51 PM, Nick Coghlan  wrote:
>
>> On 6 July 2015 at 12:42, David Mertz  wrote:
>> > I think I might be able to "volunteer" for the task of writing/editing
>> the
>> > "What's New in 3.5" docs.  I saw David's comment on it today, so
>> obviously
>> > haven't yet had a chance to run it by my employer (Continuum
>> Analytics), but
>> > I have a hunch they would allow me to do it at least in large part as
>> paid
>> > time.  I am experienced as a technical writer, follow python-dev, write
>> > about new features, but am *not*, however, my self an existing core
>> > developer.
>>
>> I think the last point may be a positive rather than a negative when
>> it comes to effectively describing new features :)
>>
>> > If there is interest in this, or at least it seems plausible, I can run
>> it
>> > by my employer tomorrow to see about getting enough time allocated
>> (using
>> > David Murray's past experience as a guideline for what's likely to be
>> > needed).
>>
>> That would be very helpful! I'd definitely be able to find the time to
>> review and merge updates, it's the research-and-writing side that
>> poses a problem for me (appreciating a task is worth doing isn't the
>> same thing as wanting to do it myself!).
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>>
>
>
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>  ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP-498 & PEP-501: Literal String Formatting/Interpolation

[Gregory P. Smith]

On Sun, Aug 9, 2015 at 3:25 PM Brett Cannon  wrote:

>
> On Sun, Aug 9, 2015, 13:51 Peter Ludemann via Python-Dev <
> python-dev@python.org> wrote:
>
> Most of my outputs are log messages, so this proposal won't help me
> because (I presume) it does eager evaluation of the format string and the
> logging methods are designed to do lazy evaluation. Python doesn't have
> anything like Lisp's "special forms", so there doesn't seem to be a way to
> implicitly put a lambda on the string to delay evaluation.
>
> It would be nice to be able to mark the formatting as lazy ... maybe
> another string prefix character to indicate that? (And would the 2nd
> expression in an assert statement be lazy or eager?)
>
>
> That would require a lazy string type which is beyond the scope of this
> PEP as proposed since it would require its own design choices, how much
> code would not like the different type, etc.
>
> -Brett
>

Agreed that doesn't belong in PEP 498 or 501 itself... But it is a real
need.

We left logging behind when we added str.format() and adding yet another
_third_ way to do string formatting without addressing the needs of
deferred-formatting for things like logging is annoying.

brainstorm: Imagine a deferred interpolation string with a d'' prefix..
 di'foo ${bar}' would be a new type with a __str__ method that also retains
a runtime reference to the necessary values from the scope within which it
was created that will be used for substitutions when iff/when it is
__str__()ed.  I still wouldn't enjoy reminding people to use di''
inlogging.info(di'thing happened: ${result}') all the time any more than I
like reminding people to undo their use of % and just pass the values as
additional args to the logging call... But I think people would find it
friendlier and thus be more likely to get it right on their own.  logging's
manually deferred % is an idiom i'd like to see wither away.

There's also a performance aspect to any new formatter, % is oddly pretty
fast, str.format isn't. So long as you can do stuff at compile time rather
than runtime I think these PEPs could be even faster. Constant string
pep-498 or pep-501 formatting could be broken down at compile time and
composed into the optimal set of operations to build the resulting string /
call the formatter.

So far looking over both peps, I lean towards pep-501 rather than 498:

I really prefer the ${} syntax.
I don't like arbitrary logical expressions within strings.
I dislike str only things without a similar concept for bytes.

but neither quite suits me yet.

501's __interpolate*__ builtins are good and bad at the same time.  doing
this at the module level does seem right, i like the i18n use aspect of
that, but you could also imagine these being methods so that subclasses
could override the behavior on a per-type basis.  but that probably only
makes sense if a deferred type is created due to when and how interpolates
would be called.  also, adding builtins, even __ones__ annoys me for some
reason I can't quite put my finger on.

(jumping into the threads way late)
-gps

>
> PS: As to Brett's comment about the history of string interpolation ... my
> recollection/understanding is that it started with Unix shells and the
> "$variable" notation, with the "$variable" being evaluated within "..." and
> not within '...'. Perl, PHP, Make (and others) picked this up. There seems
> to be a trend to avoid the bare "$variable" form and instead use
> "${variable}" everywhere, mainly because "${...}" is sometimes required to
> avoid ambiguities (e.g. "There were $NUMBER ${THING}s.")
>
> PPS: For anyone wishing to improve the existing format options, Common
> Lisp's FORMAT 
> and Prolog's format/2
> 
> have some capabilities that I miss from time to time in Python.
>
> On 9 August 2015 at 11:22, Eric V. Smith  wrote:
>
> On 8/9/2015 1:38 PM, Brett Cannon wrote:
> >
> >
> > On Sun, 9 Aug 2015 at 01:07 Stefan Behnel 
> > > wrote:
> >
> > Eric V. Smith schrieb am 08.08.2015 um 03:39:
> > > Following a long discussion on python-ideas, I've posted my draft
> of
> > > PEP-498. It describes the "f-string" approach that was the subject
> of
> > > the "Briefer string format" thread. I'm open to a better title than
> > > "Literal String Formatting".
> > >
> > > I need to add some text to the discussion section, but I think
> it's in
> > > reasonable shape. I have a fully working implementation that I'll
> get
> > > around to posting somewhere this weekend.
> > >
> > > >>> def how_awesome(): return 'very'
> > > ...
> > > >>> f'f-strings are {how_awesome()} awesome!'
> > > 'f-strings are very awesome!'
> > >
> > > I'm open to any suggestions to improve the PEP. Thanks for your
> > feedback.
> >
> > [copying my comment from python-ideas here]
> >
> >  

Re: [Python-Dev] Profile Guided Optimization active by-default

[Gregory P. Smith]

On Sat, Aug 22, 2015 at 9:27 AM Brett Cannon  wrote:

> On Sat, Aug 22, 2015, 09:17 Guido van Rossum  wrote:
>
> How about we first add a new Makefile target that enables PGO, without
> turning it on by default? Then later we can enable it by default.
>
>
There already is one and has been for many years.  make profile-opt.

I even setup a buildbot for it last year.

The problem with the existing profile-opt build in our default Makefile.in
is that is uses a horrible profiling workload (pybench, ugh) so it leaves a
lot of improvements behind.

What all Linux distros (Debian/Ubuntu and Redhat at least; nothing else
matters) do for their Python builds is to use profile-opt but they replace
the profiling workload with a stable set of the Python unittest suite
itself. Results are much better all around.  Generally a 20% speedup.

Anyone deploying Python who is *not* using a profile-opt build is wasting
CPU resources.

Whether it should be *the default* or not *is a different question*.  The
Makefile is optimized for CPython developers who certainly do not want to
run two separate builds and a profile-opt workload every time they type
make to test out their changes.

But all binary release builds should use it.

I agree. Updating the Makefile so it's easier to use PGO is great, but we
> should do a release with it as opt-in and go from there.
>
> Also, I have my doubts about regrtest. How sure are we that it represents
> a typical Python load? Tests are often using a different mix of operations
> than production code.
>
> That was also my question. You said that "it provides the best performance
> improvement", but compared to what; what else was tried? And what
> difference does it make to e.g. a Django app that is trained on their own
> simulated workload compared to using regrtest? IOW is regrtest displaying
> the best across-the-board performance because it stresses the largest swath
> of Python and thus catches generic patterns in the code but individuals
> could get better performance with a simulated workload?
>

This isn't something to argue about.  Just use regrtest and compare the
before and after with the benchmark suite.  It really does exercise things
well.  People like to fear that it'll produce code optimized for the test
suite itself or something.  No.  Python as an interpreter is very
realistically exercised by running it as it is simply running a lot of code
and a good variety of code including the extension modules that benefit
most such as regexes, pickle, json, xml, etc.

Thomas tried the test suite and a variety of other workloads when looking
at what to use at work.  The testsuite works out generally the best.  Going
beyond that seems to be a wash.

What we tested and decided to use on our own builds after benchmarking at
work was to build with:

make profile-opt PROFILE_TASK="-m test.regrtest -w -uall,-audio -x test_gdb
test_multiprocessing"

In general if a test is unreliable or takes an extremely long time, exclude
it for your sanity.  (i'd also kick out test_subprocess on 2.7; we replaced
subprocess with subprocess32 in our build so that wasn't an issue)

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Profile Guided Optimization active by-default

[Gregory P. Smith]

PGO is unrelated to the particular CPU the profiling is done on. (It is
conceivable that it'd make a small difference but I've never observed that
in practice)

On Tue, Aug 25, 2015, 9:28 AM Xavier Combelle 
wrote:

Pardon me if I'm not in the right place to ask the following naive
question. (say me if it's the case)

Does Profile Guided Optimization performance improvements are specific to
the chip where the built is done or the performance is better on a larger
set of chips?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Profile Guided Optimization active by-default

[Gregory P. Smith]

On Mon, Aug 24, 2015, 11:19 PM Nick Coghlan  wrote:

On 25 August 2015 at 05:52, Gregory P. Smith  wrote:
> What we tested and decided to use on our own builds after benchmarking at
> work was to build with:
>
> make profile-opt PROFILE_TASK="-m test.regrtest -w -uall,-audio -x
test_gdb
> test_multiprocessing"
>
> In general if a test is unreliable or takes an extremely long time,
exclude
> it for your sanity.  (i'd also kick out test_subprocess on 2.7; we
replaced
> subprocess with subprocess32 in our build so that wasn't an issue)

Having the "production ready" make target be "make profile-opt"
doesn't strike me as the most intuitive thing in the world.

I agree we want the "./configure && make" sequence to be oriented
towards local development builds rather than highly optimised
production ones, so perhaps we could provide a "make production"
target that enables PGO with an appropriate training set from
regrtest, and also complains if "--with-pydebug" is configured?


Regards,
Nick.

--
Nick Coghlan   |   ncoghlan@ gmail.com
   |   Brisbane, Australia


Agreed. Also, printing a message out at the end of a default make all build
suggesting people use make production for additional performance instead
might help advertise it.

make install could possibly depend on make production as well?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP-8 wart... it recommends short names because of DOS

[Gregory P. Smith]

https://www.python.org/dev/peps/pep-0008/#names-to-avoid

*"Since module names are mapped to file names, and some file systems are
case insensitive and truncate long names, it is important that module names
be chosen to be fairly short -- this won't be a problem on Unix, but it may
be a problem when the code is transported to older Mac or Windows versions,
or DOS."*

There haven't been computers with less than 80 character file or path name
element length limits in wide use in decades... ;)

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 484 -- proposal to allow @overload in non-stub files

[Gregory P. Smith]

On Wed, Oct 21, 2015 at 6:51 PM Guido van Rossum  wrote:

> Well the whole point is not to have to figure out how to implement that
> right now.
>
> On Wed, Oct 21, 2015 at 6:45 PM, Random832  wrote:
>
>> Guido van Rossum  writes:
>> > The proposal is to allow this to be written as follows in
>> > implementation (non-stub) modules:
>> >
>> > class Foo(Generic[T]):
>> > @overload
>> > def __getitem__(self, i: int) -> T: ...
>> > @overload
>> > def __getitem__(self, s: slice) -> Foo[T]: ...
>> > def __getitem__(self, x):
>> > 
>> >
>> > The actual implementation must be last, so at run time it will
>> > override the definition.
>>
>
I think this *could* be fine. It is certainly readable. And, as is already
possible in .pyi files, more accurately expressive than the Union which
doesn't imply a parameter type to return value type relationship.

What would it Foo.__getitem__.__annotations__ contain in this situation?
It'd unfortunately be an empty dict if implemented in the most trivial
fashion rather than a dict containing your Unions... Do we care?

Note that it would also slow down module import time as the code for each
of the earlier ... definitions with annotation structures and @overload
decorator calls is executed, needlessly creating objects and structures
that are immediately discarded upon each subsequent definition.

-gps


>
>> How about this to allow overloads to have actual implementations?
>>
>> @overloaded
>> def __getitem__(self, x):
>> 
>>
>> @overloaded returns a function which will check the types against the
>> overloads (or anyway any overloads that have actual implementations),
>> call them returning the result if applicable, otherwise call the
>> original function.
>>
>> Some magic with help() would improve usability, too - it could print all
>> the overloads and their docstrings.  Maybe even @overload('__getitem__')
>> def __get_int(self, i: int), to make it so order doesn't matter.
>>
>> That just leaves the question of how's this all gonna work with
>> subclasses.
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 4 musings (was Re: Deleting with setting C API functions)

[Gregory P. Smith]

Except that we should skip version 4 and go directly to 5 in homage to
http://www.montypython.net/scripts/HG-handgrenade.php.

On Wed, Dec 2, 2015 at 9:13 AM Guido van Rossum  wrote:

> On Wed, Dec 2, 2015 at 8:57 AM, Barry Warsaw  wrote:
>
>> On Dec 02, 2015, at 08:35 AM, Guido van Rossum wrote:
>>
>> >I wholeheartedly agree with what Nick writes there
>>
>> As do I.
>>
>> One interesting point will be what *nix calls the /usr/bin thingie for
>> Python
>> 4.  It would seem weird to call it /usr/bin/python3 and symlink it to say
>> /usr/bin/python4.0 but maybe that's the most practical solution.  OTOH, by
>> 2023, Python 2 will at worst be in source-only security release mode, if
>> not
>> finally retired so maybe we can reclaim /usr/bin/python by then.  Oh
>> well, PEP
>> 394 will hash all that out I'm sure.
>>
>
> Maybe the criteria for switching to 4 would be that all traces of 2 are
> gone.
>
>
>> One other potentially disruptive change would be when Python's Einstein,
>> er
>> David Beazley, finally cracks the nut of the GIL.  Should that require a
>> new
>> backward incompatible C API, Python 4.0 would be the time to do it.
>>
>
> There would still have to be a backward compatibility API for a very long
> time. So I don't see why this particular change (however eagerly
> anticipated! :-) should force a major version bump.
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 509: Add a private version to dict

[Gregory P. Smith]

On Mon, Jan 11, 2016 at 8:50 AM Victor Stinner 
wrote:

> Hi,
>
> After a first round on python-ideas, here is the second version of my
> PEP. The main changes since the first version are that the dictionary
> version is no more exposed at the Python level and the field type now
> also has a size of 64-bit on 32-bit platforms.
>
> The PEP is part of a serie of 3 PEP adding an API to implement a
> static Python optimizer specializing functions with guards. The second
> PEP is currently discussed on python-ideas and I'm still working on
> the third PEP.
>
> Thanks to Red Hat for giving me time to experiment on this.
>
>
> HTML version:
> https://www.python.org/dev/peps/pep-0509/
>
>
> PEP: 509
> Title: Add a private version to dict
> Version: $Revision$
> Last-Modified: $Date$
> Author: Victor Stinner 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 4-January-2016
> Python-Version: 3.6
>
>
> Abstract
> 
>
> Add a new private version to builtin ``dict`` type, incremented at each
> change, to implement fast guards on namespaces.
>
>
> Rationale
> =
>
> In Python, the builtin ``dict`` type is used by many instructions. For
> example, the ``LOAD_GLOBAL`` instruction searchs for a variable in the
> global namespace, or in the builtins namespace (two dict lookups).
> Python uses ``dict`` for the builtins namespace, globals namespace, type
> namespaces, instance namespaces, etc. The local namespace (namespace of
> a function) is usually optimized to an array, but it can be a dict too.
>
> Python is hard to optimize because almost everything is mutable: builtin
> functions, function code, global variables, local variables, ... can be
> modified at runtime. Implementing optimizations respecting the Python
> semantics requires to detect when "something changes": we will call
> these checks "guards".
>
> The speedup of optimizations depends on the speed of guard checks. This
> PEP proposes to add a version to dictionaries to implement fast guards
> on namespaces.
>
> Dictionary lookups can be skipped if the version does not change which
> is the common case for most namespaces. The performance of a guard does
> not depend on the number of watched dictionary entries, complexity of
> O(1), if the dictionary version does not change.
>
> Example of optimization: copy the value of a global variable to function
> constants.  This optimization requires a guard on the global variable to
> check if it was modified. If the variable is modified, the variable must
> be loaded at runtime when the function is called, instead of using the
> constant.
>
> See the `PEP 510 -- Specialized functions with guards
> `_ for the concrete usage of
> guards to specialize functions and for the rationale on Python static
> optimizers.
>
>
> Guard example
> =
>
> Pseudo-code of an fast guard to check if a dictionary entry was modified
> (created, updated or deleted) using an hypothetical
> ``dict_get_version(dict)`` function::
>
> UNSET = object()
>
> class GuardDictKey:
> def __init__(self, dict, key):
> self.dict = dict
> self.key = key
> self.value = dict.get(key, UNSET)
> self.version = dict_get_version(dict)
>
> def check(self):
> """Return True if the dictionary entry did not changed."""
>
> # read the version field of the dict structure
> version = dict_get_version(self.dict)
> if version == self.version:
> # Fast-path: dictionary lookup avoided
> return True
>
> # lookup in the dictionary
> value = self.dict.get(self.key, UNSET)
> if value is self.value:
> # another key was modified:
> # cache the new dictionary version
> self.version = version
> return True
>
> # the key was modified
> return False
>
>
> Usage of the dict version
> =
>
> Specialized functions using guards
> --
>
> The `PEP 510 -- Specialized functions with guards
> `_ proposes an API to support
> specialized functions with guards. It allows to implement static
> optimizers for Python without breaking the Python semantics.
>
> Example of a static Python optimizer: the astoptimizer of the `FAT
> Python `_ project
> implements many optimizations which require guards on namespaces.
> Examples:
>
> * Call pure builtins: to replace ``len("abc")`` with ``3``, guards on
>   ``builtins.__dict__['len']`` and ``globals()['len']`` are required
> * Loop unrolling: to unroll the loop ``for i in range(...): ...``,
>   guards on ``builtins.__dict__['range']`` and ``globals()['range']``
>   are required
>
>
> Pyjion
> --
>
> According of Brett Cannon, one of the t

Re: [Python-Dev] Update PEP 7 to require curly braces in C

[Gregory P. Smith]

On Sun, Jan 17, 2016 at 11:12 AM Brett Cannon  wrote:

> While doing a review of http://bugs.python.org/review/26129/ I asked to
> have curly braces put around all `if` statement bodies. Serhiy pointed out
> that PEP 7 says curly braces are optional:
> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change
> that.
>
> My argument is to require them to prevent bugs like the one Apple made
> with OpenSSL about two years ago:
> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
> curly braces is purely an aesthetic thing while leaving them out can lead
> to actual bugs.
>
> Anyone object if I update PEP 7 to remove the optionality of curly braces
> in PEP 7?
>

+1, always using {}s is just good C style.

(and, duh, of course we do *not* go modifying code for this retroactively,
pep8 vs our existing python code is evidence of that)

If I had _my_ way we'd require clang format for C/C++ files and yapf for
all Python files before accepting a commit.  Like any good modern open
source project should.  People who don't like defensive bug reducing coding
practices should be glad I don't get my way. :P

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Code formatter bot

[Gregory P. Smith]

Indeed, automated code formatting is a good thing. But a bot is the wrong
approach. You want a code formatting checker as a potential pre-submit hook
(like we have had for white space issues in the past), but until you have
super high confidence in it you need to make sure it is not a blocker for a
commit or push, just a default check that can be explicitly skipped for the
infrequent cases where it is wrong. It's a workflow thing. Devs should have
their editor setup to auto-run the correct formatter or easily run it if
not automatic.

-gps

On Tue, Jan 19, 2016 at 4:29 PM Senthil Kumaran  wrote:

>
> On Tue, Jan 19, 2016 at 12:59 PM, francismb  wrote:
>
>> Pros, Cons, where could be applicable (new commits, new workflow, it
>> doesn't make sense), ...
>>
>
> -1. formatting should be done by humans (with the help of tools) before
> committing.
> It should not be left to a robot to make automatic changes.
>
> We already some pre-commit hooks which do basic checks. If anything more
> automated is desirable then enhancement to the pre-commit hooks could be
> the place to look for.
> As far as I know, none of the core devs have expressed any complaints the
> pre-commit hooks.
>
>
> --
> Senthil
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Speed] speed.python.org

[Gregory P. Smith]

Displaying ratios linearly rather than on a log scale axis can be
misleading depending on what you are looking for.  (feature request: allow
a log scale?)

major kudos to everyone involved in getting this setup!

On Fri, Feb 5, 2016 at 11:06 PM Nick Coghlan  wrote:

> On 6 February 2016 at 04:07, Brett Cannon  wrote:
> > On Thu, 4 Feb 2016 at 05:46 Nick Coghlan  wrote:
> >> Heh, cdecimal utterly demolishing the old pure Python decimal module
> >> on the telco benchmark means normalising against CPython 3.5 rather
> >> than 2.7 really isn't very readable :)
> >
> > I find viewing the graphs using the horizontal layout is much easier to
> read
> > (the bars are a lot thicker and everything zooms in more).
>
> That comment was based on the horizontal layout - the telco benchmark
> runs ~53x faster in Python 3 than it does in Python 2 (without
> switching to cdecimal), so you end up with all the other benchmarks
> being squashed into the leftmost couple of grid cells.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Improving docs for len() of set

[Gregory P. Smith]

On Mon, Feb 8, 2016 at 8:24 AM Ben Hoyt  wrote:

> Hi folks,
>
> Just a suggestion for a documentation tweak. Currently the docs for len()
> on a set say this:
>
>.. describe:: len(s)
>
>   Return the cardinality of set *s*.
>
> I'm a relatively seasoned programmer, but I don't really have a maths
> background, and I didn't know what "cardinality" meant. I could kind of
> grok it by context, but could we change this to something like the
> following?
>
>.. describe:: len(s)
>
>   Return the number of elements in set *s* (cardinality of *s*).
>
>
Agreed. Done. :)

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue #26204: compiler now emits a SyntaxWarning on constant statement

[Gregory P. Smith]

On Mon, Feb 8, 2016 at 9:44 AM Victor Stinner 
wrote:

> Hi,
>
> I changed the Python compiler to ignore any kind "constant
> expressions", whereas it only ignored strings and integers before:
> http://bugs.python.org/issue26204
>
> The compiler now also emits a SyntaxWarning on such case. IMHO the
> warning can help to detect bugs for developers who just learnt Python.
>
> The warning is *not* emited for strings, since triple quoted strings
> are a common syntax for multiline comments.
>
> The warning is *not* emited neither for ellispis (...) since "f():
> ..." is a legit syntax for abstract function.
>
> Are you ok with the new warning?
>

I'm +1 on this.

-gps


>
>
> New behaviour:
>
> haypo@smithers$ ./python
> Python 3.6.0a0 (default:759a975e1230, Feb  8 2016, 18:21:23)
> >>> def f():
> ...  False
> ...
> :2: SyntaxWarning: ignore constant statement
>
> >>> import dis; dis.dis(f)
>   2   0 LOAD_CONST   0 (None)
>   3 RETURN_VALUE
>
>
> Old behaviour:
>
> haypo@smithers$ python3
> Python 3.4.3 (default, Jun 29 2015, 12:16:01)
> >>> def f():
> ...  False
> ...
> >>> import dis; dis.dis(f)
>   2   0 LOAD_CONST   1 (False)
>   3 POP_TOP
>   4 LOAD_CONST   0 (None)
>   7 RETURN_VALUE
>
>
>
> Before strings and numbers were already ignored. Example:
>
> haypo@smithers$ python3
> Python 3.4.3 (default, Jun 29 2015, 12:16:01)
>
> >>> def f():
> ...  123
> ...
> >>> import dis; dis.dis(f)
>   2   0 LOAD_CONST   0 (None)
>   3 RETURN_VALUE
>
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Disabling changing sys.argv[0] with runpy.run_module(...alter_sys=True)

[Gregory P. Smith]

On Tue, Feb 16, 2016 at 9:00 PM Mike Kaplinskiy 
wrote:

> Hey folks,
>
> I hope this is the right list for this sort of thing (python-ideas seemed
> more far-fetched).
>
> For some context: there is currently a issue with pex that causes
> sys.modules lookups to stop working for __main__. In turns this makes
> unittest.run() & pkg_resources.resource_* fail. The root cause is that pex
> uses runpy.run_module with alter_sys=False. The fix should be to just pass
> alter_sys=True, but that changes sys.argv[0] and various existing pex files
> depend on that being the pex file. You can read more at
> https://github.com/pantsbuild/pex/pull/211 .
>
> Conservatively, I'd like to propose adding an argument to disable this
> behavior. The current behavior breaks a somewhat reasonable invariant that
> you can restart your program via `os.execv([sys.executable] + sys.argv)`.
>

I don't know enough about pex to really dig into what it is trying to do so
this is tangential to answering your question but:

sys.executable may be None. ex: If you're an embedded Python interpreter
there is no Python executable. It cannot be blindly used re-execute the
current process.

sys.argv represents the C main() argv array. Your inclination (in the
linked to bug above) to leave sys.argv[0] alone is a good one.

-gps

Moreover it might be user-friendly to add a `argv=sys.argv[1:]` argument to
> set & restore the full arguments to the module, where `argv=None` disables
> argv[0] switching.
>
> What do you think?
>
> Mike.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buffer overflow bug in GNU C's getaddrinfo()

[Gregory P. Smith]

On Wed, Feb 17, 2016 at 12:12 PM Andrew Barnert via Python-Dev <
python-dev@python.org> wrote:

> On Feb 17, 2016, at 10:44, MRAB  wrote:
> >
> > Is this something that we need to worry about?
> >
> > Extremely severe bug leaves dizzying number of software and devices
> vulnerable
> >
> http://arstechnica.com/security/2016/02/extremely-severe-bug-leaves-dizzying-number-of-apps-and-devices-vulnerable/
>
> Is there a workaround that Python and/or Python apps should be doing, or
> is this just a matter of everyone on glibc 2.9+ needs to update their glibc?
>

There are no workarounds that we could put within Python. People need to
update their glibc and reboot. All *useful(*)* Linux distros have already
released update packages.

All of the infrastructure running Linux needs the update applied and a
reboot (I'm guessing our infrastructure peeps have already done that).  But
this also includes Linux buildbots run by our random set of buildbot donors.

-gps

(*) off topic: Raspbian Wheezy is apparently not on the useful list.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bug in build system for cross-platform builds

[Gregory P. Smith]

On Sun, Mar 13, 2016 at 7:41 PM Martin Panter  wrote:

> On 13 March 2016 at 01:13, Russell Keith-Magee 
> wrote:
> > The patches that I've uploaded to Issue23670 [1] show a full
> cross-platform
> > [1] http://bugs.python.org/issue23670
> > build process. After you apply that patch, the iOS directory contains a
> > meta-Makefile that manages the build process.
>
> Thanks very much for pointing that out. This has helped me understand
> a lot more things. Only now do I realize that the four files generated
> by pgen and _freeze_importlib are actually already committed into the
> Mercurial repository:
>
> Include/graminit.h
> Python/graminit.c
> Python/importlib.h
> Python/importlib_external.h
>
> A question for other Python developers: Why are these generated files
> stored in the repository? The graminit ones seem to have been there
> since forever (1990). It seems the importlib ones were there due to a
> bootstrapping problem, but now that is solved. Antoine
>  said he kept them in
> the repository on purpose, but I want to know why.
>
> If we ignore the cross compiling use case, would there be any other
> consequences of removing these generated files from the repository?
> E.g. would it affect the Windows build process?
>
> I have two possible solutions in mind: either remove the generated
> files from the repository and always build them, or keep them but do
> not automatically regenerate them every build. Since they are
> generated files, not source files, I would prefer to remove them, but
> I want to know the consequences first.
>

They should not be regenerated every build, if they are, that seems like a
bug in the makefile to me (or else the timestamp checks that make does vs
how your code checkout happened).  Having them checked in is convenient for
cross builds as it is one less thing that needs a build-host-arch build.

my 2 cents,
-gps


>
> > On Sat, Mar 12, 2016 at 8:48 AM, Martin Panter 
> wrote:
> >> On 11 March 2016 at 23:16, Russell Keith-Magee  >
> >> wrote:
> >> >
> >> > On Sat, Mar 12, 2016 at 6:38 AM, Martin Panter 
> >> > wrote:
> >> >> I don't understand. After I run Make, it looks like I get working
> >> >> executables leftover at Programs/_freeze_importlib and Parser/pgen.
> Do
> >> >> you mean to install these programs with "make install" or something?
> >> >
> >> >
> >> > Making them part of the installable artefacts would be one option, but
> >> > they
> >> > don't have to be installed, just preserved.
> >>
> >> What commands are you running that cause them to not be preserved at
> >> the end of the build?
> >
> >
> > I don't know - this is where I hit the end of my understanding of the
> build
> > process. All I know for certain is that 3.4.2 doesn't have this problem;
> > 3.5.1 does, and Issue22359 (fixed in [3]) is the source of the problem. A
> > subsequent fix [4] introduced an additional problem with
> _freeze_importlib.
> >
> > [3] https://hg.python.org/cpython/rev/565b96093ec8
> > [4] https://hg.python.org/cpython/rev/02e3bf65b2f8
>
> After my realization about the generated files, I think I can solve
> this one. Before the changes you identified, the build process
> probably thought the generated files were up to date, so it didn't
> need to cross-compile pgen or _freeze_importlib. If the generated file
> timestamps were out of date (e.g. depending on the order they are
> checked out or extracted), the first native build stage would have
> fixed them up.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New OpenSSL - has anyone ever looked at (in)compatibility with LibreSSL

[Gregory P. Smith]

Don't forget BoringSSL.

On Wed, Mar 9, 2016 at 9:30 AM Michael Felt  wrote:

> Can look at it. There has been a lot of discussion, iirc, between OpenSSL
> and LibreSSL re: version identification.
> Thx for the reference.
>
>
> On 08-Mar-16 14:55, Hasan Diwan wrote:
>
>
> On 8 March 2016 at 00:49, Michael Felt  wrote:
>
>> As a relative newcomer I may have missed a long previous discussion re:
>> linking with OpenSSL and/or LibreSSL.
>> In an ideal world this would be rtl linking, i.e., underlying
>> complexities of *SSL libraries are hidden from applications.
>>
>> In short, when I saw this http://bugs.python.org/issue26465 Title:
>> Upgrade OpenSSL shipped with python installers, it reminded me I need to
>> start looking at LibreSSL again - and that, if not already done - might be
>> something "secure" for python as well.
>>
>
> According to the libressl website, one of the projects primary goals is to
> remain "backwards-compatible with OpenSSL", which is to say, to either
> have code work without changes or to fail gracefully when it uses the
> deprecated bits. It does seem it ships with OpenBSD. There is an issue open
> on bugs to address whatever incompatibilities remain between LibreSSL and
> OpenSSL[1]. Perhaps you might want to take a look at that? -- H
> 1. https://bugs.python.org/issue23177
>
>>
>> Michael
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/hasan.diwan%40gmail.com
>>
>
>
>
> --
> OpenPGP: http://hasan.d8u.us/gpg.asc
> Sent from my mobile device
> Envoyé de mon portable
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New OpenSSL - has anyone ever looked at (in)compatibility with LibreSSL

[Gregory P. Smith]

On Mon, Mar 14, 2016 at 4:56 PM Nathaniel Smith  wrote:

> Should people outside google pay attention to boringssl? The first
> thing it says on the website is:
>
> "Although BoringSSL is an open source project, it is not intended for
> general use, as OpenSSL is. We don’t recommend that third parties
> depend upon it. Doing so is likely to be frustrating because there are
> no guarantees of API or ABI stability."
>

Heh, good point.  I guess not.  :)


> On Mon, Mar 14, 2016 at 4:40 PM, Gregory P. Smith  wrote:
> > Don't forget BoringSSL.
> >
> > On Wed, Mar 9, 2016 at 9:30 AM Michael Felt  wrote:
> >>
> >> Can look at it. There has been a lot of discussion, iirc, between
> OpenSSL
> >> and LibreSSL re: version identification.
> >> Thx for the reference.
> >>
> >>
> >> On 08-Mar-16 14:55, Hasan Diwan wrote:
> >>
> >>
> >> On 8 March 2016 at 00:49, Michael Felt  wrote:
> >>>
> >>> As a relative newcomer I may have missed a long previous discussion re:
> >>> linking with OpenSSL and/or LibreSSL.
> >>> In an ideal world this would be rtl linking, i.e., underlying
> >>> complexities of *SSL libraries are hidden from applications.
> >>>
> >>> In short, when I saw this http://bugs.python.org/issue26465 Title:
> >>> Upgrade OpenSSL shipped with python installers, it reminded me I need
> to
> >>> start looking at LibreSSL again - and that, if not already done -
> might be
> >>> something "secure" for python as well.
> >>
> >>
> >> According to the libressl website, one of the projects primary goals is
> to
> >> remain "backwards-compatible with OpenSSL", which is to say, to either
> have
> >> code work without changes or to fail gracefully when it uses the
> deprecated
> >> bits. It does seem it ships with OpenBSD. There is an issue open on
> bugs to
> >> address whatever incompatibilities remain between LibreSSL and
> OpenSSL[1].
> >> Perhaps you might want to take a look at that? -- H
> >> 1. https://bugs.python.org/issue23177
> >>>
> >>>
> >>> Michael
> >>> ___
> >>> Python-Dev mailing list
> >>> Python-Dev@python.org
> >>> https://mail.python.org/mailman/listinfo/python-dev
> >>> Unsubscribe:
> >>>
> https://mail.python.org/mailman/options/python-dev/hasan.diwan%40gmail.com
> >>
> >>
> >>
> >>
> >> --
> >> OpenPGP: http://hasan.d8u.us/gpg.asc
> >> Sent from my mobile device
> >> Envoyé de mon portable
> >>
> >>
> >> ___
> >> Python-Dev mailing list
> >> Python-Dev@python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> >> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
> >
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
> >
>
>
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Defining a path protocol

[Gregory P. Smith]

Note: While I do not object to the bike shed colors being proposed, if you
call the attribute .__path__ that is somewhat confusing when thinking about
the import system which declares that *"any module that contains a __path__
attribute is considered a package"*.

So would module.__path__ become a Path instance in a potential future
making module.__path__.__path__ meaningfully confusing? ;)

I'm not worried about people who shove pathlib.Path instances in as values
into sys.modules and expect anything but pain. :P

__gps__



On Wed, Apr 6, 2016 at 3:46 PM Brett Cannon  wrote:

> On Wed, 6 Apr 2016 at 15:22 Paul Moore  wrote:
>
>> On 6 April 2016 at 20:39, Brett Cannon  wrote:
>> >> I'm a little confused by this. To support the older pathlib, they have
>> >> to do patharg = str(patharg), because *none* of the proposed
>> >> attributes (path or __path__) will exist.
>> >>
>> >> The getattr trick is needed to support the *new* pathlib, when you
>> >> need a real string. Currently you need a string if you call stdlib
>> >> functions or builtins. If we fix the stdlib/builtins, the need goes
>> >> away for those cases, but remains if you need to call libraries that
>> >> *don't* support pathlib (os.path will likely be one of those) or do
>> >> direct string manipulation.
>> >>
>> >> In practice, I see the getattr trick as an "easy fix" for libraries
>> >> that want to add support but in a minimally-intrusive way. On that
>> >> basis, making the trick easy to use is important, which argues for an
>> >> attribute.
>> >
>> > So then where's the confusion? :) You seem to get the points. I
>> personally
>> > find `path.__path__() if hasattr(path, '__path__') else path` also
>> readable
>> > (if obviously a bit longer).
>>
>> The confusion is that you seem to be saying that people can use
>> getattr(path, '__path__', path) to support older versions of Python.
>> But the older versions are precisely the ones that don't have __path__
>> so you won't be supporting them.
>>
>
> Because pathlib is provisional the change will go into the next releases
> of Python 3.4, 3.5, and in 3.6 so new-old will have whatever we do. :) I
> think the key point is that this sort of thing will occur before you have
> access to some new built-in or something.
>
>
>>
>> >> >> >  3. Built-in? (name is dependent on #1 if we add one)
>> >> >>
>> >> >> fspath() -- and it would be handy to have a function that return
>> either
>> >> >> the __fspath__ results, or the string (if it was one), or raise an
>> >> >> exception if neither of the above work out.
>> >>
>> >> fspath regardless of the name chosen in #1 - a new builtin called path
>> >> just has too much likelihood of clashing with user code.
>> >>
>> >> But I'm not sure we need a builtin. I'm not at all clear how
>> >> frequently we expect user code to need to use this protocol. Users
>> >> can't use the builtin if they want to be backward compatible, But code
>> >> that doesn't need backward compatibility can probably just work with
>> >> pathlib (and the stdlib support for it) directly. For display, the
>> >> implicit conversion to str is fine. For "get me a string representing
>> >> the path", is the "path" attribute being abandoned in favour of this
>> >> special method?
>> >
>> > Yes.
>>
>> OK. So the idiom to get a string from a known Path object would be any of:
>>
>> 1. str(path)
>> 2. fspath(path)
>> 3. path.__path__()
>>
>> (1) is safe if you know you have a Path object, but could incorrectly
>> convert non-Path objects. (2) is safe in all cases. (3) is ugly. Did I
>> miss any options?
>>
>
> Other than path.__path__ being an attribute, nope.
>
>
>>
>> So I think we need a builtin.
>>
>
> Well, the ugliness shouldn't survive forever if the community shifts over
> to using pathlib while the built-in will. We also don't have a built-in for
> __index__() so it depends on whether we expect this sort of thing to be the
> purview of library authors or if normal people will be interacting with it
> (it's probably both during the transition, but I don't know afterwards).
>
>
>>
>> Code that needs to be backward compatible will still have to use
>> str(path), because neither the builtin nor the __path__ protocol will
>> exist in older versions of Python.
>
>
> str(path) will definitely work, path.__path__ will work if you're running
> the next set of bugfix releases. fspath(path) will only work in Python 3.6
> and newer.
>
>
>> Maybe a compatibility library could
>> add
>>
>> try:
>> fspath
>> except NameError:
>> try:
>> import pathlib
>> def fspath(p):
>> if isinstance(p, pathlib.Path):
>> return str(p)
>> return p
>> except ImportError:
>> def fspath(p):
>> return p
>>
>> It's messy, like all compatibility code, but it allows code to use
>> fspath(p) in older versions.
>>
>
> I would tweak it to check for __fspath__ before it resorted to calling
> str(), but yes, that could be something people use.
>

Re: [Python-Dev] python2 and python3 and vim

[Gregory P. Smith]

On Sun, Nov 3, 2013 at 8:59 AM, R. David Murray wrote:

> I came across this in the VIM documentation:
>
> Vim can be built in four ways (:version output):
> 1. No Python support(-python, -python3)
> 2. Python 2 support only(+python or +python/dyn, -python3)
> 3. Python 3 support only(-python, +python3 or +python3/dyn)
> 4. Python 2 and 3 support   (+python/dyn, +python3/dyn)
>
> Some more details on the special case 4:
>
> When Python 2 and Python 3 are both supported they must be loaded
> dynamically.
>
> When doing this on Linux/Unix systems and importing global symbols,
> this leads
> to a crash when the second Python version is used.  So either global
> symbols
> are loaded but only one Python version is activated, or no global
> symbols are
> loaded. The latter makes Python's "import" fail on libraries that
> expect the
> symbols to be provided by Vim.
>
> I've never played with embedding Python.  Does this make sense to
> anyone who has?  Is there some limitation in our embedding and/or
> import machinery here, or is this more likely to be a shortcoming on
> the VIM side?
>
> Mostly I'm asking out of curiosity in hopes of learning something;
> I doubt I'll have enough motivation to make time to work on solving this.
>

I'm not used to it being possible to have multiple different embedded
Python interpreters in one process at all. The Python C API symbols
exported by one python interpreter will conflict with another python
interpreter. We don't provide isolation of interpreters within a process.
IIRC, if you do have multiple even of the same version within a single
process they are still all sharing the same python memory allocator pools
and GIL and more.

That someone even went through the hoops to attempt to get vim to allow
having some semblance of both python 2 and python 3 embedded dynamically at
runtime seems like overkill to me.

Pick one and go with it. Vim should declare at some point in 2014 that the
embedded Python in vim is now Python 3 only and move on. The rare users who
actually use the embedded bazillion scripting languages within vim will
need to update their code, ideally to be 2 and 3 compatible.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASED] Python 3.3.3 final

[Gregory P. Smith]

How else would you know that anyone is listening? :)

[yay, thanks for our being release monkey!]

-gps


On Tue, Nov 19, 2013 at 11:40 AM, Georg Brandl  wrote:

> Am 19.11.2013 17:14, schrieb Mark Lawrence:
> > On 19/11/2013 06:59, Georg Brandl wrote:
> >>
> >> To download Python 3.3.3 rc2 visit:
> >>
> >>  http://www.python.org/download/releases/3.3.3/
> >>
> >
> > Please make your mind up, final or rc2?
> >
> > Thanks everybody for your efforts, much appreciated :)
>
> It's my firm belief that every announce should have a small error
> to appease the gods of regression. *ahem*
>
> :)
> Georg
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] can someone create a buildbot slave name & password for me?

[Gregory P. Smith]

I've got a nice ARM box to add to the pool (it should be a lot faster and
more reliable than warsaw-ubuntu-arm; hopefully making its way to stable).

proposed slave name: gps-ubuntu-exynos5-armv7l

description: Ubuntu ARM - Dual Cortex A15 2GiB  (ie: a repurposed samsung
ARM chromebook)

thanks,
-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] can someone create a buildbot slave name & password for me?

[Gregory P. Smith]

On Sat, Nov 23, 2013 at 12:31 PM, Barry Warsaw  wrote:

> On Nov 23, 2013, at 10:34 AM, Gregory P. Smith wrote:
>
> >I've got a nice ARM box to add to the pool (it should be a lot faster and
> >more reliable than warsaw-ubuntu-arm; hopefully making its way to stable).
>
> Once that's up, let me know.  I'd like to eventually retire this box.  It's
> unsupported hardware and barely limping along.
>

It's up and running:

http://buildbot.python.org/all/buildslaves/gps-ubuntu-exynos5-armv7l
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] buildbot's are needlessly compiling -O0

[Gregory P. Smith]

our buildbots are setup to configure --with-pydebug which also
unfortunately causes them to compile with -O0... this results in a python
binary that is excruciatingly slow and makes the entire test suite run take
a long time.

given that nobody is ever going to run a gdb or another debugger on the
buildbot generated transient binaries themselves how about we speed all of
the buildbot's up by adding CFLAGS=-O2 to the configure command line?

Sure, the compile step will take a bit longer but that is dwarfed by the
test time as it is:

http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/3224
http://buildbot.python.org/all/builders/ARMv7%203.x/builds/7
http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/639

It should dramatically decrease the turnaround latency for buildbot results.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix test_fcntl to run properly on systems that do not support the flags

[Gregory P. Smith]

On Mon, Nov 25, 2013 at 12:46 AM, Nick Coghlan  wrote:

>
> On 25 Nov 2013 14:46, "gregory.p.smith" 
> wrote:
> >
> > http://hg.python.org/cpython/rev/cac7319c5972
> > changeset:   87537:cac7319c5972
> > branch:  2.7
> > parent:      87534:3981e57a7bdc
> > user:Gregory P. Smith 
> > date:Mon Nov 25 04:45:27 2013 +
> > summary:
> >   Fix test_fcntl to run properly on systems that do not support the flags
> > used in the "does the value get passed in properly" test.
> >
> > files:
> >   Lib/test/test_fcntl.py |  3 +++
> >   1 files changed, 3 insertions(+), 0 deletions(-)
> >
> >
> > diff --git a/Lib/test/test_fcntl.py b/Lib/test/test_fcntl.py
> > --- a/Lib/test/test_fcntl.py
> > +++ b/Lib/test/test_fcntl.py
> > @@ -113,7 +113,10 @@
> >  self.skipTest("F_NOTIFY or DN_MULTISHOT unavailable")
> >  fd = os.open(os.path.dirname(os.path.abspath(TESTFN)),
> os.O_RDONLY)
> >  try:
> > +# This will raise OverflowError if issue1309352 is present.
> >  fcntl.fcntl(fd, cmd, flags)
> > +except IOError:
> > +pass  # Running on a system that doesn't support these
> flags.
> >  finally:
> >  os.close(fd)
>
> Raising a skip is generally preferred to marking a test that can't be
> executed as passing.
>
For this test this is still a pass because no OverflowError was raised.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

[Gregory P. Smith]

On Tue, Dec 17, 2013 at 8:43 AM, Stefan Krah  wrote:

> Maciej Fijalkowski  wrote:
> > I would like to discuss on the language summit a potential inclusion
> > of cffi[1] into stdlib. This is a project Armin Rigo has been working
> > for a while, with some input from other developers.
>
> I've tried cffi (admittedly only in a toy script) and find it very nice
> to use.
>
> Here's a comparison (pi benchmark) between wrapping libmpdec using a
> C-extension (_decimal), cffi and ctypes:
>
>
> +---+--+--+-+
> |   | _decimal |  ctypes  |   cffi  |
> +===+==+==+=+
> | cpython-tip (with-system-ffi) |   0.19s  |   5.40s  |  5.14s  |
> +---+--+--+-+
> | cpython-2.7 (with-system-ffi) |n/a   |   4.46s  |  5.18s  |
> +---+--+--+-+
> |  Ubuntu-cpython-2.7   |n/a   |   3.63s  |-|
> +---+--+--+-+
> |  pypy-2.2.1-linux64   |n/a   |  125.9s  |  0.94s  |
> +---+--+--+-+
> | pypy3-2.1-beta1-linux64   |n/a   |  264.9s  |  2.93s  |
> +---+--+--+-+
>
>
> I guess the key points are that C-extensions are hard to beat and that
> cffi performance on pypy-2 is outstanding. Additionally it's worth noting
> that Ubuntu does something in their Python build that we should do, too.
>

Ubuntu compiles their Python with FDO (feedback directed optimization /
profile guided optimization) enabled. All distros should do this if they
don't already. It's generally 20% interpreter speedup. Our makefile already
supports it but it isn't the default build as it takes a long time given
that it needs to compile everything twice and do a profiled benchmark run
between compilations.

-gps


>
> +1 for cffi in the stdlib.
>
>
>
> Stefan Krah
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cffi in stdlib

[Gregory P. Smith]

On Thu, Dec 19, 2013 at 2:07 AM, Maciej Fijalkowski wrote:

> On Thu, Dec 19, 2013 at 3:17 AM, Gregory P. Smith  wrote:
> >
> >
> >
> > On Tue, Dec 17, 2013 at 8:43 AM, Stefan Krah 
> wrote:
> >>
> >> Maciej Fijalkowski  wrote:
> >> > I would like to discuss on the language summit a potential inclusion
> >> > of cffi[1] into stdlib. This is a project Armin Rigo has been working
> >> > for a while, with some input from other developers.
> >>
> >> I've tried cffi (admittedly only in a toy script) and find it very nice
> >> to use.
> >>
> >> Here's a comparison (pi benchmark) between wrapping libmpdec using a
> >> C-extension (_decimal), cffi and ctypes:
> >>
> >>
> >> +---+--+--+-+
> >> |   | _decimal |  ctypes  |   cffi  |
> >> +===+==+==+=+
> >> | cpython-tip (with-system-ffi) |   0.19s  |   5.40s  |  5.14s  |
> >> +---+--+--+-+
> >> | cpython-2.7 (with-system-ffi) |n/a   |   4.46s  |  5.18s  |
> >> +---+--+--+-+
> >> |  Ubuntu-cpython-2.7   |n/a   |   3.63s  |-|
> >> +---+--+--+-+
> >> |  pypy-2.2.1-linux64   |n/a   |  125.9s  |  0.94s  |
> >> +---+--+--+-+
> >> | pypy3-2.1-beta1-linux64   |n/a   |  264.9s  |  2.93s  |
> >> +---+--+--+-+
> >>
> >>
> >> I guess the key points are that C-extensions are hard to beat and that
> >> cffi performance on pypy-2 is outstanding. Additionally it's worth
> noting
> >> that Ubuntu does something in their Python build that we should do, too.
> >
> >
> > Ubuntu compiles their Python with FDO (feedback directed optimization /
> > profile guided optimization) enabled. All distros should do this if they
> > don't already. It's generally 20% interpreter speedup. Our makefile
> already
> > supports it but it isn't the default build as it takes a long time given
> > that it needs to compile everything twice and do a profiled benchmark run
> > between compilations.
> >
> > -gps
>
> Hey Greg.
>
> We found out that this only speedups benchmarks that you tried during
> profiling and not others, so we disabled it for the default pypy
> build. Can you provide me with some more detailed study on how it
> speeds up interpreters in general and CPython in particular?
>

That's a common concern for profile based builds but it turns out not to
matter a whole lot which workloads you choose for the CPython interpreter
to collect profiles for a FDO build. I believe ubuntu's packages just use
the test suite. In our own tests at work this produced good results.
Interpreter loops and other common code paths in the interpreter have a
*lot* of low hanging fruit in terms of more optimal code generation.

Link time optimization adds additional benefits IF you can get it working
(not always easy or reliable right now as Matthias mentions in issue17781).

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.x vs 3.x survey results

[Gregory P. Smith]

Somewhere you need to describe the survey methodology, who was surveyed,
how were they selected, etc.


On Thu, Jan 2, 2014 at 2:54 PM, Dan Stromberg  wrote:

> On Thu, Jan 2, 2014 at 1:34 PM, Antoine Pitrou 
> wrote:
> > On Thu, 2 Jan 2014 13:10:36 -0800
> > Dan Stromberg  wrote:
> >> Is there a better place to put this than:
> >> http://stromberg.dnsalias.org/~strombrg/python-2.x-vs-3.x-survey/
> >
> > Thank you for doing this!
>
> My pleasure.
>
> > If wiki.python.org supports file uploads, it may be the place for
> > publishing the results.
>
> I put it at https://wiki.python.org/moin/2.x-vs-3.x-survey , with a
> link from https://wiki.python.org/moin/Python2orPython3 .  I put it in
> the "CategoryImplementations" category.
>
> The text I added to https://wiki.python.org/moin/Python2orPython3
> reads "Some people just don't want to use Python 3.x, which is their
> prerogative. However, they are in the minority.", where "in the
> minority" is a link to the survey page.
>
> Does anyone want to vet it before I tell more people where to find it?
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] version numbers mismatched in google search results.

[Gregory P. Smith]

I also get search results with Python 1.5.0p2 showing up.

Search for PyArg_ParseTuple. The first result is a URL with /2/ in it who's
search result title says "3.3.3" but opening it is the correct 2.x
documentation. The second result is the ancient Python 1.5.0 docs. ;)

Should the ancient /release/ docs have redirects setup or be somehow marked
as no crawl?  http://docs.python.org/release/1.5.2p2/ext/parseTuple.html is
the humorous result in this case.

"I want to know how the API I'm using behaved 15 years ago!", said no one
ever.

-gps



On Sat, Jan 25, 2014 at 9:34 PM, Benjamin Peterson wrote:

>
>
> On Sat, Jan 25, 2014, at 07:04 PM, Nick Coghlan wrote:
> > Which suggests that the Google web crawler *is* spidering the dev
> > docs, which we generally don't want :P
>
> I've now added a robots.txt to disallow crawling /dev.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The docstring hack for signature information has to go

[Gregory P. Smith]

On Mon, Feb 3, 2014 at 8:04 AM, Larry Hastings  wrote:

>  On 02/03/2014 07:08 AM, Barry Warsaw wrote:
>
> On Feb 03, 2014, at 06:43 AM, Larry Hastings wrote:
>
>
>  But that only fixes part of the problem.  Our theoretical extension that
> wants to be binary-compatible with 3.3 and 3.4 still has a problem: how can
> they support signatures?  They can't give PyMethodDefEx structures to 3.3, it
> will blow up.  But if they don't use PyMethodDefEx, they can't have
> signatures.
>
>  Can't an extension writer #ifdef around this?  Yeah, it's ugly, but it's a
> pretty standard approach for making C extensions multi-version compatible.
>
>
> For source compatibility, yes.  But I thought the point of the binary ABI
> was to allow compiling a single extension that worked unmodified with
> multiple versions of Python.  If we simply don't support that, then an
> ifdef would be fine.
>

Wouldn't your proposal to extend the PyMethodDef structure would require
ifdef's and make it impossible to include the type information in something
compiled against the 3.3 headers that you want to use in 3.4 without
recompiling?

If you don't like seeing an sig= at the front of the docstring couldn't you
just move it to the end of the docstring.

I don't think messiness in docstrings when running something read for 3.4
under 3.3 is a big deal.

[side note]
I consider it CRAZY for anyone to load a binary extension module compiled
for one version in a later version of Python. People do it, I know, but
they're insane. I wish we didn't bother trying to support that crap. I know
this isn't going to change in 3.4. Just ranting.
[/side note]

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final

[Gregory P. Smith]

For 3.4.0rc2 the commit to merge from issue20621 is 52ab9e1ff46a.


On Sun, Feb 16, 2014 at 3:45 PM, Paul Moore  wrote:

>  is significant enough to be
> resulting in a 3.3.5 release - can you make sure the 3.4 fix goes in?
> I'm not sure how to find the revision number that contains the fix to
> follow the process you outline above, so I'm just mentioning it here &
> on the issue to make sure it's not missed...
>
> Paul
>
> On 16 February 2014 23:25, Larry Hastings  wrote:
> >
> >
> > Right now we're in the "release candidate" phase of 3.4.  3.4.0 rc1 has
> been
> > released, and the next release will be rc2.
> >
> > You might think that anything you check in to the "default" branch in
> Python
> > trunk will go into 3.4.0 rc2, and after that ships, checkins would go
> into
> > 3.4.0 final.  Ho ho ho!  That's not true!  Instead, anything checked in
> to
> > "default" between my last revision for "rc1" (e64ae8b82672) and 3.4.0
> final
> > will by default go into 3.4.1.  Only fixes that I cherry-pick into my
> local
> > branch will go into 3.4.0 rc2 and final.  And my local branch will remain
> > private until 3.4.0 final ships!
> >
> > If you have a Terribly Important Fix That Must Go Into 3.4.0 rc2 or
> final,
> > please go to the issue tracker and create a new issue with the following
> > attributes:
> >
> > The title should start with "3.4 cherry-pick: " followed by the revision
> id
> > and a short summary
> >   example: "3.4 cherry-pick: b328f8ccbccf __getnewargs__ fix"
> > The version should be "Python 3.4"
> > The assignee should be "larry"
> > The priority should be "release blocker"
> > The comment should *also* contain the revision id (the tracker will turn
> it
> > into a link)
> >
> > I'm also working on automatically publishing the merged/unmerged revision
> > status to a web page.  You can see a mock-up here:
> >
> > http://www.midwinter.com/~larry/3.4.merge.status.html
> >
> > The page is marked "beta" because it doesn't have real data yet--I'm
> still
> > experimenting with my automation, so I haven't created the real 3.4 local
> > branch yet.  Again, just to be crystal-clear: the revisions marked
> "merged"
> > on that page are just experiments, they aren't actually merged for 3.4.
> > Once I'm ready for real merging, I'll remove the beta warning.
> >
> > (By the way: on that page, clicking on a revision takes you to the
> revision
> > web page.  Clicking on the first line of the comment expands it to show
> the
> > complete comment.)
> >
> >
> > Please use your best judgment before asking that a revision be
> cherry-picked
> > into 3.4.0.  Our goal in the release candidate phase is to stabilize
> Python,
> > and to do that we must stop changing it.  Only important interface
> changes,
> > new features, or bugfixes should be checked in now, and preferably they
> > should be low-risk.
> >
> > Cheers,
> >
> >
> > /arry
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/p.f.moore%40gmail.com
> >
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final

[Gregory P. Smith]

http://bugs.python.org/issue20651 filed to track this as larry requested.


On Sun, Feb 16, 2014 at 7:09 PM, Gregory P. Smith  wrote:

> For 3.4.0rc2 the commit to merge from issue20621 is 52ab9e1ff46a.
>
>
> On Sun, Feb 16, 2014 at 3:45 PM, Paul Moore  wrote:
>
>> <http://bugs.python.org/issue20621> is significant enough to be
>> resulting in a 3.3.5 release - can you make sure the 3.4 fix goes in?
>> I'm not sure how to find the revision number that contains the fix to
>> follow the process you outline above, so I'm just mentioning it here &
>> on the issue to make sure it's not missed...
>>
>> Paul
>>
>> On 16 February 2014 23:25, Larry Hastings  wrote:
>> >
>> >
>> > Right now we're in the "release candidate" phase of 3.4.  3.4.0 rc1 has
>> been
>> > released, and the next release will be rc2.
>> >
>> > You might think that anything you check in to the "default" branch in
>> Python
>> > trunk will go into 3.4.0 rc2, and after that ships, checkins would go
>> into
>> > 3.4.0 final.  Ho ho ho!  That's not true!  Instead, anything checked in
>> to
>> > "default" between my last revision for "rc1" (e64ae8b82672) and 3.4.0
>> final
>> > will by default go into 3.4.1.  Only fixes that I cherry-pick into my
>> local
>> > branch will go into 3.4.0 rc2 and final.  And my local branch will
>> remain
>> > private until 3.4.0 final ships!
>> >
>> > If you have a Terribly Important Fix That Must Go Into 3.4.0 rc2 or
>> final,
>> > please go to the issue tracker and create a new issue with the following
>> > attributes:
>> >
>> > The title should start with "3.4 cherry-pick: " followed by the
>> revision id
>> > and a short summary
>> >   example: "3.4 cherry-pick: b328f8ccbccf __getnewargs__ fix"
>> > The version should be "Python 3.4"
>> > The assignee should be "larry"
>> > The priority should be "release blocker"
>> > The comment should *also* contain the revision id (the tracker will
>> turn it
>> > into a link)
>> >
>> > I'm also working on automatically publishing the merged/unmerged
>> revision
>> > status to a web page.  You can see a mock-up here:
>> >
>> > http://www.midwinter.com/~larry/3.4.merge.status.html
>> >
>> > The page is marked "beta" because it doesn't have real data yet--I'm
>> still
>> > experimenting with my automation, so I haven't created the real 3.4
>> local
>> > branch yet.  Again, just to be crystal-clear: the revisions marked
>> "merged"
>> > on that page are just experiments, they aren't actually merged for 3.4.
>> > Once I'm ready for real merging, I'll remove the beta warning.
>> >
>> > (By the way: on that page, clicking on a revision takes you to the
>> revision
>> > web page.  Clicking on the first line of the comment expands it to show
>> the
>> > complete comment.)
>> >
>> >
>> > Please use your best judgment before asking that a revision be
>> cherry-picked
>> > into 3.4.0.  Our goal in the release candidate phase is to stabilize
>> Python,
>> > and to do that we must stop changing it.  Only important interface
>> changes,
>> > new features, or bugfixes should be checked in now, and preferably they
>> > should be low-risk.
>> >
>> > Cheers,
>> >
>> >
>> > /arry
>> >
>> > ___
>> > Python-Dev mailing list
>> > Python-Dev@python.org
>> > https://mail.python.org/mailman/listinfo/python-dev
>> > Unsubscribe:
>> >
>> https://mail.python.org/mailman/options/python-dev/p.f.moore%40gmail.com
>> >
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>>
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Gregory P. Smith]

On Thu, Mar 27, 2014 at 3:05 PM, Antoine Pitrou  wrote:

> On Thu, 27 Mar 2014 18:47:59 +
> Brett Cannon  wrote:
> > On Thu Mar 27 2014 at 2:42:40 PM, Guido van Rossum 
> wrote:
> >
> > > Much better, but I'm still not happy with including %s at all.
> Otherwise
> > > it's accept-worthy. (How's that for pressure. :-)
> > >
> >
> > But if we only add %b and leave out %s then how is this going to lead to
> > Python 2/3 compatible code since %b is not in Python 2? Or am I
> > misunderstanding you?
>
> I think we have reached a point where adding porting-related facilities
> in 3.5 may actually slow down the pace of porting, rather than
> accelerate it (because people will then wait for 3.5 to start porting
> stuff).
>

I understand that sentiment but that is an unjustified fear. It is not a
good reason not to do it. Projects are already trying to port stuff today
and running into roadblocks when it comes to ascii-compatible bytes
formatting for real world data formats in code needing to be 2.x
compatible. I'm pulling out my practicality beats purity card here.

Mercurial is one of the large Python 2.4-2.7 code bases that needs this
feature in order to support Python 3 in a sane manner. (+Augie Fackler to
look at the latest http://legacy.python.org/dev/peps/pep-0461/ to confirm
usefulness)

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Gregory P. Smith]

On Thu, Mar 27, 2014 at 3:47 AM, Victor Stinner wrote:

> The PEP 461 looks good to me. It's a nice addition to Python 3.5 and
> the PEP is well defined.
>

+1


> I can help to implement it. Maybe, it would be nice to provide an
> implementation as a third-party party module on PyPI for Python
> 2.6-3.4.
>

That is possible and would enable bytes formatting on earlier 3.x versions.
I'm not sure if there is any value in backporting to 2.x as those already
have such formatting with Python 2's str.__mod__ % operator.

Though I don't know what it'd look like as an API as a module.
Brainstorming: It'd either involve function calls to format instead of % or
a container class to wrap format strings in with a __mod__ method that
calls the bytes formatting code instead of native str % formatting when
needed.

>From a 2.x-3.x compatible code standpoint the above could exist but the
container class constructor would be a no-op on Python 2.
  if sys.version_info[0] == 2:
BytesFormatter = str
  else:
class BytesFormatter: ... def __mod__ ...

-gps


> Note: I fixed a typo in your PEP (reST syntax).
>
> Victor
>
> 2014-03-26 23:47 GMT+01:00 Ethan Furman :
> >> This one is wrong:
> >>
> > repr(b'abc').encode('ascii', 'backslashreplace')
> >>
> >> b"b'abc'"
> >
> >
> > Fixed, thanks.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows installers and OpenSSL

[Gregory P. Smith]

Yep. All binary Python distributions that bundle SSL support need updating.
But... what MRAB said.

We also *likely* have SSL certificates and SSH host keys on
python.orginfrastructure that need to be revoked and new certs
reissued
*after* all of those machines have been patched and their services
restarted. Including hg.python.org.

-gps



On Thu, Apr 10, 2014 at 10:51 AM, MRAB  wrote:

> On 2014-04-10 14:41, Paul Moore wrote:
>
>> Given the OpenSSL vulnerability and the fact that we bundle OpenSSL
>> with the Windows installers (1.0.1e in Python 3.4.0) should we be
>> releasing updated installers?
>>
>>  I'd say yes, but, then, I wouldn't be doing any of the work...
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Existence of pythonNN.zip in sys.path

[Gregory P. Smith]

On Mon, May 5, 2014 at 2:52 PM, Anthony Tuininga  wrote:

> On Mon, May 5, 2014 at 3:50 PM, Paul Moore  wrote:
>
>> On 5 May 2014 22:32, Anthony Tuininga  wrote:
>> > So my question is: can I safely make use of this "feature"? It has
>> remained
>> > in place since at least Python 2.6 but I don't want to assume anything.
>> > Please advise! Thanks.
>>
>> I believe this is a 100% supported feature of Python (since 2.6, as
>> you mention). Some of the zip import/run features have unfortunately
>> been underdocumented, and I suspect that this is simply one of those.
>>
>> Paul
>>
>
> That's great news. Thanks!
>
> Anthony
>

There is at least one caveat to using a .zip file for the standard library:
http://bugs.python.org/issue19081

Never change the .zip file in use by a running process.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Raspberry Pi Buildbot

[Gregory P. Smith]

On Fri, Jun 13, 2014 at 3:24 AM, Tal Einat  wrote:

> Is there one? If not, would you like me to set one up? I've got one at
> home with Raspbian installed not doing anything, it could easily run a
> buildslave.
>
> Poking around on buildbot.python.org/all/builders, I can only see one
> ARM buildbot[1], and it's just called "ARM v7".
>

The ARM v7 buildbot is mine. It's a Samsung chromebook with a dual core
exynos5 cpu and usb3 SSD. ie: It's *at least* 10x faster than a raspberry
pi.

I don't think a pi buildbot would add much value but if you want to run
one, feel free. It should live in the unstable pool.

-gps
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Question about PEP 484

[Gregory P. Smith]

On Mon, Jul 16, 2018 at 1:44 PM Guido van Rossum  wrote:

> As one of the authors of PEP 484, *I* never thought there was an ambiguity
> here. The intention was for stub files to conform to the same grammar as
> regular .py files, but with a different interpretation.
>
> > "Have the same syntax as regular Python modules" and "are files
> containing type hints" are at odds with each other.
>
> That depends. *same syntax as regular Python* is normative while
> *containing type hints* is an informal description of intent.
>
> I happen to be somewhat familiar with the situation that lead to this
> question -- pytype has its own parser for stub files that cannot parse all
> Python constructs. But claiming that PEP 484 is ambiguous feels wrong, and
> if we really need to clarify it the only way to go is to make "same syntax
> as regular Python" more clearly normative. Type checkers should of course
> feel free ignore everything they don't care about.
>

I feel like the "same syntax as regular Python" is too broad a statement.
That effectively requires a version specific Python interpreter to execute
the files.  With at least four different Python static analyzers in
existence today, keeping the behavior of all of them consistent is
important.  Otherwise pyi files will be (are being) created that are
analyzer specific and break other type checkers when distributed.

ex: We're encountering pyi files with conditional logic in them.  I believe
we've encountered pyi files with del statements in them?  Both of these are
a slippery slope towards being turing complete in something that isn't
supposed to be code.  I don't like this.  Interface declarations should not
contain logic.  If we allow conditions, we need to explicitly define what
we do allow in the PEP.  (if+else and del?  what inputs are allowed for the
expression in if statements?).  Otherwise at some point someone is going to
create a pyi file containing loops, function calls, and generator
expressions and expect it to _do_ something.  The horror!  Lets avoid that.

PEP-484 does contain the text, "This also reinforces the notion that no
runtime behavior should be expected of stub files." But reinforcing a
notion is not what I read as a concrete statement.

I'd rather see that say something like, "There must not be any runtime
behavior from a stub file. They will be parsed for information, not
executed."  Wordsmith that all you want, I'm not pedantic enough. :)
I expect someone pedantic to happily point out that a def or class or
assignment to ... with an annotation is runtime behavior... technically
correct, but that isn't how people like me think of them in this context.

We use a Pythonic syntax for stubs to be consistent with the language, that
doesn't mean they are code.

I wrote more than I thought I would here, I'll stop now. :)

-gps


>
> Regarding the unicode issue, that is indeed unfortunate, and there's a
> long but inconclusive discussion at
> https://github.com/python/typing/issues/208. (If you want a longer
> discussion here please start a new subject.)
>
> On Mon, Jul 16, 2018 at 1:02 PM, Adam Cataldo via Python-Dev <
> python-dev@python.org> wrote:
>
>> Thanks Brett and Teddy,
>>
>> Just so it doesn't get lost in the shuffle as folks dive into details,
>> I'll re-ask my earlier question about stub files. Assuming there is
>> consensus that there is ambiguity to resolve in the current definition, is
>> updating the section on stub files the preferred option? The only
>> alternative I can think of is to pull this out into a separate PEP. I
>> frankly have no opinion on what the best way to capture this is. We're
>> happy to help out either way.
>>
>> On Mon, Jul 16, 2018 at 12:48 PM Teddy Sudol  wrote:
>>
>>> Hi, my name is Teddy Sudol. I work with Adam and Rebecca on pytype.
>>>
>>> The explanation of stub files is unclear. The section you linked starts
>>> with, "Stub files are files containing type hints that are only for use by
>>> the type checker, not at runtime." According to
>>> https://www.python.org/dev/peps/pep-0484/#acceptable-type-hints, type
>>> hints may be classes, abstract base classes, types defined in the `types`
>>> and `typing` modules, type variables, type aliases and None. Further in the
>>> section you linked, PEP 484 also states, "Stub files have the same syntax
>>> as regular Python modules," and, "no runtime behavior should be expected of
>>> stub files."
>>>
>>> "Have the same syntax as regular Python modules" and "are files
>>> containing type hints" are at odds with each other. This has led to
>>> compatibility issues between Mypy and pytype. For example, `b''` is not a
>>> valid type annotation, but until a month ago, `codecs.pyi` in typeshed used
>>> exactly that:
>>> https://github.com/python/typeshed/commit/6bbf3d89eb9b6c3fd5b0c0f632b2ad9258cecf15#diff-5f6f48c425bc0c283784cf5277880c0cL95.
>>> If statements can be useful for things like version checks, but on the
>>> other hand, pyi files aren't supposed to have

Re: [Python-Dev] Finding Guido's replacement

[Gregory P. Smith]

On Tue, Jul 24, 2018 at 12:27 PM Abdur-Rahmaan Janhangeer <
arj.pyt...@gmail.com> wrote:

> not googler i mean google. they requested a change to a pep
>

"They"?  nah.  "Google LLC" did not request anything.  People who happen to
be working for Google on a Google owned project asked a question seeking
clarification / codification of some details.  There is nothing more to
read into that.  Adam was merely introducing himself by stating some
background info as he hasn't participated on the list much in the past.

-gps | destroyer of conspiracies

PS The number of core devs / committers who are Googlers has been high for
well over a decade (myself included).  As is true for many other large open
source friendly companies as well.  I believe Microsoft has the most
*active* committers employed at the moment.



>
> Abdur-Rahmaan Janhangeer
> https://github.com/Abdur-rahmaanJ
> Mauritius
>
> As the Googler who appeared on the mailing list, I can say this was just a
>> coincidence. I was a bit nervous no one would respond though, given Guido's
>> vacation :)
>>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com