[Python-Dev] Benchmark results across all major Python implementations

[Brett Cannon]

I gave the opening keynote at PyCon CA and then gave the same talk at
PyData NYC on the various interpreters of Python (Jupyter notebook of my
presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I
figured people here might find the benchmark numbers interesting so I'm
sharing the link here.

I'm still hoping someday speed.python.org becomes a thing so I never have
to spend so much time benchmarking so may Python implementations ever again
and this sort of thing is just part of what we do to keep the
implementation ecosystem healthy.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Benchmark results across all major Python implementations

[Maciej Fijalkowski]

Hi Brett

Any thoughts on improving the benchmark set (I think all of
{cpython,pypy,pyston} introduced new benchmarks to the set).
"speed.python.org" becoming a thing is generally stopped on "noone
cares enough to set it up".

Cheers,
fijal


On Mon, Nov 16, 2015 at 9:18 PM, Brett Cannon  wrote:
> I gave the opening keynote at PyCon CA and then gave the same talk at PyData
> NYC on the various interpreters of Python (Jupyter notebook of my
> presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I
> figured people here might find the benchmark numbers interesting so I'm
> sharing the link here.
>
> I'm still hoping someday speed.python.org becomes a thing so I never have to
> spend so much time benchmarking so may Python implementations ever again and
> this sort of thing is just part of what we do to keep the implementation
> ecosystem healthy.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Benchmark results across all major Python implementations

[R. David Murray]

On Mon, 16 Nov 2015 21:23:49 +0100, Maciej Fijalkowski  wrote:
> Any thoughts on improving the benchmark set (I think all of
> {cpython,pypy,pyston} introduced new benchmarks to the set).
> "speed.python.org" becoming a thing is generally stopped on "noone
> cares enough to set it up".

Actually, with some help from Intel, it is getting there.  You can see
the 'benchmarks' entry in the buildbot console:

http://buildbot.python.org/all/console

but it isn't quite working yet.  We are also waiting on a review of the
salt state:

https://github.com/python/psf-salt/pull/74

(All work done by Zach Ware.)

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Benchmark results across all major Python implementations

[Brett Cannon]

On Mon, 16 Nov 2015 at 12:24 Maciej Fijalkowski  wrote:

> Hi Brett
>
> Any thoughts on improving the benchmark set (I think all of
> {cpython,pypy,pyston} introduced new benchmarks to the set).
>

We should probably start a mailing list and finally hash out a common set
of benchmarks that we all agree are reasonable for measuring performance. I
think we all generally agree that high-level benchmarks are good and
micro-benchmarks aren't that important for cross-implementation comparisons
(obviously they have a usefulness when trying to work on a specific feature
set, but that should be considered specific to an implementation and not to
some globally accepted set of benchmarks). So we should identify benchmarks
which somewhat represent real world workloads and try to have a balanced
representation that doesn't lean one way or another (e.g., not all string
manipulation or scientific computing, but both should obviously be
represented).


> "speed.python.org" becoming a thing is generally stopped on "noone
> cares enough to set it up".
>

Oh, I know. I didn't say this could be considered wishful thinking since I
know I have enough on my plate to prevent me from making it happen.

-Brett


>
> Cheers,
> fijal
>
>
> On Mon, Nov 16, 2015 at 9:18 PM, Brett Cannon  wrote:
> > I gave the opening keynote at PyCon CA and then gave the same talk at
> PyData
> > NYC on the various interpreters of Python (Jupyter notebook of my
> > presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I
> > figured people here might find the benchmark numbers interesting so I'm
> > sharing the link here.
> >
> > I'm still hoping someday speed.python.org becomes a thing so I never
> have to
> > spend so much time benchmarking so may Python implementations ever again
> and
> > this sort of thing is just part of what we do to keep the implementation
> > ecosystem healthy.
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
> >
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Benchmark results across all major Python implementations

[Brian Curtin]

On Monday, November 16, 2015, Brett Cannon > wrote:

>
>
> On Mon, 16 Nov 2015 at 12:24 Maciej Fijalkowski  wrote:
>
>> Hi Brett
>>
>> Any thoughts on improving the benchmark set (I think all of
>> {cpython,pypy,pyston} introduced new benchmarks to the set).
>>
>
> We should probably start a mailing list
>

There is/was a sp...@python.org list.


> "speed.python.org" becoming a thing is generally stopped on "noone
>> cares enough to set it up".
>>
>
> Oh, I know. I didn't say this could be considered wishful thinking since I
> know I have enough on my plate to prevent me from making it happen.
>

There was a grant given years ago to improve some of this stuff but I don't
believe the work ever saw the light of day.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Benchmark results across all major Python implementations

[Brian Curtin]

On Monday, November 16, 2015, Brett Cannon > wrote:

>
>
> On Mon, 16 Nov 2015 at 12:24 Maciej Fijalkowski  wrote:
>
>> Hi Brett
>>
>> Any thoughts on improving the benchmark set (I think all of
>> {cpython,pypy,pyston} introduced new benchmarks to the set).
>>
>
> We should probably start a mailing list
>

There is/was a sp...@python.org list.


> "speed.python.org" becoming a thing is generally stopped on "noone
>> cares enough to set it up".
>>
>
> Oh, I know. I didn't say this could be considered wishful thinking since I
> know I have enough on my plate to prevent me from making it happen.
>

There was a grant given years ago to improve some of this stuff but I don't
believe the work ever saw the light of day.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Benchmark results across all major Python implementations

[Zachary Ware]

On Mon, Nov 16, 2015 at 3:38 PM, Brian Curtin  wrote:
> On Monday, November 16, 2015, Brett Cannon  wrote:

>>>
>>> Hi Brett
>>>
>>> Any thoughts on improving the benchmark set (I think all of
>>> {cpython,pypy,pyston} introduced new benchmarks to the set).
>>
>>
>> We should probably start a mailing list
>
>
> There is/was a sp...@python.org list.

Is, but seems to be heavily moderated without active moderation.  I
sent an offer to speed-owner this morning to help with moderation, but
as yet no response.  I know I have a couple of messages waiting in the
queue :)

On Mon, 16 Nov 2015 at 12:24 Maciej Fijalkowski  wrote:
> "speed.python.org" becoming a thing is generally stopped on "noone
> cares enough to set it up".

Setup is done in my dev environment, it's now blocking on people more
qualified than me to review and merge, then final tweaking of the
buildbot setup.  For those interested:

The part in most need of review is the changes to the PSF Salt
configuration to set up and run Codespeed on speed.python.org.  The
changes can be found in PR #74 on the psf-salt Github repo[0].

The Codespeed instance is housed in a Github repo owned by me[1]
currently.  There's one small patch to the codespeed code (which was
merged upstream this morning), the rest of the changes in my fork are
adapting a copy of the sample_project to be our own instance.  I've
been told that the grant proposal from long ago expected the use of
codespeed2 instead of codespeed.  I chose codespeed over codespeed2
because it appeared to be easier to get set up and running (which may
or may not have actually been true), but also because I've not seen
codespeed2 in action anywhere.

The final piece that could use review is the changes to our benchmark
repository, currently available in a sandbox repo on hg.python.org[2].
My original plan had been to use PyPy's benchmark suite, but as you
can tell from the logs of the existing buildslave, cpython doesn't run
that suite especially well, and the cpython suite has the advantage of
working with Python 2 and 3 out of the box.

Please have a look, give it a try if you can, and let me know what
needs improvement!

[0] https://github.com/python/psf-salt/pull/74
[1] https://github.com/zware/codespeed
[2] https://hg.python.org/sandbox/zware-benchmarks

-- 
Zach
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Benchmark results across all major Python implementations

[Tim Golden]

On 16/11/2015 22:23, Zachary Ware wrote:

On Mon, Nov 16, 2015 at 3:38 PM, Brian Curtin  wrote:

On Monday, November 16, 2015, Brett Cannon  wrote:




Hi Brett

Any thoughts on improving the benchmark set (I think all of
{cpython,pypy,pyston} introduced new benchmarks to the set).



We should probably start a mailing list



There is/was a sp...@python.org list.


Is, but seems to be heavily moderated without active moderation.  I
sent an offer to speed-owner this morning to help with moderation, but
as yet no response.  I know I have a couple of messages waiting in the
queue :)


Just to help things along, I've added myself as list owner and released 
a bunch of messages.


TJG
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Benchmark results across all major Python implementations

[Jim Baker]

Brett,

Very cool, I'm glad to see that Jython's performance was competitive under
most of these benchmarks. I would also be interested in joining the
proposed mailing list.

re elementtree - I assume the benchmarking is usually done with
cElementTree. However Jython currently lacks a Java equivalent, so
importing cElementTree just uses the pure Python version. Hence the
significant performance difference of approx. 40x for etree_parse and 16x
for etree_iterparse.

- Jim

On Mon, Nov 16, 2015 at 1:18 PM, Brett Cannon  wrote:

> I gave the opening keynote at PyCon CA and then gave the same talk at
> PyData NYC on the various interpreters of Python (Jupyter notebook of my
> presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I
> figured people here might find the benchmark numbers interesting so I'm
> sharing the link here.
>
> I'm still hoping someday speed.python.org becomes a thing so I never have
> to spend so much time benchmarking so may Python implementations ever again
> and this sort of thing is just part of what we do to keep the
> implementation ecosystem healthy.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Guido van Rossum]

So I dropped the ball on this too -- I was going to either have a look
or tell you quickly to find a BDFL-delegate, but ended up doing
neither. Skimming it now I really don't think I'm the right person to
review this. Maybe you could ask Alex Gaynor?

On Tue, Nov 10, 2015 at 4:47 PM, Nick Coghlan  wrote:
> Hi folks,
>
> I have a confession to make - I dropped the ball on the HTTPS
> verification backport proposals in PEP 493, and let the upstream and
> downstream approval processes get out of sequence.
>
> As a result, the RHEL 7.2 beta released back in September incorporates
> the HTTPS verification feature backport based on the current PEP 493
> draft, even though that hasn't formally been pronounced as an Active
> recommendation by python-dev yet.
>
> Accordingly, I'm belatedly submitting it for pronouncement now:
> https://www.python.org/dev/peps/pep-0493/
>
> There's currently no BDFL-Delegate assigned, so if Guido doesn't want
> to handle it, we'll need to address that question first.
>
> Our last discussion back in July seemed to show that folks either
> didn't care about the question (because they're using unmodified
> upstream versions so the PEP didn't affect them), or else thought the
> approach described in the PEP was reasonable, so I'm hoping the
> consequences of my mistake won't be too severe.
>
> Regards,
> Nick.
>
> P.S. I'm aware that this looks like presenting a fait accompli at a
> point where it's too late to realistically say "No", but the truth is
> that preparation for the Python in Education miniconf at PyCon
> Australia ramped up immediately after the July discussion, and then I
> personally got confused as to the scope of what was being included in
> 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493
> being deferred to a later release, but it's actually the whole package
> of 466+476+493). That's my fault for trying to keep track of too many
> things at once (and thus failing at some of them), not anyone else's.
>
> 
>
> PEP: 493
> Title: HTTPS verification recommendations for Python 2.7 redistributors
> Version: $Revision$
> Last-Modified: $Date$
> Author: Nick Coghlan ,
> Robert Kuska ,
> Marc-André Lemburg 
> Status: Draft
> Type: Informational
> Content-Type: text/x-rst
> Created: 10-May-2015
> Post-History: 06-Jul-2015
>
>
> Abstract
> 
>
> PEP 476 updated Python's default handling of HTTPS certificates to be
> appropriate for communication over the public internet. The Python 2.7 long
> term maintenance series was judged to be in scope for this change, with the
> new behaviour introduced in the Python 2.7.9 maintenance release.
>
> This PEP provides recommendations to downstream redistributors wishing to
> provide a smoother migration experience when helping their users to manage
> this change in Python's default behaviour.
>
>
> Rationale
> =
>
> PEP 476 changed Python's default behaviour to better match the needs and
> expectations of developers operating over the public internet, a category
> which appears to include most new Python developers. It is the position of
> the authors of this PEP that this was a correct decision.
>
> However, it is also the case that this change *does* cause problems for
> infrastructure administrators operating private intranets that rely on
> self-signed certificates, or otherwise encounter problems with the new default
> certificate verification settings.
>
> The long term answer for such environments is to update their internal
> certificate management to at least match the standards set by the public
> internet, but in the meantime, it is desirable to offer these administrators
> a way to continue receiving maintenance updates to the Python 2.7 series,
> without having to gate that on upgrades to their certificate management
> infrastructure.
>
> PEP 476 did attempt to address this question, by covering how to revert the
> new settings process wide by monkeypatching the ``ssl`` module to restore the
> old behaviour. Unfortunately, the ``sitecustomize.py`` based technique 
> proposed
> to allow system administrators to disable the feature by default in their
> Standard Operating Environment definition has been determined to be
> insufficient in at least some cases. The specific case of interest to the
> authors of this PEP is the one where a Linux distributor aims to provide
> their users with a
> `smoother migration path
> `__
> than the standard one provided by consuming upstream CPython 2.7 releases
> directly, but other potential challenges have also been pointed out with
> updating embedded Python runtimes and other user level installations of 
> Python.
>
> Rather than allowing a plethora of mutually incompatibile migration techniques
> to bloom, this PEP proposes two alternative approaches that redistributors
> may take when addressing these problems. Redistributors may choose to 
> implement
> one

Re: [Python-Dev] Benchmark results across all major Python implementations

[Stewart, David C]

Last June we started publishing a daily performance report of the latest Python 
tip against the previous day's run and some established synch point. We mail 
these to the community to act as a "canary in the coal mine." I wrote about it 
at https://01.org/lp/blog/0-day-challenge-what-pulse-internet

You can see our manager-style dashboard of a couple of key workloads at 
http://languagesperformance.intel.com/
(I have this running constantly on a dedicated screen in my office).

Would love to get better workloads if we can.

Dave

From: Python-Dev on behalf of Brett Cannon
Date: Monday, November 16, 2015 at 12:18 PM
To: python-dev
Subject: [Python-Dev] Benchmark results across all major Python implementations

I gave the opening keynote at PyCon CA and then gave the same talk at PyData 
NYC on the various interpreters of Python (Jupyter notebook of my presentation 
can be found at bit.ly/pycon-ca-keynote; no 
video yet). I figured people here might find the benchmark numbers interesting 
so I'm sharing the link here.

I'm still hoping someday speed.python.org becomes a 
thing so I never have to spend so much time benchmarking so may Python 
implementations ever again and this sort of thing is just part of what we do to 
keep the implementation ecosystem healthy.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Nick Coghlan]

On 17 November 2015 at 09:07, Guido van Rossum  wrote:
> So I dropped the ball on this too -- I was going to either have a look
> or tell you quickly to find a BDFL-delegate, but ended up doing
> neither. Skimming it now I really don't think I'm the right person to
> review this. Maybe you could ask Alex Gaynor?

If Alex is interested, that could definitely work. I've also asked
Christian Heimes if he might be interested in handling it, as he
worked on several of the SSL module changes that were backported, and
he also works on FreeIPA [1] now, which will potentially be affected
by the changes to RHEL & CentOS 7.

Regards,
Nick.

[1] FreeIPA is an open source Identity Management & Authentication
system: http://www.freeipa.org/

>
> On Tue, Nov 10, 2015 at 4:47 PM, Nick Coghlan  wrote:
>> Hi folks,
>>
>> I have a confession to make - I dropped the ball on the HTTPS
>> verification backport proposals in PEP 493, and let the upstream and
>> downstream approval processes get out of sequence.
>>
>> As a result, the RHEL 7.2 beta released back in September incorporates
>> the HTTPS verification feature backport based on the current PEP 493
>> draft, even though that hasn't formally been pronounced as an Active
>> recommendation by python-dev yet.
>>
>> Accordingly, I'm belatedly submitting it for pronouncement now:
>> https://www.python.org/dev/peps/pep-0493/
>>
>> There's currently no BDFL-Delegate assigned, so if Guido doesn't want
>> to handle it, we'll need to address that question first.
>>
>> Our last discussion back in July seemed to show that folks either
>> didn't care about the question (because they're using unmodified
>> upstream versions so the PEP didn't affect them), or else thought the
>> approach described in the PEP was reasonable, so I'm hoping the
>> consequences of my mistake won't be too severe.
>>
>> Regards,
>> Nick.
>>
>> P.S. I'm aware that this looks like presenting a fait accompli at a
>> point where it's too late to realistically say "No", but the truth is
>> that preparation for the Python in Education miniconf at PyCon
>> Australia ramped up immediately after the July discussion, and then I
>> personally got confused as to the scope of what was being included in
>> 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493
>> being deferred to a later release, but it's actually the whole package
>> of 466+476+493). That's my fault for trying to keep track of too many
>> things at once (and thus failing at some of them), not anyone else's.
>>
>> 
>>
>> PEP: 493
>> Title: HTTPS verification recommendations for Python 2.7 redistributors
>> Version: $Revision$
>> Last-Modified: $Date$
>> Author: Nick Coghlan ,
>> Robert Kuska ,
>> Marc-André Lemburg 
>> Status: Draft
>> Type: Informational
>> Content-Type: text/x-rst
>> Created: 10-May-2015
>> Post-History: 06-Jul-2015
>>
>>
>> Abstract
>> 
>>
>> PEP 476 updated Python's default handling of HTTPS certificates to be
>> appropriate for communication over the public internet. The Python 2.7 long
>> term maintenance series was judged to be in scope for this change, with the
>> new behaviour introduced in the Python 2.7.9 maintenance release.
>>
>> This PEP provides recommendations to downstream redistributors wishing to
>> provide a smoother migration experience when helping their users to manage
>> this change in Python's default behaviour.
>>
>>
>> Rationale
>> =
>>
>> PEP 476 changed Python's default behaviour to better match the needs and
>> expectations of developers operating over the public internet, a category
>> which appears to include most new Python developers. It is the position of
>> the authors of this PEP that this was a correct decision.
>>
>> However, it is also the case that this change *does* cause problems for
>> infrastructure administrators operating private intranets that rely on
>> self-signed certificates, or otherwise encounter problems with the new 
>> default
>> certificate verification settings.
>>
>> The long term answer for such environments is to update their internal
>> certificate management to at least match the standards set by the public
>> internet, but in the meantime, it is desirable to offer these administrators
>> a way to continue receiving maintenance updates to the Python 2.7 series,
>> without having to gate that on upgrades to their certificate management
>> infrastructure.
>>
>> PEP 476 did attempt to address this question, by covering how to revert the
>> new settings process wide by monkeypatching the ``ssl`` module to restore the
>> old behaviour. Unfortunately, the ``sitecustomize.py`` based technique 
>> proposed
>> to allow system administrators to disable the feature by default in their
>> Standard Operating Environment definition has been determined to be
>> insufficient in at least some cases. The specific case of interest to the
>> authors of this PEP is the one where a Linux distributor aims to provide
>> the

Re: [Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Guido van Rossum]

Hm, making Christian the BDFL-delegate would mean two out of three
authors *and* the BDFL-delegate all working for Red Hat, which clearly
has a stake (and IIUC has already committed to this approach ahead of
PEP approval). SO then it would look like this is just rubber-stamping
Red Hat's internal decision process (if it's a process -- sounds more
like an accident :-).

So, Alex, do you want to approve this PEP?

On Mon, Nov 16, 2015 at 3:54 PM, Nick Coghlan  wrote:
> On 17 November 2015 at 09:07, Guido van Rossum  wrote:
>> So I dropped the ball on this too -- I was going to either have a look
>> or tell you quickly to find a BDFL-delegate, but ended up doing
>> neither. Skimming it now I really don't think I'm the right person to
>> review this. Maybe you could ask Alex Gaynor?
>
> If Alex is interested, that could definitely work. I've also asked
> Christian Heimes if he might be interested in handling it, as he
> worked on several of the SSL module changes that were backported, and
> he also works on FreeIPA [1] now, which will potentially be affected
> by the changes to RHEL & CentOS 7.
>
> Regards,
> Nick.
>
> [1] FreeIPA is an open source Identity Management & Authentication
> system: http://www.freeipa.org/
>
>>
>> On Tue, Nov 10, 2015 at 4:47 PM, Nick Coghlan  wrote:
>>> Hi folks,
>>>
>>> I have a confession to make - I dropped the ball on the HTTPS
>>> verification backport proposals in PEP 493, and let the upstream and
>>> downstream approval processes get out of sequence.
>>>
>>> As a result, the RHEL 7.2 beta released back in September incorporates
>>> the HTTPS verification feature backport based on the current PEP 493
>>> draft, even though that hasn't formally been pronounced as an Active
>>> recommendation by python-dev yet.
>>>
>>> Accordingly, I'm belatedly submitting it for pronouncement now:
>>> https://www.python.org/dev/peps/pep-0493/
>>>
>>> There's currently no BDFL-Delegate assigned, so if Guido doesn't want
>>> to handle it, we'll need to address that question first.
>>>
>>> Our last discussion back in July seemed to show that folks either
>>> didn't care about the question (because they're using unmodified
>>> upstream versions so the PEP didn't affect them), or else thought the
>>> approach described in the PEP was reasonable, so I'm hoping the
>>> consequences of my mistake won't be too severe.
>>>
>>> Regards,
>>> Nick.
>>>
>>> P.S. I'm aware that this looks like presenting a fait accompli at a
>>> point where it's too late to realistically say "No", but the truth is
>>> that preparation for the Python in Education miniconf at PyCon
>>> Australia ramped up immediately after the July discussion, and then I
>>> personally got confused as to the scope of what was being included in
>>> 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493
>>> being deferred to a later release, but it's actually the whole package
>>> of 466+476+493). That's my fault for trying to keep track of too many
>>> things at once (and thus failing at some of them), not anyone else's.
>>>
>>> 
>>>
>>> PEP: 493
>>> Title: HTTPS verification recommendations for Python 2.7 redistributors
>>> Version: $Revision$
>>> Last-Modified: $Date$
>>> Author: Nick Coghlan ,
>>> Robert Kuska ,
>>> Marc-André Lemburg 
>>> Status: Draft
>>> Type: Informational
>>> Content-Type: text/x-rst
>>> Created: 10-May-2015
>>> Post-History: 06-Jul-2015
>>>
>>>
>>> Abstract
>>> 
>>>
>>> PEP 476 updated Python's default handling of HTTPS certificates to be
>>> appropriate for communication over the public internet. The Python 2.7 long
>>> term maintenance series was judged to be in scope for this change, with the
>>> new behaviour introduced in the Python 2.7.9 maintenance release.
>>>
>>> This PEP provides recommendations to downstream redistributors wishing to
>>> provide a smoother migration experience when helping their users to manage
>>> this change in Python's default behaviour.
>>>
>>>
>>> Rationale
>>> =
>>>
>>> PEP 476 changed Python's default behaviour to better match the needs and
>>> expectations of developers operating over the public internet, a category
>>> which appears to include most new Python developers. It is the position of
>>> the authors of this PEP that this was a correct decision.
>>>
>>> However, it is also the case that this change *does* cause problems for
>>> infrastructure administrators operating private intranets that rely on
>>> self-signed certificates, or otherwise encounter problems with the new 
>>> default
>>> certificate verification settings.
>>>
>>> The long term answer for such environments is to update their internal
>>> certificate management to at least match the standards set by the public
>>> internet, but in the meantime, it is desirable to offer these administrators
>>> a way to continue receiving maintenance updates to the Python 2.7 series,
>>> without having to gate that on upgrades to their certificate man

[Python-Dev] Reading Python source file

[Serhiy Storchaka]

I'm working on rewriting Python tokenizer (in particular the part that 
reads and decodes Python source file). The code is complicated. For now 
there are such cases:


* Reading from the string in memory.
* Interactive reading from the file.
* Reading from the file:
  - Raw reading ignoring encoding in parser generator.
  - Raw reading UTF-8 encoded file.
  - Reading and recoding to UTF-8.

The file is read by the line. It makes hard to check correctness of the 
first line if the encoding is specified in the second line. And it makes 
very hard problems with null bytes and with desynchronizing buffered C 
and Python files. All this problems can be easily solved if read all 
Python source file in memory and then parse it as string. This would 
allow to drop a large complex and buggy part of code.


Are there disadvantages in this solution? As for memory consumption, the 
source text itself will consume only small part of the memory consumed 
by AST tree and other structures. As for performance, reading and 
decoding all file can be faster then by the line.


[1] http://bugs.python.org/issue25643

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Guido van Rossum]

If you free the memory used for the source buffer before starting code
generation you should be good.

On Mon, Nov 16, 2015 at 5:53 PM, Serhiy Storchaka  wrote:
> I'm working on rewriting Python tokenizer (in particular the part that reads
> and decodes Python source file). The code is complicated. For now there are
> such cases:
>
> * Reading from the string in memory.
> * Interactive reading from the file.
> * Reading from the file:
>   - Raw reading ignoring encoding in parser generator.
>   - Raw reading UTF-8 encoded file.
>   - Reading and recoding to UTF-8.
>
> The file is read by the line. It makes hard to check correctness of the
> first line if the encoding is specified in the second line. And it makes
> very hard problems with null bytes and with desynchronizing buffered C and
> Python files. All this problems can be easily solved if read all Python
> source file in memory and then parse it as string. This would allow to drop
> a large complex and buggy part of code.
>
> Are there disadvantages in this solution? As for memory consumption, the
> source text itself will consume only small part of the memory consumed by
> AST tree and other structures. As for performance, reading and decoding all
> file can be faster then by the line.
>
> [1] http://bugs.python.org/issue25643
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[MRAB]

On 2015-11-17 01:53, Serhiy Storchaka wrote:

I'm working on rewriting Python tokenizer (in particular the part that
reads and decodes Python source file). The code is complicated. For now
there are such cases:

* Reading from the string in memory.
* Interactive reading from the file.
* Reading from the file:
- Raw reading ignoring encoding in parser generator.
- Raw reading UTF-8 encoded file.
- Reading and recoding to UTF-8.

The file is read by the line. It makes hard to check correctness of the
first line if the encoding is specified in the second line. And it makes
very hard problems with null bytes and with desynchronizing buffered C
and Python files. All this problems can be easily solved if read all
Python source file in memory and then parse it as string. This would
allow to drop a large complex and buggy part of code.

Are there disadvantages in this solution? As for memory consumption, the
source text itself will consume only small part of the memory consumed
by AST tree and other structures. As for performance, reading and
decoding all file can be faster then by the line.

[1] http://bugs.python.org/issue25643

As I understand it, *nix expects the shebang to be b'#!', which means 
that the
first line should be ASCII-compatible (it's possible that the UTF-8 BOM 
might

be present). This kind of suggests that encodings like UTF-16 would cause a
problem on such systems.

The encoding line also needs to be ASCII-compatible.

I believe that the recent thread "Support of UTF-16 and UTF-32 source
encodings" also concluded that UTF-16 and UTF-32 shouldn't be supported.

This means that you could treat the first 2 lines as though they were some
kind of extended ASCII (Latin-1?), the line ending being '\n' or '\r' or
'\r\n'.

Once you'd identify the encoding, you could decode everything (including the
shebang line) using that encoding.

(What should happen if the encoding line then decoded differently, i.e.
encoding_line.decode(encoding) != encoding_line.decode('latin-1')?)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reading Python source file

[Random832]

MRAB  writes:
> As I understand it, *nix expects the shebang to be b'#!',
> which means that the first line should be ASCII-compatible
> (it's possible that the UTF-8 BOM might be present).

The UTF-8 BOM interferes with it on Mac OSX and Linux, at
least.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] Daily reference leaks (97e2a6810f7f): sum=10

[Brett Cannon]

Just an FYI there seems to be a consistent, minor refcount leak found by
test_capi that has been there for what seems like a couple of weeks.

On Mon, 16 Nov 2015 at 00:42  wrote:

> results for 97e2a6810f7f on branch "default"
> 
>
> test_asyncio leaked [0, 0, 3] memory blocks, sum=3
> test_capi leaked [1, 1, 1] references, sum=3
> test_functools leaked [0, 2, 2] memory blocks, sum=4
>
>
> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
> '3:3:/home/psf-users/antoine/refleaks/reflogBLsY2a', '--timeout', '7200']
> ___
> Python-checkins mailing list
> python-check...@python.org
> https://mail.python.org/mailman/listinfo/python-checkins
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com