Re: [Python-Dev] C code: %s vs %U

[Serhiy Storchaka]

26.03.14 03:43, Ethan Furman написав(ла):

%s is a string.

%U is unicode?

If so, then %s should only be used when it is certain the string in
question has no unicode in it?


%s is UTF-8 encoded string.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Victor Stinner]

2014-03-25 23:37 GMT+01:00 Ethan Furman :
> ``%a`` will call ``ascii()`` on the interpolated value.

I'm not sure that I understood correctly: is the "%a" format
supported? The result of ascii() is a Unicode string. Does it mean
that ("%a" % obj) should give the same result than
ascii(obj).encode('ascii', 'strict')?

Would it be possible to add a table or list to summarize supported
format characters? I found:

- single byte: %c
- integer: %d, %u, %i, %o, %x, %X, %f, %g, "etc." (can you please
complete "etc." ?)
- bytes and __bytes__ method: %s
- ascii(): %a


I guess that the implementation of %a can avoid a conversion from
ASCII ("PyUnicode_DecodeASCII" in the following code) and then a
conversion to ASCII again (in bytes%args):

PyObject *
PyObject_ASCII(PyObject *v)
{
PyObject *repr, *ascii, *res;

repr = PyObject_Repr(v);
if (repr == NULL)
return NULL;

if (PyUnicode_IS_ASCII(repr))
return repr;

/* repr is guaranteed to be a PyUnicode object by PyObject_Repr */
ascii = _PyUnicode_AsASCIIString(repr, "backslashreplace");
Py_DECREF(repr);
if (ascii == NULL)
return NULL;

res = PyUnicode_DecodeASCII(   < HERE
PyBytes_AS_STRING(ascii),
PyBytes_GET_SIZE(ascii),
NULL);

Py_DECREF(ascii);
return res;
}

>  This is intended
> as a debugging aid, rather than something that should be used in production.

I don't understand the purpose of this sentence. Does it mean that %a
must not be used? IMO this sentence can be removed.

> Non-ASCII values will be encoded to either ``\xnn`` or ``\u``
> representation.

Unicode is larger than that! print(ascii(chr(0x10))) => '\U0010'

> Use cases include developing a new protocol and writing
> landmarks into the stream; debugging data going into an existing protocol
> to see if the problem is the protocol itself or bad data; a fall-back for a
> serialization format; or even a rudimentary serialization format when
> defining ``__bytes__`` would not be appropriate [8].

I understand the debug use case. I'm not convinced by the serialization idea :-)

> .. note::
>
> If a ``str`` is passed into ``%a``, it will be surrounded by quotes.

And:

- bytes gets a "b" prefix and surrounded by quotes as well  (b'...')
- the quote ' is escaped as \' if the string contains quotes ' and "

Can you also please add examples for %a?

>>> b"%a" % 123
b'123'
>>> b"%s" % ascii(b"bytes")
b"b'bytes'"
>>> b"%s" % "text"   # hum, it's not easy to see surrounding quotes with this 
>>> examples
b"'text'"

The following more complex examples are maybe not needed:

>>> b"%a" % "euro:€"
b"'euro:\\u20ac'"
>>> b"%a" % """quotes >'"<"""
b'\'quotes >\\\'"<\''

> Proposed variations
> ===
>

It would be fair to mention also a whole different PEP, Antoine's PEP 460!

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C code: %s vs %U

[Antoine Pitrou]

On Tue, 25 Mar 2014 18:43:30 -0700
Ethan Furman  wrote:
> %s is a string.
> 
> %U is unicode?

What is the context?



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C code: %s vs %U

[Antoine Pitrou]

On Wed, 26 Mar 2014 11:48:44 +0100
Antoine Pitrou  wrote:
> On Tue, 25 Mar 2014 18:43:30 -0700
> Ethan Furman  wrote:
> > %s is a string.
> > 
> > %U is unicode?
> 
> What is the context?

Ok, I suppose it's PyUnicode_Format? :-)
(I was initially thinking PyArg_ParseTuple, but it doesn't use "%")

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C code: %s vs %U

[Victor Stinner]

2014-03-26 12:02 GMT+01:00 Antoine Pitrou :
> Ok, I suppose it's PyUnicode_Format? :-)

I don't think that PyUnicode_Format supports %U.

For PyUnicode_FromFormatV(), %U expects a Python Unicode object,
whereas "%s" expects a ASCII (or latin1?) encoded byte string "const
char*".

For PyArg_ParseTuple, "s" accepts str and bytes and encodes str to
UTF-8, whereas "U" expects str.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen

[Victor Stinner]

Hi,

For your information, asyncio.subprocess.Process is limited. It's not
possible yet to connect pipes between two processes. Something like
"cat | wc -l" where the cat stdin comes from Python.

It's possible to enhance the API to implement that, but the timeframe
was too short to implement it before Python 3.4.

Victor

2014-03-25 23:19 GMT+01:00 Antoine Pitrou :
>
> Hi,
>
> On core-mentorship someone asked about PEP 3145 - Asynchronous I/O for
> subprocess.popen.  I answered that asyncio now has subprocess support
> (including non-blocking I/O on the three standard stream pipes), so
> it's not obvious anything else is needed.
>
> Should we change the PEP's status to Rejected or Superseded?
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7

[Nick Coghlan]

Guido and Antoine persuaded me that selective backports would be a
better idea for the network security enhancements than the wholesale
module backports previously suggested, while Alex and Donald provided
the necessary additional details, so here's a revised version of the
PEP. Despite making it more explicit, I deleted more lines than I
added, strongly suggesting that switching to selective backports was
the right call :)

I dealt with the SSL module the way Donald suggested: excluding the
RAND_* functions, rather than listing everything else.

I also changed the headings to make it clear the listed alternatives
were rejected ideas, made the footnotes a bit more readable, and
tidied up the wording in a few places.

Diff: http://hg.python.org/peps/rev/8527f6e2beb0
Web: http://www.python.org/dev/peps/pep-0466/

==
PEP: 466
Title: Network Security Enhancement Exception for Python 2.7
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan ,
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 23-Mar-2014
Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014


Abstract


Most CPython tracker issues are classified as errors in behaviour or
proposed enhancements. Most patches to fix behavioural errors are
applied to all active maintenance branches.  Enhancement patches are
restricted to the default branch that becomes the next Python version.

This cadence works reasonably well during Python's normal 18-24 month
feature release cycle, which is still applicable to the Python 3 series.
However, the age of the standard library in Python 2 has now reached a point
where it is sufficiently far behind the state of the art in network security
protocols for it to be causing real problems in use cases where upgrading to
Python 3 in the near term may not be feasible.

In recognition of the additional practical considerations that have arisen
during the 4+ year maintenance cycle for Python 2.7, this PEP allows
Python 2.7 standard library components that have implications for the
overall security of the internet to be updated in line with the
corresponding Python 3 feature releases.

Specifically, the exception allows a critical set of network security
related features to be backported from Python 3.4 to the upcoming Python
2.7.7 maintenance release.

While this PEP does not make any changes to the core development team's
handling of security-fix-only branches that are no longer in active
maintenance, it *does* recommend that commercial redistributors providing
extended support periods for the Python standard library either backport
these features to their supported versions, or else explicitly disclaim
support for the use of older versions in roles that involve connecting
directly to the public internet.


Exemption Policy


Under this policy, the following features SHOULD be backported from Python
3.4 to the upcoming Python 2.7.7 maintenance release:

* in the ``os`` module:

  * persistent file descriptor for ``os.urandom()``.

* in the ``hmac`` module:

  * constant time comparison function (``hmac.compare_digest()``).

* in the ``hashlib`` module:

  * password hashing function (``hashlib.pbkdf2_hmac()``).
  * details of hash algorithm availability (``hashlib.algorithms_guaranteed``
and ``hashlib.algorithms_guaranteed``).

* in the ``ssl`` module:

  * this module is almost entirely synchronised with its Python 3
counterpart, bringing TLSv2, SSLContext manipulation, Server Name
Identification, access to platform certificate stores, standard
library support for peer hostname validation and more to the Python 2
series.
  * the only ``ssl`` module features *not* backported under this policy are
the ``ssl.RAND_*`` functions that provide access to OpenSSL's random
number generation capabilities - use ``os.urandom()`` instead.

As part of this policy, permission is also granted to upgrade to newer
feature releases of OpenSSL when preparing the binary installers
for new maintenance releases of Python 2.7.


Backwards Compatibility Considerations
==

As in the Python 3 series, the backported ``ssl.create_default_context()``
API is granted a backwards compatibility exemption that permits the
protocol, options, cipher and other settings of the created SSL context to
be made

This PEP does *not* grant any exemptions to the usual backwards
compatibility policy for maintenance releases. Instead, by explicitly
encouraging the use of feature based checks, it is designed to make it easier
to write more secure cross-version compatible Python software, while still
limiting the risk of breaking currently working software when upgrading to
a new Python 2.7 maintenance release.

In all cases where this policy allows new features to be backported to
the Python 2.7 release series, it is possible to write cross-version
compatible code that operates by "feature detection" (for example, checking
for p

Re: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7

[Donald Stufft]

Typo I think:

As in the Python 3 series, the backported ssl.create_default_context() API
is granted a backwards compatibility exemption that permits the protocol,
options, cipher and other settings of the created SSL context to be made

made what?

On Mar 26, 2014, at 8:00 AM, Nick Coghlan  wrote:

> Guido and Antoine persuaded me that selective backports would be a
> better idea for the network security enhancements than the wholesale
> module backports previously suggested, while Alex and Donald provided
> the necessary additional details, so here's a revised version of the
> PEP. Despite making it more explicit, I deleted more lines than I
> added, strongly suggesting that switching to selective backports was
> the right call :)
> 
> I dealt with the SSL module the way Donald suggested: excluding the
> RAND_* functions, rather than listing everything else.
> 
> I also changed the headings to make it clear the listed alternatives
> were rejected ideas, made the footnotes a bit more readable, and
> tidied up the wording in a few places.
> 
> Diff: http://hg.python.org/peps/rev/8527f6e2beb0
> Web: http://www.python.org/dev/peps/pep-0466/
> 
> ==
> PEP: 466
> Title: Network Security Enhancement Exception for Python 2.7
> Version: $Revision$
> Last-Modified: $Date$
> Author: Nick Coghlan ,
> Status: Draft
> Type: Informational
> Content-Type: text/x-rst
> Created: 23-Mar-2014
> Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014
> 
> 
> Abstract
> 
> 
> Most CPython tracker issues are classified as errors in behaviour or
> proposed enhancements. Most patches to fix behavioural errors are
> applied to all active maintenance branches.  Enhancement patches are
> restricted to the default branch that becomes the next Python version.
> 
> This cadence works reasonably well during Python's normal 18-24 month
> feature release cycle, which is still applicable to the Python 3 series.
> However, the age of the standard library in Python 2 has now reached a point
> where it is sufficiently far behind the state of the art in network security
> protocols for it to be causing real problems in use cases where upgrading to
> Python 3 in the near term may not be feasible.
> 
> In recognition of the additional practical considerations that have arisen
> during the 4+ year maintenance cycle for Python 2.7, this PEP allows
> Python 2.7 standard library components that have implications for the
> overall security of the internet to be updated in line with the
> corresponding Python 3 feature releases.
> 
> Specifically, the exception allows a critical set of network security
> related features to be backported from Python 3.4 to the upcoming Python
> 2.7.7 maintenance release.
> 
> While this PEP does not make any changes to the core development team's
> handling of security-fix-only branches that are no longer in active
> maintenance, it *does* recommend that commercial redistributors providing
> extended support periods for the Python standard library either backport
> these features to their supported versions, or else explicitly disclaim
> support for the use of older versions in roles that involve connecting
> directly to the public internet.
> 
> 
> Exemption Policy
> 
> 
> Under this policy, the following features SHOULD be backported from Python
> 3.4 to the upcoming Python 2.7.7 maintenance release:
> 
> * in the ``os`` module:
> 
>  * persistent file descriptor for ``os.urandom()``.
> 
> * in the ``hmac`` module:
> 
>  * constant time comparison function (``hmac.compare_digest()``).
> 
> * in the ``hashlib`` module:
> 
>  * password hashing function (``hashlib.pbkdf2_hmac()``).
>  * details of hash algorithm availability (``hashlib.algorithms_guaranteed``
>and ``hashlib.algorithms_guaranteed``).
> 
> * in the ``ssl`` module:
> 
>  * this module is almost entirely synchronised with its Python 3
>counterpart, bringing TLSv2, SSLContext manipulation, Server Name
>Identification, access to platform certificate stores, standard
>library support for peer hostname validation and more to the Python 2
>series.
>  * the only ``ssl`` module features *not* backported under this policy are
>the ``ssl.RAND_*`` functions that provide access to OpenSSL's random
>number generation capabilities - use ``os.urandom()`` instead.
> 
> As part of this policy, permission is also granted to upgrade to newer
> feature releases of OpenSSL when preparing the binary installers
> for new maintenance releases of Python 2.7.
> 
> 
> Backwards Compatibility Considerations
> ==
> 
> As in the Python 3 series, the backported ``ssl.create_default_context()``
> API is granted a backwards compatibility exemption that permits the
> protocol, options, cipher and other settings of the created SSL context to
> be made
> 
> This PEP does *not* grant any exemptions to the usual backwards
> compatibility policy for maintenance releases. I

Re: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7

[Nick Coghlan]

On 26 March 2014 22:05, Donald Stufft  wrote:
> Typo I think:
>
> As in the Python 3 series, the backported ssl.create_default_context() API
> is granted a backwards compatibility exemption that permits the protocol,
> options, cipher and other settings of the created SSL context to be made
>
> made what?

Incomplete edit that I didn't notice until after posting. Fixed in the
web version now, along with the copy & paste error in the list of
hashlib attributes to be backported.

The first two backwards compatibility paragraphs:

===
As in the Python 3 series, the backported ssl.create_default_context()
API is granted a backwards compatibility exemption that permits the
protocol, options, cipher and other settings of the created SSL
context to be updated in maintenance releases to use higher default
security settings. This allows them to appropriately balance
compatibility and security at the time of the maintenance release,
rather than at the time of the original feature release.

This PEP does not grant any other exemptions to the usual backwards
compatibility policy for maintenance releases. Instead, by explicitly
encouraging the use of feature based checks, it is designed to make it
easier to write more secure cross-version compatible Python software,
while still limiting the risk of breaking currently working software
when upgrading to a new Python 2.7 maintenance release.
===

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7

[Donald Stufft]

On Mar 26, 2014, at 8:00 AM, Nick Coghlan  wrote:

> Guido and Antoine persuaded me that selective backports would be a
> better idea for the network security enhancements than the wholesale
> module backports previously suggested, while Alex and Donald provided
> the necessary additional details, so here's a revised version of the
> PEP. Despite making it more explicit, I deleted more lines than I
> added, strongly suggesting that switching to selective backports was
> the right call :)
> 
> I dealt with the SSL module the way Donald suggested: excluding the
> RAND_* functions, rather than listing everything else.
> 
> I also changed the headings to make it clear the listed alternatives
> were rejected ideas, made the footnotes a bit more readable, and
> tidied up the wording in a few places.
> 
> Diff: http://hg.python.org/peps/rev/8527f6e2beb0
> Web: http://www.python.org/dev/peps/pep-0466/
> 
> ==
> PEP: 466
> Title: Network Security Enhancement Exception for Python 2.7
> Version: $Revision$
> Last-Modified: $Date$
> Author: Nick Coghlan ,
> Status: Draft
> Type: Informational
> Content-Type: text/x-rst
> Created: 23-Mar-2014
> Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014
> 
> 

This looks reasonable to me still and still solves the major problems that 
trying to securely
use the 2.7 series has.

+1 From me.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C code: %s vs %U

[Martin v. Löwis]

[Assuming you are talking about PyUnicode_FromFormatV]
> %s is a string.

No. %s is a char*; C does not have a "string" type.
The string behind the pointer should be UTF-8 encoded;
other encodings are tolerated through the "replace" error
handler.

> %U is unicode?

No. This is a PyObject* whose Python type is 'str'
(i.e. an object for which PyUnicode_Check succeeds)

> If so, then %s should only be used when it is certain the string in
> question has no unicode in it?

No. If you have a char*, use %s; using %U would crash.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C code: %s vs %U

[Ethan Furman]

On 03/26/2014 03:48 AM, Antoine Pitrou wrote:

On Tue, 25 Mar 2014 18:43:30 -0700
Ethan Furman  wrote:

%s is a string.

%U is unicode?


What is the context?


A patch I'm adapting has these lines:

  static PyObject*
  module_getattr(PyObject *m, PyObject *name)
  {
  char *mod_name_str;
  ...
  PyErr_Format(PyExc_AttributeError,
  "module '%s' has no attribute '%U'", mod_name_str, name);

So it looks like %s is referring to a simple string, and %U is referring to a 
PyObject, but I was hoping for verification.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASED] Python 3.4.0

[Daniel Pimentel (d4n1)]

Good job men :D


2014-03-17 14:18 GMT-03:00 Alioune Dia :

> yeah , asyncio is a great module, congrat for all jobs you are doing
> --Ad | Dakar
>
>
> 2014-03-17 18:11 GMT+01:00 Giampaolo Rodola' :
>
>> The what's new looks truly amazing, with pathlib and asyncio being my
>> favourite additions.
>> Thanks for all the hard work.
>>
>>
>> On Mon, Mar 17, 2014 at 5:57 PM, Ryan Gonzalez  wrote:
>>
>>> YES!!! +1 to the authors of the statistics and pathlib modules.
>>>
>>>
>>> On Mon, Mar 17, 2014 at 1:29 AM, Larry Hastings wrote:
>>>

 On behalf of the Python development team, I'm thrilled to announce
 the official release of Python 3.4.


 Python 3.4 includes a range of improvements of the 3.x series, including
 hundreds of small improvements and bug fixes.  Major new features and
 changes in the 3.4 release series include:

 * PEP 428, a "pathlib" module providing object-oriented filesystem paths
 * PEP 435, a standardized "enum" module
 * PEP 436, a build enhancement that will help generate introspection
information for builtins
 * PEP 442, improved semantics for object finalization
 * PEP 443, adding single-dispatch generic functions to the standard
 library
 * PEP 445, a new C API for implementing custom memory allocators
 * PEP 446, changing file descriptors to not be inherited by default
in subprocesses
 * PEP 450, a new "statistics" module
 * PEP 451, standardizing module metadata for Python's module import
 system
 * PEP 453, a bundled installer for the *pip* package manager
 * PEP 454, a new "tracemalloc" module for tracing Python memory
 allocations
 * PEP 456, a new hash algorithm for Python strings and binary data
 * PEP 3154, a new and improved protocol for pickled objects
 * PEP 3156, a new "asyncio" module, a new framework for asynchronous I/O


 To download Python 3.4.0 visit:

 http://www.python.org/download/releases/3.4.0/


 This is a production release.  Please report any issues you notice to:

  http://bugs.python.org/


 Enjoy!


 --
 Larry Hastings, Release Manager
 larry at hastings.org
 (on behalf of the entire python-dev team and 3.4's contributors)
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: https://mail.python.org/mailman/options/python-dev/
 rymg19%40gmail.com

>>>
>>>
>>>
>>> --
>>> Ryan
>>> If anybody ever asks me why I prefer C++ to C, my answer will be simple:
>>> "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
>>> nul-terminated."
>>>
>>>
>>> --
>>> https://mail.python.org/mailman/listinfo/python-list
>>>
>>>
>>
>>
>> --
>> Giampaolo - http://grodola.blogspot.com
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/dia.aliounes%40gmail.com
>>
>>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/d4n1h4ck%40gmail.com
>
>


-- 
Msc. Daniel Pimentel (d4n1 )
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7

[Nick Coghlan]

On 26 Mar 2014 23:12, "Cory Benfield"  wrote:
>
> Nick,
>
> On 26 March 2014 12:00, Nick Coghlan  wrote:
> > As one example, the Python 2 ``ssl`` module does not support the Server
> > Name Identification standard.
>
> Tiny note: SNI is 'Server Name Indication', not 'Identification'. =)

Hah, you know, I've never actually looked that up, because I assumed I knew
what it stood for based on context :)

Cheers,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7

[Barry Warsaw]

On Mar 26, 2014, at 10:00 PM, Nick Coghlan wrote:

>Guido and Antoine persuaded me that selective backports would be a
>better idea for the network security enhancements than the wholesale
>module backports previously suggested, while Alex and Donald provided
>the necessary additional details, so here's a revised version of the
>PEP. Despite making it more explicit, I deleted more lines than I
>added, strongly suggesting that switching to selective backports was
>the right call :)

Thanks.  I think the narrower scope and details greatly improves this PEP and
resolves most of the concerns I had about it.  It maintains the backward
compatibility pledge but relaxes the new feature backport prohibition where
security concerns trump.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7

[Brett Cannon]

On Wed Mar 26 2014 at 8:02:08 AM, Nick Coghlan  wrote:

> Guido and Antoine persuaded me that selective backports would be a
> better idea for the network security enhancements than the wholesale
> module backports previously suggested, while Alex and Donald provided
> the necessary additional details, so here's a revised version of the
> PEP. Despite making it more explicit, I deleted more lines than I
> added, strongly suggesting that switching to selective backports was
> the right call :)
>
> I dealt with the SSL module the way Donald suggested: excluding the
> RAND_* functions, rather than listing everything else.
>
> I also changed the headings to make it clear the listed alternatives
> were rejected ideas, made the footnotes a bit more readable, and
> tidied up the wording in a few places.
>
> Diff: http://hg.python.org/peps/rev/8527f6e2beb0
> Web: http://www.python.org/dev/peps/pep-0466/
>
> ==
> PEP: 466
> Title: Network Security Enhancement Exception for Python 2.7
> Version: $Revision$
> Last-Modified: $Date$
> Author: Nick Coghlan ,
> Status: Draft
> Type: Informational
> Content-Type: text/x-rst
> Created: 23-Mar-2014
> Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014
>

 [SNIP]


> Exemption Policy
> 
>
> Under this policy, the following features SHOULD be backported from Python
> 3.4 to the upcoming Python 2.7.7 maintenance release:
>
> * in the ``os`` module:
>
>   * persistent file descriptor for ``os.urandom()``.
>
> * in the ``hmac`` module:
>
>   * constant time comparison function (``hmac.compare_digest()``).
>
> * in the ``hashlib`` module:
>
>   * password hashing function (``hashlib.pbkdf2_hmac()``).
>   * details of hash algorithm availability (``hashlib.algorithms_
> guaranteed``
> and ``hashlib.algorithms_guaranteed``).
>

You said algorithms_guaranteed twice. I assume that wasn't for emphasis. =)

I'm +1 on this version of the PEP.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466: Proposed policy change for handling network security enhancements

[Brett Cannon]

On Wed Mar 26 2014 at 2:15:39 AM, Georg Brandl  wrote:

> Am 26.03.2014 00:06, schrieb Nick Coghlan:
> >
> > On 26 Mar 2014 08:32, "Georg Brandl"  > > wrote:
> >>
> >> Am 25.03.2014 23:15, schrieb Nick Coghlan:
> >> >
> >> > On 26 Mar 2014 01:19, "Brett Cannon"  > 
> >> > >> wrote:
> >> >> As long as we make it clear we have chosen to change our
> >> > backwards-compatibility guarantees in the name of security and have a
> link to
> >> > the last backwards-compatible release then I agree as well.
> >> >
> >> > I am not sure how this meme got started, but let me be clear: the
> proposed
> >> > policy DOES NOT provide blanket permission to break backwards
> compatibility in
> >> > the affected modules. It only allows ADDING new features to bring
> these modules
> >> > into line with their Python 3 counterparts, making it easier for
> third party
> >> > packages like requests to do the right thing in a cross-version
> compatible way.
> >>
> >> We know. That's what we mean by that.
> >
> > That's not what Brett said - he called 2.7.6 the "last backwards
> compatible
> > release". That's not correct, as even under my proposal, 2.7.7+ will
> still be
> > backwards compatible.
>
> Yeah, I took "backwards-compatibility guarantees" to also mean the "no new
> features" guarantee, but you're right that the two can be separated.
>

That's also the way I took it; not that code would be inherently broken
from 2.7.0 -> 2.7.x, but that the "let's stay conservative and add
*nothing*" would stop being followed since something in 2.7.x wouldn't work
in 2.7.0. IOW I expect (except maybe in extreme edge cases) that unmodified
code would work across 2.7 as long as it works in 2.7.0 no matter what
happens with this PEP.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C code: %s vs %U

[Ethan Furman]

On 03/26/2014 06:22 AM, � wrote:

[Assuming you are talking about PyUnicode_FromFormatV]

%s is a string.


No. %s is a char*; C does not have a "string" type.
The string behind the pointer should be UTF-8 encoded;
other encodings are tolerated through the "replace" error
handler.


%U is unicode?


No. This is a PyObject* whose Python type is 'str'
(i.e. an object for which PyUnicode_Check succeeds)


If so, then %s should only be used when it is certain the string in
question has no unicode in it?


No. If you have a char*, use %s; using %U would crash.


Many thanks!

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7

[Cory Benfield]

Nick,

On 26 March 2014 12:00, Nick Coghlan  wrote:
> As one example, the Python 2 ``ssl`` module does not support the Server
> Name Identification standard.

Tiny note: SNI is 'Server Name Indication', not 'Identification'. =)

Otherwise, I'm +1 on this as well.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Ethan Furman]

On 03/26/2014 03:10 AM, Victor Stinner wrote:

2014-03-25 23:37 GMT+01:00 Ethan Furman:


``%a`` will call ``ascii()`` on the interpolated value.


I'm not sure that I understood correctly: is the "%a" format
supported? The result of ascii() is a Unicode string. Does it mean
that ("%a" % obj) should give the same result than
ascii(obj).encode('ascii', 'strict')?


Changed to:
---
``%a`` will give the equivalent of
``repr(some_obj).encode('ascii', 'backslashreplace')`` on the interpolated
value.  Use cases include developing a new protocol and writing landmarks
into the stream; debugging data going into an existing protocol to see if
the problem is the protocol itself or bad data; a fall-back for a serialization
format; or any situation where defining ``__bytes__`` would not be appropriate
but a readable/informative representation is needed [8].
---



Would it be possible to add a table or list to summarize supported
format characters? I found:

- single byte: %c
- integer: %d, %u, %i, %o, %x, %X, %f, %g, "etc." (can you please
complete "etc." ?)
- bytes and __bytes__ method: %s
- ascii(): %a


Changed to:
---
%-interpolation
---

All the numeric formatting codes (``d``, ``i``, ``o``, ``u``, ``x``, ``X``,
``e``, ``E'', ``f``, ``F``, ``g``, ``G``, and any that are subsequently added
to Python 3) will be supported, and will work as they do for str, including
the padding, justification and other related modifiers (currently ``#``, ``0``,
``-``, `` `` (space), and ``+`` (plus any added to Python 3)).  The only
non-numeric codes allowed are ``c``, ``s``, and ``a``.

For the numeric codes, the only difference between ``str`` and ``bytes`` (or
``bytearray``) interpolation is that the results from these codes will be
ASCII-encoded text, not unicode.  In other words, for any numeric formatting
code `%x`::
---



I don't understand the purpose of this sentence. Does it mean that %a
must not be used? IMO this sentence can be removed.


The sentence about %a being for debugging has been removed.



Non-ASCII values will be encoded to either ``\xnn`` or ``\u``
representation.


Unicode is larger than that! print(ascii(chr(0x10))) => '\U0010'


Removed.  With the explicit reference to the 'backslashreplace' error handler any who want to know what it might look 
like can refer to that.




.. note::

 If a ``str`` is passed into ``%a``, it will be surrounded by quotes.


And:

- bytes gets a "b" prefix and surrounded by quotes as well  (b'...')
- the quote ' is escaped as \' if the string contains quotes ' and "


Shouldn't be an issue now with the new definition which no longer references 
the ascii() function.



Can you also please add examples for %a?

---
Examples::

>>> b'%a' % 3.14
b'3.14'

>>> b'%a' % b'abc'
b'abc'

>>> b'%a' % 'def'
b"'def'"
---



Proposed variations
===



It would be fair to mention also a whole different PEP, Antoine's PEP 460!


My apologies for the omission.
---
A competing PEP, ``PEP 460 Add binary interpolation and formatting`` [9], also
exists.

.. [9] http://python.org/dev/peps/pep-0460/
---

Thank you, Victor.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Thomas Wouters]

On Tue, Mar 25, 2014 at 11:37 PM, Ethan Furman  wrote:

> In particular, ``%s`` will not accept numbers (use a numeric format code
> for
> that), nor ``str`` (encode it to ``bytes``).
>

I don't understand this restriction, and there isn't a rationale for it in
the PEP (other than "you can already use numeric formats", which doesn't
explain why it's undesirable to have it anyway.) It is extremely common in
existing 2.x code to use %s for anything, just like people use {} for
anything with str.format. Not supporting this feels like it would be
problematic for porting code. Did this come up in the earlier discussions?

-- 
Thomas Wouters 

Hi! I'm an email virus! Think twice before sending your email to help me
spread!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7

[Alex Gaynor]

This mostly looks good to me, however I'm not sure I understand the point of
this sentence: "Rather, it is intended to send a clear signal to potential
corporate contributors that the core development team are willing to accept
offers of corporate assistance in putting this policy into effect [...]". It's
fairly evident to me that the folks most likely to actually do the work of
implementing this are myself and Donald. This PEP really has nothing to do with
corporate contribution, so I think this sentence ought to be removed.

Alex

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7

[Ethan Furman]

On 03/26/2014 05:00 AM, Nick Coghlan wrote:


The Mac OS X binary installers historically followed the same policy as
other POSIX installations and dynamically linked to the Apple provided
OpenSSL libraries. However, Apple has now ceased updating these
cross-platform libraries, instead requiring that even cross-platform
developers adopt Mac OS X specific interfaces to access up to date security
infrastructure on their platform. Accordingly, and independently of this
PEP, the Mac OS X binary installers were already going to be switched to
statically linker newer versions of OpenSSL [4]_


Should be 'link', not 'linker'.

Looks good.

+1

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Ethan Furman]

On 03/26/2014 08:14 AM, Thomas Wouters wrote:

On Tue, Mar 25, 2014 at 11:37 PM, Ethan Furman wrote:

In particular, ``%s`` will not accept numbers (use a numeric format code for
that), nor ``str`` (encode it to ``bytes``).


I don't understand this restriction, and there isn't a rationale for it in the PEP 
(other than "you can already use
numeric formats", which doesn't explain why it's undesirable to have it 
anyway.) It is extremely common in existing 2.x
code to use %s for anything


And that's the problem -- in 2.x %s works always, but 3.x for bytes and 
bytearray %s will fail in numerous situations.

It seems to me the main reason for using %s instead of %d is that 'some_var' may have a number, or it may have the 
textual representation of that number; in 3.x the first would succeed, the second would fail.  That's the kind of 
intermittent failure we do not want.


The PEP is not designed to make it so 2.x code can be ported as-is, but rather that 2.x code can be cleaned up (if 
necessary) and then run the same in both 2.x and 3.x (at least as far as byte and bytearray %-formatting is concerned).



 Did this come up in the earlier discussions?


https://mail.python.org/pipermail/python-dev/2014-January/131576.html

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] collections.sortedtree

[Marko Rauhamaa]

I have made a full implementation of a balanced tree and would like to
know what the process is to have it considered for inclusion in Python
3.

To summarize, the implementation closely parallels dict() features and
resides in _collectionsmodule.c under the name collections.sortedtree.
It uses solely the "<" operator to compare keys. I have chosen the AVL
tree as an implementation technique.

The views support a number of optional arguments:

sorteddict.keys(reversed=False, start=unspecified, inclusive=True)


The primary objective of having a balanced tree in the standard library
is to support ordered access in an efficient manner. The typical
applications would include timers (networking), aging (cache) and prefix
patterns (routing).


Marko
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Ethan Furman]

On 03/26/2014 01:31 PM, Marko Rauhamaa wrote:


I have made a full implementation of a balanced tree and would like to
know what the process is to have it considered for inclusion in Python
3.


Open a ticket on the tracker [1], post your code to that ticket, sign the CLA [2], answer questions, etc., that come up 
in the code review.


I believe Raymond Hettinger is the Guardian of the collections module; add him 
as nosy.

--
~Ethan~

[1] http://http://bugs.python.org
[2] https://www.python.org/psf/contrib/contrib-form
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Benjamin Peterson]

On Wed, Mar 26, 2014, at 13:31, Marko Rauhamaa wrote:
> 
> I have made a full implementation of a balanced tree and would like to
> know what the process is to have it considered for inclusion in Python
> 3.

It's not a bad idea. (I believe others have proposed an red-black tree.)
Certainly, it requires a PEP and a few months of bikesheding, though.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Guido van Rossum]

Actually, the first step is publish it on PyPI, the second is to get a fair
number of happy users there. The bar for getting something included into
the stdlib is pretty high -- you need to demonstrate that there is a need
*and* that having it as a 3rd party module is a problem. And that once it's
in, (a) it will be stable, and (b) someone who cares about it and knows the
code thoroughly is available maintain it for years.


On Wed, Mar 26, 2014 at 1:53 PM, Ethan Furman  wrote:

> On 03/26/2014 01:31 PM, Marko Rauhamaa wrote:
>
>>
>> I have made a full implementation of a balanced tree and would like to
>> know what the process is to have it considered for inclusion in Python
>> 3.
>>
>
> Open a ticket on the tracker [1], post your code to that ticket, sign the
> CLA [2], answer questions, etc., that come up in the code review.
>
> I believe Raymond Hettinger is the Guardian of the collections module; add
> him as nosy.
>
> --
> ~Ethan~
>
> [1] http://http://bugs.python.org
> [2] https://www.python.org/psf/contrib/contrib-form
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Barry Warsaw]

On Mar 26, 2014, at 01:55 PM, Benjamin Peterson wrote:

>It's not a bad idea. (I believe others have proposed an red-black tree.)
>Certainly, it requires a PEP and a few months of bikesheding, though.

Generally, PEPs aren't necessary for simple or relatively uncontroversial
additions to existing modules or the stdlib.

If you're proposing an addition to an existing module, file a tracker issue
and nosy the module expert (for collections, it's Raymond Hettinger[1]).
Others may see the new issue and decide to nosy themselves in if they want to
provide additional feedback and input.

If you're proposing an entirely new module, it's best to publish it first on
PyPI, get lots of users, and listen to their feedback as you evolve the
library.  Once you feel like it's had plenty of time to bake as a third party
module *and* the API is sufficiently stable that you won't mind the Python
stdlib 18 month development cycle, then you can propose that it be included in
Python.

Cheers,
-Barry

[1] http://docs.python.org/devguide/experts.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 5): selected network security enhancements for Python 2.7

[Nick Coghlan]

On 27 Mar 2014 01:28, "Alex Gaynor"  wrote:
>
> This mostly looks good to me, however I'm not sure I understand the point
of
> this sentence: "Rather, it is intended to send a clear signal to potential
> corporate contributors that the core development team are willing to
accept
> offers of corporate assistance in putting this policy into effect [...]".
It's
> fairly evident to me that the folks most likely to actually do the work of
> implementing this are myself and Donald. This PEP really has nothing to
do with
> corporate contribution, so I think this sentence ought to be removed.

Sure, I can reword that bit. I think I've made my point on that front by
now, anyway, so preserving it in the web version isn't really necessary :)

Cheers,
Nick.

>
> Alex
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] On the necessity of PEPs [was "collections.sortedtree"]

[Benjamin Peterson]

On Wed, Mar 26, 2014, at 14:25, Barry Warsaw wrote:
> On Mar 26, 2014, at 01:55 PM, Benjamin Peterson wrote:
> 
> >It's not a bad idea. (I believe others have proposed an red-black tree.)
> >Certainly, it requires a PEP and a few months of bikesheding, though.
> 
> Generally, PEPs aren't necessary for simple or relatively uncontroversial
> additions to existing modules or the stdlib.

I would have said that, too, several years ago, but I think we've been
requiring (or using anyway) PEPs for a lot more things now. OrderedDict
had a PEP for example.

I'm not sure if that's a good thing or not.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Antoine Pitrou]

On Wed, 26 Mar 2014 22:31:56 +0200
Marko Rauhamaa  wrote:
> 
> The primary objective of having a balanced tree in the standard library
> is to support ordered access in an efficient manner. The typical
> applications would include timers (networking), aging (cache)

Wouldn't a heapq work as well for those two?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"]

[Barry Warsaw]

On Mar 26, 2014, at 02:27 PM, Benjamin Peterson wrote:

>I would have said that, too, several years ago, but I think we've been
>requiring (or using anyway) PEPs for a lot more things now. OrderedDict
>had a PEP for example.
>
>I'm not sure if that's a good thing or not.

Hmm, me neither!  I guess if someone *wants* to go through the PEP gauntlet, I
won't stop them.  It builds character.

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Marko Rauhamaa]

Guido van Rossum :

> Actually, the first step is publish it on PyPI, the second is to get a
> fair number of happy users there. The bar for getting something
> included into the stdlib is pretty high -- you need to demonstrate
> that there is a need *and* that having it as a 3rd party module is a
> problem.

I hear you about the process.

About the need part, I'm wondering if you haven't felt it in
implementing the timers for asyncio. I have had that need in several
network programming projects and have ended up using my AVL tree
implementation (C and Python).

Well, time will tell if frequent canceled timers end up piling up the
heap queue.


Marko
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"]

[Donald Stufft]

On Mar 26, 2014, at 5:30 PM, Barry Warsaw  wrote:

> On Mar 26, 2014, at 02:27 PM, Benjamin Peterson wrote:
> 
>> I would have said that, too, several years ago, but I think we've been
>> requiring (or using anyway) PEPs for a lot more things now. OrderedDict
>> had a PEP for example.
>> 
>> I'm not sure if that's a good thing or not.
> 
> Hmm, me neither!  I guess if someone *wants* to go through the PEP gauntlet, I
> won't stop them.  It builds character.
> 
> -Barry
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

Is that what it’s called? “character”  >:]

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Terry Reedy]

On 3/26/2014 4:59 PM, Guido van Rossum wrote:

Actually, the first step is publish it on PyPI, the second is to get a
fair number of happy users there. The bar for getting something included
into the stdlib is pretty high -- you need to demonstrate that there is
a need *and* that having it as a 3rd party module is a problem. And that
once it's in, (a) it will be stable, and (b) someone who cares about it
and knows the code thoroughly is available maintain it for years.


Perhaps the collections doc should mention that there are other 
specializes container types available on PyPI and either list some or 
point to a wiki page listing some. There must be at least 10 that could 
be included in such a list.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Victor Stinner]

2014-03-26 15:35 GMT+01:00 Ethan Furman :
> ---
> Examples::
>
> >>> b'%a' % 3.14
> b'3.14'
>
> >>> b'%a' % b'abc'
> b'abc'

This one is wrong:

>>> repr(b'abc').encode('ascii', 'backslashreplace')
b"b'abc'"

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Guido van Rossum]

I haven't felt it, heapq feels natural to me for this use case. :-)

I'm aware of the issue of frequent cancelled timers, but chose to wait and
see rather than preemptively fix it (only so many hours in a day). IIRC
pyftplib has a clever cleanup algorithm that we can easily add if that
usage pattern becomes popular.


On Wed, Mar 26, 2014 at 2:36 PM, Marko Rauhamaa  wrote:

> Guido van Rossum :
>
> > Actually, the first step is publish it on PyPI, the second is to get a
> > fair number of happy users there. The bar for getting something
> > included into the stdlib is pretty high -- you need to demonstrate
> > that there is a need *and* that having it as a 3rd party module is a
> > problem.
>
> I hear you about the process.
>
> About the need part, I'm wondering if you haven't felt it in
> implementing the timers for asyncio. I have had that need in several
> network programming projects and have ended up using my AVL tree
> implementation (C and Python).
>
> Well, time will tell if frequent canceled timers end up piling up the
> heap queue.
>
>
> Marko
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Nick Coghlan]

On 27 Mar 2014 07:02, "Guido van Rossum"  wrote:
>
> Actually, the first step is publish it on PyPI, the second is to get a
fair number of happy users there. The bar for getting something included
into the stdlib is pretty high -- you need to demonstrate that there is a
need *and* that having it as a 3rd party module is a problem. And that once
it's in, (a) it will be stable, and (b) someone who cares about it and
knows the code thoroughly is available maintain it for years.

The "why not a third party module?" bar also got a fair bit higher with
Python 3.4 - by bundling pip, we have deliberately made third party modules
easier to consume, thus weakening the convenience argument that applies to
stdlib inclusion.

The Python Packaging Authority are also working to reduce the barriers to
distribution and consumption of C extensions, which will again weaken the
argument for stdlib inclusion of third party C extensions. The metadata 2.0
efforts are also designed to streamline the process of conversion to
platform specific formats like RPM.

It's not that I don't see a sorted tree as valuable - I do. I just also see
it as a fairly specialised type. The main case I could see being made for
inclusion is if it made a major difference to the implementation of
something else that was already in the stdlib. I believe that's where most
other additions to the collections and types modules have come from in
recent releases - we wanted to *use* them, and it seemed better to do the
work to design an appropriate public API rather than keep them private
(enum and generic function support arrived the same way, although we
haven't actually redesigned pprint to rely on the latter yet).

Cheers,
Nick.

>
>
> On Wed, Mar 26, 2014 at 1:53 PM, Ethan Furman  wrote:
>>
>> On 03/26/2014 01:31 PM, Marko Rauhamaa wrote:
>>>
>>>
>>> I have made a full implementation of a balanced tree and would like to
>>> know what the process is to have it considered for inclusion in Python
>>> 3.
>>
>>
>> Open a ticket on the tracker [1], post your code to that ticket, sign
the CLA [2], answer questions, etc., that come up in the code review.
>>
>> I believe Raymond Hettinger is the Guardian of the collections module;
add him as nosy.
>>
>> --
>> ~Ethan~
>>
>> [1] http://http://bugs.python.org
>> [2] https://www.python.org/psf/contrib/contrib-form
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Marko Rauhamaa]

Antoine Pitrou :

> Wouldn't a heapq work as well for those two?

In my experience, networking entities typically start a timer at each
interaction and cancel the pending one. So you have numerous timers that
virtually never expire. You might have 100 interactions per second, each
canceling and restarting a 10-minute timer.

I don't know first hand if that causes heap queues to cause measurable
heap or CPU pressure.


Marko
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"]

[Antoine Pitrou]

On Wed, 26 Mar 2014 17:37:40 -0400
Donald Stufft  wrote:

> 
> On Mar 26, 2014, at 5:30 PM, Barry Warsaw  wrote:
> 
> > On Mar 26, 2014, at 02:27 PM, Benjamin Peterson wrote:
> > 
> >> I would have said that, too, several years ago, but I think we've been
> >> requiring (or using anyway) PEPs for a lot more things now. OrderedDict
> >> had a PEP for example.
> >> 
> >> I'm not sure if that's a good thing or not.
> > 
> > Hmm, me neither!  I guess if someone *wants* to go through the PEP 
> > gauntlet, I
> > won't stop them.  It builds character.
> > 
> > -Barry
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: 
> > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
> 
> Is that what it’s called? “character”  >:]

Preferably unicode.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Antoine Pitrou]

On Wed, 26 Mar 2014 23:41:42 +0200
Marko Rauhamaa  wrote:
> Antoine Pitrou :
> 
> > Wouldn't a heapq work as well for those two?
> 
> In my experience, networking entities typically start a timer at each
> interaction and cancel the pending one. So you have numerous timers that
> virtually never expire. You might have 100 interactions per second, each
> canceling and restarting a 10-minute timer.
> 
> I don't know first hand if that causes heap queues to cause measurable
> heap or CPU pressure.

Each individual heapq operation (push or pop) will be O(log n). That's
not different from a balanced search tree (although of course the
constant multiplier may vary).

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Marko Rauhamaa]

Terry Reedy :

> Perhaps the collections doc should mention that there are other
> specializes container types available on PyPI and either list some or
> point to a wiki page listing some. There must be at least 10 that
> could be included in such a list.

There *is* a relatively high threshold of importing C extensions from an
external source. If I build an application making use of them and advise
coworkers to use it, they would likely balk at having to compile them.
Not all machines have a development toolkit.

Furthermore:

   # which pip
   /usr/bin/which: no pip in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:\
   /usr/sbin:/usr/bin:/root/bin)
   # yum install pip
   No package pip available.


Marko
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Marko Rauhamaa]

Antoine Pitrou :

> Marko Rauhamaa  wrote:
>> In my experience, networking entities typically start a timer at each
>> interaction and cancel the pending one. So you have numerous timers
>> that virtually never expire. You might have 100 interactions per
>> second, each canceling and restarting a 10-minute timer.
>
> Each individual heapq operation (push or pop) will be O(log n). That's
> not different from a balanced search tree (although of course the
> constant multiplier may vary).

Yes, but if I have 1000 connections with one active timer each. The size
of my sorted tree is 1000 timer objects. There are typically no expiries
to react to.

If the canceled timer lingers in the heapq till its expiry (in 10
minutes), the size is 100 * 10 * 60 = 60,000. The CPU has to wake up
constantly to clear the expired timers.

In practice, none of that might matter.


Marko
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Antoine Pitrou]

On Wed, 26 Mar 2014 23:57:27 +0200
Marko Rauhamaa  wrote:
> Antoine Pitrou :
> 
> > Marko Rauhamaa  wrote:
> >> In my experience, networking entities typically start a timer at each
> >> interaction and cancel the pending one. So you have numerous timers
> >> that virtually never expire. You might have 100 interactions per
> >> second, each canceling and restarting a 10-minute timer.
> >
> > Each individual heapq operation (push or pop) will be O(log n). That's
> > not different from a balanced search tree (although of course the
> > constant multiplier may vary).
> 
> Yes, but if I have 1000 connections with one active timer each. The size
> of my sorted tree is 1000 timer objects. There are typically no expiries
> to react to.
> 
> If the canceled timer lingers in the heapq till its expiry (in 10
> minutes), the size is 100 * 10 * 60 = 60,000. The CPU has to wake up
> constantly to clear the expired timers.

Ideally, I think you should be able to replace the cancelled item with
the last item in the heap and then fix the heap in logarithmic time,
but the heapq API doesn't seem to provide a way to do this.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Ben Darnell]

On Wed, Mar 26, 2014 at 6:02 PM, Antoine Pitrou  wrote:

> On Wed, 26 Mar 2014 23:57:27 +0200
> Marko Rauhamaa  wrote:
> > Antoine Pitrou :
> >
> > > Marko Rauhamaa  wrote:
> > >> In my experience, networking entities typically start a timer at each
> > >> interaction and cancel the pending one. So you have numerous timers
> > >> that virtually never expire. You might have 100 interactions per
> > >> second, each canceling and restarting a 10-minute timer.
> > >
> > > Each individual heapq operation (push or pop) will be O(log n). That's
> > > not different from a balanced search tree (although of course the
> > > constant multiplier may vary).
> >
> > Yes, but if I have 1000 connections with one active timer each. The size
> > of my sorted tree is 1000 timer objects. There are typically no expiries
> > to react to.
> >
> > If the canceled timer lingers in the heapq till its expiry (in 10
> > minutes), the size is 100 * 10 * 60 = 60,000. The CPU has to wake up
> > constantly to clear the expired timers.
>
> Ideally, I think you should be able to replace the cancelled item with
> the last item in the heap and then fix the heap in logarithmic time,
> but the heapq API doesn't seem to provide a way to do this.
>

Heaps provide easy access to the first item, but you can't find the last
element without scanning the whole thing.  With a heap, I think the best
approach to the timer-cancellation problem is to periodically rebuild the
heap from the non-cancelled items (Tornado keeps a count of cancellations
and rebuilds the heap when the number of cancelled timeouts is half of the
total.  If cancellations are infrequent then they're just left in the heap
until their time comes due).

-Ben


>
> Regards
>
> Antoine.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Guido van Rossum]

Yeah, so the pyftp fix is to keep track of how many timers were cancelled,
and if the number exceeds a threshold it just recreates the heap, something
like

heap = [x for x in heap if not x.cancelled]
heapify(heap)


On Wed, Mar 26, 2014 at 3:02 PM, Antoine Pitrou  wrote:

> On Wed, 26 Mar 2014 23:57:27 +0200
> Marko Rauhamaa  wrote:
> > Antoine Pitrou :
> >
> > > Marko Rauhamaa  wrote:
> > >> In my experience, networking entities typically start a timer at each
> > >> interaction and cancel the pending one. So you have numerous timers
> > >> that virtually never expire. You might have 100 interactions per
> > >> second, each canceling and restarting a 10-minute timer.
> > >
> > > Each individual heapq operation (push or pop) will be O(log n). That's
> > > not different from a balanced search tree (although of course the
> > > constant multiplier may vary).
> >
> > Yes, but if I have 1000 connections with one active timer each. The size
> > of my sorted tree is 1000 timer objects. There are typically no expiries
> > to react to.
> >
> > If the canceled timer lingers in the heapq till its expiry (in 10
> > minutes), the size is 100 * 10 * 60 = 60,000. The CPU has to wake up
> > constantly to clear the expired timers.
>
> Ideally, I think you should be able to replace the cancelled item with
> the last item in the heap and then fix the heap in logarithmic time,
> but the heapq API doesn't seem to provide a way to do this.
>
> Regards
>
> Antoine.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Antoine Pitrou]

On Wed, 26 Mar 2014 18:14:19 -0400
Ben Darnell  wrote:
> > >
> > > If the canceled timer lingers in the heapq till its expiry (in 10
> > > minutes), the size is 100 * 10 * 60 = 60,000. The CPU has to wake up
> > > constantly to clear the expired timers.
> >
> > Ideally, I think you should be able to replace the cancelled item with
> > the last item in the heap and then fix the heap in logarithmic time,
> > but the heapq API doesn't seem to provide a way to do this.
> 
> Heaps provide easy access to the first item, but you can't find the last
> element without scanning the whole thing.

I was talking about the last element, not the largest. The point of
replacing with the last element is that shrinking is O(1).

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"]

[Ethan Furman]

On 03/26/2014 02:46 PM, Antoine Pitrou wrote:

On Wed, 26 Mar 2014 17:37:40 -0400
Donald Stufft wrote:

On Mar 26, 2014, at 5:30 PM, Barry Warsaw wrote:

I guess if someone *wants* to go through the PEP gauntlet, I
won't stop them.  It builds character.


Is that what it’s called? “character”  >:]


Preferably unicode.


Indeed, a lot more swear symbols available in Unicode!  ;)

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Dan Stromberg]

On Wed, Mar 26, 2014 at 1:55 PM, Benjamin Peterson  wrote:
> On Wed, Mar 26, 2014, at 13:31, Marko Rauhamaa wrote:
>>
>> I have made a full implementation of a balanced tree and would like to
>> know what the process is to have it considered for inclusion in Python
>> 3.
>
> It's not a bad idea. (I believe others have proposed an red-black tree.)
> Certainly, it requires a PEP and a few months of bikesheding, though.

I'd recommend a plain binary tree (for random keys), red-black tree
and treap (both for ordered or mostly-ordered keys, where red-black
tree gives low operation duration standard deviation, and treap gives
low average operation duration).

https://en.wikipedia.org/wiki/Binary_search_tree#Performance_comparisons
http://stromberg.dnsalias.org/~strombrg/python-tree-and-heap-comparison/

It'd likely make sense to have either a pure python implementation, or
pure python and C-extended, so that Pypy and Jython can share the
feature with CPython.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Dan Stromberg]

On Wed, Mar 26, 2014 at 3:52 PM, Marko Rauhamaa  wrote:
> Dan Stromberg :
>
>> It'd likely make sense to have either a pure python implementation, or
>> pure python and C-extended, so that Pypy and Jython can share the
>> feature with CPython.
>
> Jython can build directly on Java's native SortedMap implementation. The
> API should not tie it to a tree. Optimizations and refactorings should
> be allowed. Only O(log N) worst-case behavior should be mandated.

Rare worst cases should be fine for a good reason - see python dict's.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Marko Rauhamaa]

Dan Stromberg :

> It'd likely make sense to have either a pure python implementation, or
> pure python and C-extended, so that Pypy and Jython can share the
> feature with CPython.

Jython can build directly on Java's native SortedMap implementation. The
API should not tie it to a tree. Optimizations and refactorings should
be allowed. Only O(log N) worst-case behavior should be mandated.

(And now I notice I named this thread wrong; I named my thingy
collections.sorteddict.)


Marko
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Ethan Furman]

On 03/26/2014 02:41 PM, Victor Stinner wrote:

2014-03-26 15:35 GMT+01:00 Ethan Furman :

---
Examples::

 >>> b'%a' % 3.14
 b'3.14'

 >>> b'%a' % b'abc'
 b'abc'


This one is wrong:


repr(b'abc').encode('ascii', 'backslashreplace')

b"b'abc'"


Fixed, thanks.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"]

[Eli Bendersky]

On Wed, Mar 26, 2014 at 2:27 PM, Benjamin Peterson wrote:

>
>
> On Wed, Mar 26, 2014, at 14:25, Barry Warsaw wrote:
> > On Mar 26, 2014, at 01:55 PM, Benjamin Peterson wrote:
> >
> > >It's not a bad idea. (I believe others have proposed an red-black tree.)
> > >Certainly, it requires a PEP and a few months of bikesheding, though.
> >
> > Generally, PEPs aren't necessary for simple or relatively uncontroversial
> > additions to existing modules or the stdlib.
>
> I would have said that, too, several years ago, but I think we've been
> requiring (or using anyway) PEPs for a lot more things now. OrderedDict
> had a PEP for example.
>

This is probably a natural outcome of the rising popularity of Python in
the last few years. Much more users, more core developers, more at stake...

>
> I'm not sure if that's a good thing or not.
>

YMMV but IMHO this is a good thing. PEPs provide a single point of
reference to a discussion that would otherwise be spread over multiple
centi-threads (not that PEPs don't create centi-threads, but they outlive
them in a way).

Eli
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"]

[Ethan Furman]

On 03/26/2014 07:11 PM, Eli Bendersky wrote:

On Wed, Mar 26, 2014 at 2:27 PM, Benjamin Peterson  wrote:


I'm not sure if that's a good thing or not.


YMMV but IMHO this is a good thing. PEPs provide a single point of reference to 
a discussion that would otherwise be
spread over multiple centi-threads (not that PEPs don't create centi-threads, 
but they outlive them in a way).


Plus the PEP can help prevent multiple mega-threads as the same idea is 
revisited again and again and again...

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com