Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Stefan Behnel]

Brett Cannon, 05.04.2011 01:46:

At both the VM and language summits at PyCon this year, the issue of
compatibility of the stdlib amongst the various VMs came up. Two issues came
about in regards to modules that use C code. One is that code that comes in
only as C code sucks for all other VMs that are not CPython since they all
end up having to re-implement that module themselves. Two is that modules
that have an accelerator module (e.g., heapq, warnings, etc.) can end up
with compatibility options (sorry, Raymond, for picking on heapq, but is was
what bit the PyPy people most recently =).

In lieu of all of this, here is a draft PEP to more clearly state the policy
for the stdlib when it comes to C code. Since this has come up before and
this was discussed so much at the summits I have gone ahead and checked this
in so that even if this PEP gets rejected there can be a written record as
to why.

And before anyone asks, I have already run this past the lead devs of PyPy,
Jython, and IronPython and they all support what this PEP proposes.


We recently had the discussion about reimplementing stdlib C modules in 
Cython. Accelerator modules are the obvious first step here, as they could 
be implemented in Python and compiled with Cython, instead of actually 
writing them in C in the first place. Wouldn't this be worth mentioning in 
the PEP?


Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Policy for making changes to the AST

[Martin v. Löwis]

> Ok, so it sounds like ast is *not* limited to CPython? That makes it
> harder to justify changing it just so as to ease the compilation
> process in CPython (as opposed to add new language features).

I propose a different view: the AST *is* implementation specific,
although implementations are certainly encouraged to use a similar AST
if they provide an AST module at all.

Applications of it then explicitly need to be ported to each version
of each Python implementation that supports an AST module. If the
ASTs are similar, this porting will hopefully be easy.

The only alternative I can see is to freeze the AST structure, allowing
for extensions at best. I don't think any of the implementations are in
a state where such an approach is feasible.

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Policy for making changes to the AST

[Martin v. Löwis]

> Add py.test as an application that uses the AST to support Jython,
> PyPy and CPython in a portable way.  I always assumed AST was created
> *because* bytecode was too CPython specific (but then I've never
> implemented a language).

Historically, that's incorrect. The AST structure was introduced to
simplify the implementation of (C)Python. Exposing it to Python modules
was done primarily because it's neat and was easy to do - not because
any specific use was expected, and certainly not as the primary
motivation for having an AST. It was clear (to me, at least) back then
that the AST will change over time (hence the module was originally
called _ast).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supporting Visual Studio 2010

[Martin v. Löwis]

Am 05.04.2011 00:21, schrieb Antoine Pitrou:
> On Mon, 04 Apr 2011 23:40:33 +0200
> "Martin v. Löwis"  wrote:
>> - users have expressed concerns that they constantly need to upgrade
>>   VS releases when developing for Python.
> 
> Isn't that kind of a misguided argument? It's not Python who decides the
> lifecycle of MSVC releases, it's Microsoft. We can't be blamed for the
> churn.

But we *can* be blamed for closely following the MS release cycle (if
we actually did that). For Python 3.2, we resisted.

> If getting old (Microsoft-unsupported) MSVC releases is difficult, then
> I think switching to the newest MSVC as soon as possible is the best
> strategy, since it minimizes the annoyance for people wanting to build
> extensions several years after a release is made.

OTOH, the very same people will have to buy licenses for all MSVC
releases. If we manage to skip some of them, the zoo of products you
need to install to support Python gets smaller.

Of course, if you use the stable ABI, going forward, you can decouple
from Python's product management.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supporting Visual Studio 2010

[Martin v. Löwis]

> Won't that still be an issue despite the stable ABI? Extensions on
> Windows should be linked to the same version of MSVCRT used to compile
> Python

Not if they use the stable ABI. There still might be issues if you
mix CRTs, but none related to the Python ABI - in particular, none
of those crashing conditions can arise from the stable ABI.

> This would seem to circumvent one of the core use-cases of the stable
> ABI which was not needing to recompile extensions for new versions of
> Python. Of course I could be completely wrong about all this.

Not completely, but slightly (I hope).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Issue #11707: Fast C version of functools.cmp_to_key()

[Victor Stinner]

I don't think that the following change conforms to the PEP 399: there
is only a test for the C version.

Victor

Le mardi 05 avril 2011 à 11:34 +0200, raymond.hettinger a écrit :
> http://hg.python.org/cpython/rev/a03fb2fc3ed8
> changeset:   69150:a03fb2fc3ed8
> user:Raymond Hettinger 
> date:Tue Apr 05 02:33:54 2011 -0700
> summary:
>   Issue #11707: Fast C version of functools.cmp_to_key()
> 
> diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py
> --- a/Lib/test/test_functools.py
> +++ b/Lib/test/test_functools.py
> @@ -435,18 +435,81 @@
>  self.assertEqual(self.func(add, d), "".join(d.keys()))
>  
>  class TestCmpToKey(unittest.TestCase):
> +
>  def test_cmp_to_key(self):
> +def cmp1(x, y):
> +return (x > y) - (x < y)
> +key = functools.cmp_to_key(cmp1)
> +self.assertEqual(key(3), key(3))
> +self.assertGreater(key(3), key(1))
> +def cmp2(x, y):
> +return int(x) - int(y)
> +key = functools.cmp_to_key(cmp2)
> +self.assertEqual(key(4.0), key('4'))
> +self.assertLess(key(2), key('35'))
> +
> +def test_cmp_to_key_arguments(self):
> +def cmp1(x, y):
> +return (x > y) - (x < y)
> +key = functools.cmp_to_key(mycmp=cmp1)
> +self.assertEqual(key(obj=3), key(obj=3))
> +self.assertGreater(key(obj=3), key(obj=1))
> +with self.assertRaises((TypeError, AttributeError)):
> +key(3) > 1# rhs is not a K object
> +with self.assertRaises((TypeError, AttributeError)):
> +1 < key(3)# lhs is not a K object
> +with self.assertRaises(TypeError):
> +key = functools.cmp_to_key() # too few args
> +with self.assertRaises(TypeError):
> +key = functools.cmp_to_key(cmp1, None)   # too many args
> +key = functools.cmp_to_key(cmp1)
> +with self.assertRaises(TypeError):
> +key()# too few args
> +with self.assertRaises(TypeError):
> +key(None, None)  # too many args
> +
> +def test_bad_cmp(self):
> +def cmp1(x, y):
> +raise ZeroDivisionError
> +key = functools.cmp_to_key(cmp1)
> +with self.assertRaises(ZeroDivisionError):
> +key(3) > key(1)
> +
> +class BadCmp:
> +def __lt__(self, other):
> +raise ZeroDivisionError
> +def cmp1(x, y):
> +return BadCmp()
> +with self.assertRaises(ZeroDivisionError):
> +key(3) > key(1)
> +
> +def test_obj_field(self):
> +def cmp1(x, y):
> +return (x > y) - (x < y)
> +key = functools.cmp_to_key(mycmp=cmp1)
> +self.assertEqual(key(50).obj, 50)
> +
> +def test_sort_int(self):
>  def mycmp(x, y):
>  return y - x
>  self.assertEqual(sorted(range(5), key=functools.cmp_to_key(mycmp)),
>   [4, 3, 2, 1, 0])
>  
> +def test_sort_int_str(self):
> +def mycmp(x, y):
> +x, y = int(x), int(y)
> +return (x > y) - (x < y)
> +values = [5, '3', 7, 2, '0', '1', 4, '10', 1]
> +values = sorted(values, key=functools.cmp_to_key(mycmp))
> +self.assertEqual([int(value) for value in values],
> + [0, 1, 1, 2, 3, 4, 5, 7, 10])
> +
>  def test_hash(self):
>  def mycmp(x, y):
>  return y - x
>  key = functools.cmp_to_key(mycmp)
>  k = key(10)
> -self.assertRaises(TypeError, hash(k))
> +self.assertRaises(TypeError, hash, k) 



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Nick Coghlan]

On Tue, Apr 5, 2011 at 9:46 AM, Brett Cannon  wrote:
>     try:
>     c_heapq.heappop(Spam())
>     except TypeError:
>     # "heap argument must be a list"
>     pass
>
>     try:
>     py_heapq.heappop(Spam())
>     except AttributeError:
>     # "'Foo' object has no attribute 'pop'"
>     pass
>
> This kind of divergence is a problem for users as they unwittingly
> write code that is CPython-specific. This is also an issue for other
> VM teams as they have to deal with bug reports from users thinking
> that they incorrectly implemented the module when in fact it was
> caused by an untested case.

While I agree with the PEP in principle, I disagree with the way this
example is written. Guido has stated in the past that code simply
*cannot* rely on TypeError being consistently thrown instead of
AttributeError (or vice-versa) when it comes to duck-typing. Code that
cares which of the two is thrown is wrong.

However, there actually *is* a significant semantic discrepancy in the
heapq case, which is that py_heapq is duck-typed, while c_heapq is
not:

>>> from test.support import import_fresh_module
>>> c_heapq = import_fresh_module('heapq', fresh=['_heapq'])
>>> py_heapq = import_fresh_module('heapq', blocked=['_heapq'])
>>> from collections import UserList
>>> class Seq(UserList): pass
...
>>> c_heapq.heappop(UserList())
Traceback (most recent call last):
  File "", line 1, in 
TypeError: heap argument must be a list
>>> py_heapq.heappop(UserList())
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/ncoghlan/devel/py3k/Lib/heapq.py", line 140, in heappop
lastelt = heap.pop()# raises appropriate IndexError if heap is empty
  File "/home/ncoghlan/devel/py3k/Lib/collections/__init__.py", line 848, in pop
def pop(self, i=-1): return self.data.pop(i)
IndexError: pop from empty list

Cheers,
Nick.

P.S. The reason I was bugging Guido to answer the TypeError vs
AttributeError question in the first place was to find out whether or
not I needed to get rid of the following gross inconsistency in the
behaviour of the with statement relative to other language constructs:

>>> 1()
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'int' object is not callable
>>> with 1: pass
...
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'int' object has no attribute '__exit__'



Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supporting Visual Studio 2010

[exarkun]

On 09:55 am, mar...@v.loewis.de wrote:

Am 05.04.2011 00:21, schrieb Antoine Pitrou:

On Mon, 04 Apr 2011 23:40:33 +0200
"Martin v. L�wis"  wrote:

- users have expressed concerns that they constantly need to upgrade
  VS releases when developing for Python.


Isn't that kind of a misguided argument? It's not Python who decides 
the

lifecycle of MSVC releases, it's Microsoft. We can't be blamed for the
churn.


But we *can* be blamed for closely following the MS release cycle (if
we actually did that). For Python 3.2, we resisted.
If getting old (Microsoft-unsupported) MSVC releases is difficult, 
then

I think switching to the newest MSVC as soon as possible is the best
strategy, since it minimizes the annoyance for people wanting to build
extensions several years after a release is made.


OTOH, the very same people will have to buy licenses for all MSVC
releases. If we manage to skip some of them, the zoo of products you
need to install to support Python gets smaller.


Recent Visual Studio Express editions are available as free downloads.

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supporting Visual Studio 2010

[exarkun]

On 09:58 am, mar...@v.loewis.de wrote:

Won't that still be an issue despite the stable ABI? Extensions on
Windows should be linked to the same version of MSVCRT used to compile
Python


Not if they use the stable ABI. There still might be issues if you
mix CRTs, but none related to the Python ABI - in particular, none
of those crashing conditions can arise from the stable ABI.


Does this mean new versions of distutils let you build_ext with any C 
compiler, instead of enforcing the same compiler as it has done 
previously?  That would be great.


Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supporting Visual Studio 2010

[Brian Curtin]

On Tue, Apr 5, 2011 at 07:48,  wrote:

> On 09:55 am, mar...@v.loewis.de wrote:
>
>> Am 05.04.2011 00:21, schrieb Antoine Pitrou:
>>
>>> On Mon, 04 Apr 2011 23:40:33 +0200
>>> "Martin v. Löwis"  wrote:
>>>
 - users have expressed concerns that they constantly need to upgrade
  VS releases when developing for Python.

>>>
>>> Isn't that kind of a misguided argument? It's not Python who decides the
>>> lifecycle of MSVC releases, it's Microsoft. We can't be blamed for the
>>> churn.
>>>
>>
>> But we *can* be blamed for closely following the MS release cycle (if
>> we actually did that). For Python 3.2, we resisted.
>>
>>> If getting old (Microsoft-unsupported) MSVC releases is difficult, then
>>> I think switching to the newest MSVC as soon as possible is the best
>>> strategy, since it minimizes the annoyance for people wanting to build
>>> extensions several years after a release is made.
>>>
>>
>> OTOH, the very same people will have to buy licenses for all MSVC
>> releases. If we manage to skip some of them, the zoo of products you
>> need to install to support Python gets smaller.
>>
>
> Recent Visual Studio Express editions are available as free downloads.
>
> Jean-Paul


On top of that, since you and others have asked on IRC: Visual Studio 2010
Express supports x64 compilation if you have the Windows SDK installed
alongside VS2010. No more "support" via registry and config file hacking.

http://msdn.microsoft.com/en-us/library/9yb4317s.aspx
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] clarification: subset vs equality Re: [Python-checkins] peps: Draft of PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Jim Jewett]

On 4/4/11, brett.cannon  wrote:
>   Draft of PEP 399: Pure Python/C Accelerator Module Compatibiilty
> Requirements

> +Abstract
> +
> +
> +The Python standard library under CPython contains various instances
> +of modules implemented in both pure Python and C. This PEP requires
> +that in these instances that both the Python and C code *must* be
> +semantically identical (except in cases where implementation details
> +of a VM prevents it entirely). It is also required that new C-based
> +modules lacking a pure Python equivalent implementation get special
> +permissions to be added to the standard library.

I think it is worth stating explicitly that the C version can be even
a strict subset.  It is OK for the accelerated C code to rely on the
common python version; it is just the reverse that is not OK.

-jJ
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Policy for making changes to the AST

[Nick Coghlan]

On Tue, Apr 5, 2011 at 7:49 PM, "Martin v. Löwis"  wrote:
> Historically, that's incorrect. The AST structure was introduced to
> simplify the implementation of (C)Python. Exposing it to Python modules
> was done primarily because it's neat and was easy to do - not because
> any specific use was expected, and certainly not as the primary
> motivation for having an AST. It was clear (to me, at least) back then
> that the AST will change over time (hence the module was originally
> called _ast).

_ast is actually still there under the hood - the Python just added
some nicer tools for working with it.

It's probably worth mentioning the specific non-new-feature-related
changes to the AST that Eugene proposed on the tracker:

1. Making "docstring" an attribute of the Function node rather than
leaving it embedded as the first statement in the suite (this avoids
issues where AST-based constant folding could potentially corrupt the
docstring)
2. Collapsing Num, Str, Bytes, Ellipsis into a single Literal node
type (the handling of those nodes is the same in a lot of cases)
3. Since they're keywords now, pick up True, False, None at the
parsing stage and turn them into instances of the Literal node type,
allowing the current Name-based special casing to be removed.

These are the proposed changes that would be visible to someone using
the ast.PyCF_ONLY_AST flag. Any further changes (i.e. the actual
constant folding) aren't exposed in the AST produced by that flag -
those changes happen as a pre-transformation step in the process of
turning the submitted AST into CPython bytecode.

As I said in the tracker item, I actually *like* those 3 changes and
think they streamline the AST (and the associated code) quite nicely.
However, I didn't want to tell Eugene to proceed on that basis without
getting feedback from a wider audience first.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Buildbot status

[Antoine Pitrou]

Hello,

For the record, we have 9 stable buildbots, one of which is currently
offline: 3 Windows, 2 OS X, 3 Linux and 1 Solaris.
Paul Moore's XP buildbot is back in the stable stable.
(http://www.python.org/dev/buildbot/all/waterfall?category=3.x.stable)

We also have a new 64-bit FreeBSD 8.2 buildbot donated and managed by
Stefan Krah.
(http://www.python.org/dev/buildbot/all/buildslaves/krah-freebsd)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Brett Cannon]

On Tue, Apr 5, 2011 at 01:26, Stefan Behnel  wrote:

> Brett Cannon, 05.04.2011 01:46:
>
>  At both the VM and language summits at PyCon this year, the issue of
>> compatibility of the stdlib amongst the various VMs came up. Two issues
>> came
>> about in regards to modules that use C code. One is that code that comes
>> in
>> only as C code sucks for all other VMs that are not CPython since they all
>> end up having to re-implement that module themselves. Two is that modules
>> that have an accelerator module (e.g., heapq, warnings, etc.) can end up
>> with compatibility options (sorry, Raymond, for picking on heapq, but is
>> was
>> what bit the PyPy people most recently =).
>>
>> In lieu of all of this, here is a draft PEP to more clearly state the
>> policy
>> for the stdlib when it comes to C code. Since this has come up before and
>> this was discussed so much at the summits I have gone ahead and checked
>> this
>> in so that even if this PEP gets rejected there can be a written record as
>> to why.
>>
>> And before anyone asks, I have already run this past the lead devs of
>> PyPy,
>> Jython, and IronPython and they all support what this PEP proposes.
>>
>
> We recently had the discussion about reimplementing stdlib C modules in
> Cython. Accelerator modules are the obvious first step here, as they could
> be implemented in Python and compiled with Cython, instead of actually
> writing them in C in the first place. Wouldn't this be worth mentioning in
> the PEP?
>

I consider whether Cython is used orthogonal to the PEP. If Cython is
actually used and it is deemed needed I will update the PEP.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Brett Cannon]

On Tue, Apr 5, 2011 at 05:01, Nick Coghlan  wrote:

> On Tue, Apr 5, 2011 at 9:46 AM, Brett Cannon  wrote:
> > try:
> > c_heapq.heappop(Spam())
> > except TypeError:
> > # "heap argument must be a list"
> > pass
> >
> > try:
> > py_heapq.heappop(Spam())
> > except AttributeError:
> > # "'Foo' object has no attribute 'pop'"
> > pass
> >
> > This kind of divergence is a problem for users as they unwittingly
> > write code that is CPython-specific. This is also an issue for other
> > VM teams as they have to deal with bug reports from users thinking
> > that they incorrectly implemented the module when in fact it was
> > caused by an untested case.
>
> While I agree with the PEP in principle, I disagree with the way this
> example is written. Guido has stated in the past that code simply
> *cannot* rely on TypeError being consistently thrown instead of
> AttributeError (or vice-versa) when it comes to duck-typing. Code that
> cares which of the two is thrown is wrong.
>

Which is unfortunate since least common base class is Exception. But I can
add a note to the PEP saying that this is the case and change the example.


>
> However, there actually *is* a significant semantic discrepancy in the
> heapq case, which is that py_heapq is duck-typed, while c_heapq is
> not:
>
> >>> from test.support import import_fresh_module
> >>> c_heapq = import_fresh_module('heapq', fresh=['_heapq'])
> >>> py_heapq = import_fresh_module('heapq', blocked=['_heapq'])
> >>> from collections import UserList
> >>> class Seq(UserList): pass
> ...
> >>> c_heapq.heappop(UserList())
> Traceback (most recent call last):
>  File "", line 1, in 
> TypeError: heap argument must be a list
> >>> py_heapq.heappop(UserList())
> Traceback (most recent call last):
>  File "", line 1, in 
>  File "/home/ncoghlan/devel/py3k/Lib/heapq.py", line 140, in heappop
>lastelt = heap.pop()# raises appropriate IndexError if heap is empty
>  File "/home/ncoghlan/devel/py3k/Lib/collections/__init__.py", line 848, in
> pop
>def pop(self, i=-1): return self.data.pop(i)
> IndexError: pop from empty list
>
> Cheers,
> Nick.
>
> P.S. The reason I was bugging Guido to answer the TypeError vs
> AttributeError question in the first place was to find out whether or
> not I needed to get rid of the following gross inconsistency in the
> behaviour of the with statement relative to other language constructs:
>
> >>> 1()
> Traceback (most recent call last):
>  File "", line 1, in 
> TypeError: 'int' object is not callable
> >>> with 1: pass
> ...
> Traceback (most recent call last):
>  File "", line 1, in 
> AttributeError: 'int' object has no attribute '__exit__'
>
>
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 396, Module Version Numbers

[Barry Warsaw]

I just checked in PEP 396, Module Version Numbers.  This is an informational
PEP describing how to specify version numbers using the __version__
attribute.  This has already made one round through distutils-sig so it's time
to post it here.  Comments welcome of course!

Cheers,
-Barry

P.S. Yeah, I know the $Keyword$ strings here are wrong.  I'm not sure what to
do about them though.

PEP: 396
Title: Module Version Numbers
Version: $Revision: 65628 $
Last-Modified: $Date: 2008-08-10 09:59:20 -0400 (Sun, 10 Aug 2008) $
Author: Barry Warsaw 
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 2011-03-16
Post-History: 2011-04-05


Abstract


Given that it is useful and common to specify version numbers for
Python modules, and given that different ways of doing this have grown
organically within the Python community, it is useful to establish
standard conventions for module authors to adhere to and reference.
This informational PEP describes best practices for Python module
authors who want to define the version number of their Python module.

Conformance with this PEP is optional, however other Python tools
(such as ``distutils2`` [1]_) may be adapted to use the conventions
defined here.


User Stories


Alice is writing a new module, called ``alice``, which she wants to
share with other Python developers.  ``alice`` is a simple module and
lives in one file, ``alice.py``.  Alice wants to specify a version
number so that her users can tell which version they are using.
Because her module lives entirely in one file, she wants to add the
version number to that file.

Bob has written a module called ``bob`` which he has shared with many
users.  ``bob.py`` contains a version number for the convenience of
his users.  Bob learns about the Cheeseshop [2]_, and adds some simple
packaging using classic distutils so that he can upload *The Bob
Bundle* to the Cheeseshop.  Because ``bob.py`` already specifies a
version number which his users can access programmatically, he wants
the same API to continue to work even though his users now get it from
the Cheeseshop.

Carole maintains several namespace packages, each of which are
independently developed and distributed.  In order for her users to
properly specify dependencies on the right versions of her packages,
she specifies the version numbers in the namespace package's
``setup.py`` file.  Because Carol wants to have to update one version
number per package, she specifies the version number in her module and
has the ``setup.py`` extract the module version number when she builds
the *sdist* archive.

David maintains a package in the standard library, and also produces
standalone versions for other versions of Python.  The standard
library copy defines the version number in the module, and this same
version number is used for the standalone distributions as well.


Rationale
=

Python modules, both in the standard library and available from third
parties, have long included version numbers.  There are established
de-facto standards for describing version numbers, and many ad-hoc
ways have grown organically over the years.  Often, version numbers
can be retrieved from a module programmatically, by importing the
module and inspecting an attribute.  Classic Python distutils
``setup()`` functions [3]_ describe a ``version`` argument where the
release's version number can be specified.  PEP 8 [4]_ describes the
use of a module attribute called ``__version__`` for recording
"Subversion, CVS, or RCS" version strings using keyword expansion.  In
the PEP author's own email archives, the earliest example of the use
of an ``__version__`` module attribute by independent module
developers dates back to 1995.

Another example of version information is the sqlite3 [5]_ library
with its ``sqlite_version_info``, ``version``, and ``version_info``
attributes.  It may not be immediately obvious which attribute
contains a version number for the module, and which contains a version
number for the underlying SQLite3 library.

This informational PEP codifies established practice, and recommends
standard ways of describing module version numbers, along with some
use cases for when -- and when *not* -- to include them.  Its adoption
by module authors is purely voluntary; packaging tools in the standard
library will provide optional support for the standards defined
herein, and other tools in the Python universe may comply as well.


Specification
=

#. In general, modules in the standard library SHOULD NOT have version
   numbers.  They implicitly carry the version number of the Python
   release they are included in.

#. On a case-by-case basis, standard library modules which are also
   released in standalone form for other Python versions MAY include a
   module version number when included in the standard library, and
   SHOULD include a version number when packaged separately.

#. When a module includes a version number, it SHOULD be avai

Re: [Python-Dev] Supporting Visual Studio 2010

[Glyph Lefkowitz]

On Apr 5, 2011, at 8:52 AM, exar...@twistedmatrix.com wrote:

> On 09:58 am, mar...@v.loewis.de wrote:
>>> Won't that still be an issue despite the stable ABI? Extensions on
>>> Windows should be linked to the same version of MSVCRT used to compile
>>> Python
>> 
>> Not if they use the stable ABI. There still might be issues if you
>> mix CRTs, but none related to the Python ABI - in particular, none
>> of those crashing conditions can arise from the stable ABI.
> 
> Does this mean new versions of distutils let you build_ext with any C 
> compiler, instead of enforcing the same compiler as it has done previously?  
> That would be great.

That *would* be great.  But is it possible?

 says "functions expecting FILE* are 
not part of the ABI, to avoid depending on a specific version of the Microsoft 
C runtime DLL on Windows".  Can extension modules that need to read and write 
files practically avoid all of those functions?  (If your extension module 
links a library with a different CRT, but doesn't pass functions back and forth 
to Python, is that OK?)

The PEP also says that it will allow users to "check whether their modules 
conform to the ABI", but it doesn't say how that will be done.  How can we 
build extension modules so that we're sure we're ABI-conformant?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Raymond Hettinger]

[Brett]
> This PEP requires that in these instances that both
> the Python and C code must be semantically identical

Are you talking about the guaranteed semantics
promised by the docs or are you talking about
every possible implementation detail?

ISTM that even with pure python code, we get problems
with people relying on implementation specific details.

* Two functions accept a sequence, but one accesses
  it using __len__ and __getitem__ while the other
  uses __iter__.   (This is like the Spam example
  in the PEP).

* Given pure python library code like:
   if x < y: ...
  I've seen people only implement __lt__
  but not __gt__, making it impossible to
  make even minor adjustments to the code such as:
   if y > x:  ...

* We also suffer from inconsistency in choice of
  exceptions (i.e. overly large sequence indices
  raising either an IndexError, OverflowError, or
  ValueError).

With C code, I wonder if certain implementation
differences go with the territory:

* Concurrency issues are a common semantic difference.
  For example, deque.pop() is atomic because the C
  code holds the GIL but a pure python equivalent
  would have to use locks to achieve same effect
  (and even then might introduce liveness or deadlock
  issues).

* Heapq is one of the rare examples of purely
  algorithmic code.  Much of the code in CPython
  does accesses libraries (i.e. the math module),
  interfaces with the OS, access binary data
  structures, links to third-party tools (sqlite3
  and Tkinter) or does something else that doesn't
  have pure python equivalents (at least without
  using C types).

* The C API for parsing argument tuples and keywords
  do not readily parallel the way the same are
  written in Python.  And with iterators, the argument
  checking in the C versions tends to happen when the
  iterator is instantiated, but code written with
  pure python generators doesn't have its setup and
  checking section run until next() is called the 
  first time.

* We've had a very difficult time bridging the gulf
  between python's infinite precision numbers and
  and C's fixed width numbers (for example, it took
  years to get range() to handle values greater than
  a word size).

* C code tends to be written in a way that takes
  advantage of that language's features instead of
  in a form that is a direct translation of pure
  python.  For example, I think the work being done
  on a C implementation of decimal has vastly different
  internal structures and it would be a huge challenge
  to make it semantically identical to the pure python
  version with respect to its implementation details.
  Likewise, a worthwhile C implementation of OrderedDict
  can only achieve massive space savings by having
  majorly different implementation details.

Instead of expressing the wishful thought that C
versions and pure Python versions are semantically
identical with respect to implementation details,
I would like to see more thought put into specific
limitations on C coding techniques and general
agreement on which implementation specific details
should be guaranteed:

* I would like to see a restriction on the use of
  the concrete C API such that it is *only* used
  when a exact type match has been found or created
  (i.e. if someone writes Py_ListNew(), then it
  is okay to use Py_ListSetItem()).  See 
  http://bugs.python.org/issue10977 for a discussion
  of what can go wrong.  The original json C
  was an example of code that used the concrete
  C API is a way that precluded pure python
  subclasses of list and dict.

* I would like to see better consistency on when to 
  use OverflowError vs ValueError vs IndexError.

* There should also be a discussion of whether the
  possible exceptions should be a guaranteed part
  of the API as it is in Java.  Because there were
  no guarantees (i.e. ord(x) can raise this, that,
  and the other), people tend to run an experiment
  and then rely on whatever C Python happens to do.

* There should be a discussion on when it is okay
  for a C implementation to handle only a value
  range that fits in a word.

* When there is C code, when is it okay for a user
  to assume atomic access?  Even with pure python
  code, we're not always consistent about it 
  (i.e. OrderedDict implementation is not threadsafe
  but the LRU_Cache is).

* There should be some agreement that people 
  implementing rich comparisons will implement
  all six operations so that client code doesn't
  become dependent on (xx).  For
  example, we had to add special-case logic to
  heapq years ago because Twisted implemented
  a task object that defined __le__ instead of
  __lt__, so it was usable only with an older
  version of heapq but not with min, sort, etc.

A good PEP should address these issues head-on.
Just saying that C and python code have to
be semantically identical in all implementation
details doesn't really address the issue.


[Brett]
> (sorry, Raymond, for picking on heapq, but is
>

Re: [Python-Dev] Supporting Visual Studio 2010

[Martin v. Löwis]

>  says "functions expecting
> FILE* are not part of the ABI, to avoid depending on a specific
> version of the Microsoft C runtime DLL on Windows".  Can extension
> modules that need to read and write files practically avoid all of
> those functions?

Certainly! fread/fwrite/fprintf is not part of the Python API at all,
so clearly doesn't need to be part of the ABI.

> (If your extension module links a library with a
> different CRT, but doesn't pass functions back and forth to Python,
> is that OK?)

It is (and always was). The difficult functions are PyRun_AnyFileFlags
and friends.

> The PEP also says that it will allow users to "check whether their
> modules conform to the ABI", but it doesn't say how that will be
> done.  How can we build extension modules so that we're sure we're
> ABI-conformant?

If it compiles with Py_LIMITED_API defined, and links successfully
on Windows, it should be ABI-conforming (unless you deliberately
bypass the test, e.g. by replicating struct definitions in your
own code).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supporting Visual Studio 2010

[Martin v. Löwis]

> Does this mean new versions of distutils let you build_ext with any C
> compiler, instead of enforcing the same compiler as it has done
> previously? 

No, it doesn't. distutils was considered frozen, and changes to it to
better support the ABI where rejected.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 396, Module Version Numbers

[Glenn Linderman]

On 4/5/2011 11:52 AM, Barry Warsaw wrote:

#. Module version numbers SHOULD conform to the normalized version
format specified in PEP 386 [6]_.

From PEP 386:



  Roadmap 

Distutils will deprecate its existing versions class in favor of 
NormalizedVersion. The verlib module presented in this PEP will be 
renamed to version and placed into the distutils package.




With more standardization of versions, should the version module be 
promoted to stdlib directly?



On 4/5/2011 11:52 AM, Barry Warsaw wrote:

 DEFAULT_VERSION_RE = re.compile(r'(?P\d+\.\d(?:\.\d+)?)')
 __version__ = pkgutil.get_distribution('elle').metadata['version']


The RE as given won't match alpha, beta, rc, dev, and post suffixes that 
are discussed in POP 386.


Nor will it match the code shown and quoted for the alternative 
distutils2 case.



Other comments:

Are there issues for finding and loading multiple versions of the same 
module?


Should it be possible to determine a version before loading a module?  
If yes, the version module would have to be able to find a parse version 
strings in any of the many places this PEP suggests they could be... so 
that would be somewhat complex, but the complexity shouldn't be used to 
change the answer... but if the answer is yes, it might encourage fewer 
variant cases to be supported for acceptable version definition 
locations for this PEP.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supporting Visual Studio 2010

[Greg Ewing]

Martin v. Löwis wrote:

Not if they use the stable ABI. There still might be issues if you
mix CRTs, but none related to the Python ABI - in particular, none
of those crashing conditions can arise from the stable ABI.


Won't there still be a problem of your extension module
being linked with a CRT that may not be present on the
target system?

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Policy for making changes to the AST

[Greg Ewing]

Nick Coghlan wrote:


1. Making "docstring" an attribute of the Function node rather than
leaving it embedded as the first statement in the suite (this avoids
issues where AST-based constant folding could potentially corrupt the
docstring)
2. Collapsing Num, Str, Bytes, Ellipsis into a single Literal node
type (the handling of those nodes is the same in a lot of cases)
3. Since they're keywords now, pick up True, False, None at the
parsing stage and turn them into instances of the Literal node type,
allowing the current Name-based special casing to be removed.


These all sound good to me.

--
Greg


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supporting Visual Studio 2010

[Martin v. Löwis]

Am 05.04.2011 22:43, schrieb Greg Ewing:
> Martin v. Löwis wrote:
>> Not if they use the stable ABI. There still might be issues if you
>> mix CRTs, but none related to the Python ABI - in particular, none
>> of those crashing conditions can arise from the stable ABI.
> 
> Won't there still be a problem of your extension module
> being linked with a CRT that may not be present on the
> target system?

Certainly. Anybody packaging an extension module needs to make sure
all libraries it uses are either already on the target system, or
delivered along with the extension module. Developers could refer
users to the redist package, or they could literally include the CRT
with their package (which is easier with VS2010 than it was with
VS2008).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue 11715: building Python from source on multiarch Debian/Ubuntu

[Barry Warsaw]

On Apr 02, 2011, at 10:55 AM, Stefan Krah wrote:

>In this case, it's clearly Ubuntu who is going to break things. Still,
>the proposed patch could make life a lot easier for many people.

I'd be more concerned about adding some Debian/Ubuntu special code to setup.py
if it wasn't already a rats nest of specialization.  

$ grep darwin setup.py | wc -l
41

Not to mention the checks for osf1, unixware7, and openunix8 (!).  It's a
little ugly trying to run dpkg-architecture on every platform, but I'm not
sure anything better can be done, and of course it only needs to run it once.

I'm going to apply the patch to Python 2.7, 3.1, 3.2, and 3.3 tomorrow.

Cheers,
-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Supporting Visual Studio 2010

[exarkun]

On 5 Apr, 07:58 pm, mar...@v.loewis.de wrote:

Does this mean new versions of distutils let you build_ext with any C
compiler, instead of enforcing the same compiler as it has done
previously?


No, it doesn't. distutils was considered frozen, and changes to it to
better support the ABI where rejected.


How about distutils2 then?

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements

[Terry Reedy]

On 4/5/2011 3:57 PM, Raymond Hettinger wrote:

[Brett]

This PEP requires that in these instances that both
the Python and C code must be semantically identical


Are you talking about the guaranteed semantics
promised by the docs or are you talking about
every possible implementation detail?


I personally would limit the guarantee to what the docs promise. That is 
all people should expect anyway if the Python code were executed by some 
other implementation, or by someone else's system-coded version, or even 
a different version of CPython.


This assumes that the docs have reasonably complete specifications. The 
was improved in 3.2 and should improve further as system-code 
implementers find more holes.


Exceptions are a bit of a gray area. The docs are quite uneven about 
specifying exceptions. They sometimes do, sometimes do not, even for 
similar functions. This should be another PEP though.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com