Re: [Python-Dev] 2.7.5 baking

2013-05-15 Thread M.-A. Lemburg 
On 12.05.2013 06:03, Benjamin Peterson wrote:
> The long anticipated "emergency" 2.7.5 release has now been tagged. It
> will be publicly announced as binaries arrive.
> 
> Originally, I was just going to cherrypick regression fixes onto the
> 2.7.4 release and release those as 2.7.5. I started to this but ran
> into some conflicts. Since we don't have buildbot testing of release
> branches, I decided it would be best to just cut from the maintenance
> branch.

Has the release been postponed ?

I don't see it on http://www.python.org/download/

Incidentally, the schedule already lists 2.7.5 as released on
2013-05-12 (http://www.python.org/dev/peps/pep-0373/) and
the release calendar on 2013-05-11:
https://www.google.com/calendar/feeds/b6v58qvojllt0i6ql654r1v...@group.calendar.google.com/public/basic?orderby=starttime&sortorder=descending
:-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 15 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46
2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More compact dictionaries with faster iteration

2013-05-15 Thread Christian Tismer 

Hi Raymond,

On 08.01.13 15:49, Maciej Fijalkowski wrote:

On Mon, Dec 10, 2012 at 3:44 AM, Raymond Hettinger
 wrote:

The current memory layout for dictionaries is
unnecessarily inefficient.  It has a sparse table of
24-byte entries containing the hash value, key pointer,
and value pointer.

Instead, the 24-byte entries should be stored in a
dense table referenced by a sparse table of indices.

For example, the dictionary:

 d = {'timmy': 'red', 'barry': 'green', 'guido': 'blue'}

is currently stored as:

 entries = [['--', '--', '--'],
[-8522787127447073495, 'barry', 'green'],
['--', '--', '--'],
['--', '--', '--'],
['--', '--', '--'],
[-9092791511155847987, 'timmy', 'red'],
['--', '--', '--'],
[-6480567542315338377, 'guido', 'blue']]

Instead, the data should be organized as follows:

 indices =  [None, 1, None, None, None, 0, None, 2]
 entries =  [[-9092791511155847987, 'timmy', 'red'],
 [-8522787127447073495, 'barry', 'green'],
 [-6480567542315338377, 'guido', 'blue']]

Only the data layout needs to change.  The hash table
algorithms would stay the same.  All of the current
optimizations would be kept, including key-sharing
dicts and custom lookup functions for string-only
dicts.  There is no change to the hash functions, the
table search order, or collision statistics.

The memory savings are significant (from 30% to 95%
compression depending on the how full the table is).
Small dicts (size 0, 1, or 2) get the most benefit.

For a sparse table of size t with n entries, the sizes are:

 curr_size = 24 * t
 new_size = 24 * n + sizeof(index) * t

In the above timmy/barry/guido example, the current
size is 192 bytes (eight 24-byte entries) and the new
size is 80 bytes (three 24-byte entries plus eight
1-byte indices).  That gives 58% compression.

Note, the sizeof(index) can be as small as a single
byte for small dicts, two bytes for bigger dicts and
up to sizeof(Py_ssize_t) for huge dict.

In addition to space savings, the new memory layout
makes iteration faster.  Currently, keys(), values, and
items() loop over the sparse table, skipping-over free
slots in the hash table.  Now, keys/values/items can
loop directly over the dense table, using fewer memory
accesses.

Another benefit is that resizing is faster and
touches fewer pieces of memory.  Currently, every
hash/key/value entry is moved or copied during a
resize.  In the new layout, only the indices are
updated.  For the most part, the hash/key/value entries
never move (except for an occasional swap to fill a
hole left by a deletion).

With the reduced memory footprint, we can also expect
better cache utilization.

For those wanting to experiment with the design,
there is a pure Python proof-of-concept here:

http://code.activestate.com/recipes/578375

YMMV: Keep in mind that the above size statics assume a
build with 64-bit Py_ssize_t and 64-bit pointers.  The
space savings percentages are a bit different on other
builds.  Also, note that in many applications, the size
of the data dominates the size of the container (i.e.
the weight of a bucket of water is mostly the water,
not the bucket).


Raymond
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

One question Raymond.

The compression ratios stay true provided you don't overallocate entry
list. If you do overallocate you don't really gain that much (it all
depends vastly on details), or even loose in some cases. What do you
think should the strategy be?



What is the current status of this discussion?
I'd like to know whether it is a considered alternative implementation.

There is also a discussion in python-ideas right now where this
alternative is mentioned, and I think especially for small dicts
as **kwargs, it could be a cheap way to introduce order.

Is this going on, somewhere? I'm quite interested on that.

cheers - chris

--
Christian Tismer :^)   
Software Consulting  : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More compact dictionaries with faster iteration

2013-05-15 Thread Stefan Drees 

Hi Chris,

On 15.05.13 13:32 Christian Tismer wrote:

Hi Raymond,

On 08.01.13 15:49, Maciej Fijalkowski wrote:

On Mon, Dec 10, 2012 at 3:44 AM, Raymond Hettinger
 wrote:

The current memory layout for dictionaries is
unnecessarily inefficient.  It has a sparse table of
24-byte entries containing the hash value, key pointer,
and value pointer.

...




What is the current status of this discussion?
I'd like to know whether it is a considered alternative implementation.

There is also a discussion in python-ideas right now where this
alternative is mentioned, and I think especially for small dicts
as **kwargs, it could be a cheap way to introduce order.

Is this going on, somewhere? I'm quite interested on that.


+1 I am also interested on the status. Many people seemed to have copied 
the recipe from the activestate site (was it?) but I wonder if it maybe 
was to cool to be progressed into "the field" or simply some 
understandable lack of resources?


All the best,
Stefan

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More compact dictionaries with faster iteration

2013-05-15 Thread Christian Tismer 

On 15.05.13 14:01, Stefan Drees wrote:

Hi Chris,

On 15.05.13 13:32 Christian Tismer wrote:

Hi Raymond,

On 08.01.13 15:49, Maciej Fijalkowski wrote:

On Mon, Dec 10, 2012 at 3:44 AM, Raymond Hettinger
 wrote:

The current memory layout for dictionaries is
unnecessarily inefficient.  It has a sparse table of
24-byte entries containing the hash value, key pointer,
and value pointer.

...




What is the current status of this discussion?
I'd like to know whether it is a considered alternative implementation.

There is also a discussion in python-ideas right now where this
alternative is mentioned, and I think especially for small dicts
as **kwargs, it could be a cheap way to introduce order.

Is this going on, somewhere? I'm quite interested on that.


+1 I am also interested on the status. Many people seemed to have 
copied the recipe from the activestate site (was it?) but I wonder if 
it maybe was to cool to be progressed into "the field" or simply some 
understandable lack of resources?




Right, found the references:
http://mail.python.org/pipermail/python-dev/2012-December/123028.html
http://stackoverflow.com/questions/14664620/python-dictionary-details
http://code.activestate.com/recipes/578375-proof-of-concept-for-a-more-space-efficient-faster/?in=user-178123

cheers - chris

--
Christian Tismer :^)   
Software Consulting  : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More compact dictionaries with faster iteration

2013-05-15 Thread Maciej Fijalkowski 
On Wed, May 15, 2013 at 2:36 PM, Christian Tismer  wrote:
> On 15.05.13 14:01, Stefan Drees wrote:
>>
>> Hi Chris,
>>
>> On 15.05.13 13:32 Christian Tismer wrote:
>>>
>>> Hi Raymond,
>>>
>>> On 08.01.13 15:49, Maciej Fijalkowski wrote:

 On Mon, Dec 10, 2012 at 3:44 AM, Raymond Hettinger
  wrote:
>
> The current memory layout for dictionaries is
> unnecessarily inefficient.  It has a sparse table of
> 24-byte entries containing the hash value, key pointer,
> and value pointer.
>
> ...


>>>
>>> What is the current status of this discussion?
>>> I'd like to know whether it is a considered alternative implementation.
>>>
>>> There is also a discussion in python-ideas right now where this
>>> alternative is mentioned, and I think especially for small dicts
>>> as **kwargs, it could be a cheap way to introduce order.
>>>
>>> Is this going on, somewhere? I'm quite interested on that.
>>
>>
>> +1 I am also interested on the status. Many people seemed to have copied
>> the recipe from the activestate site (was it?) but I wonder if it maybe was
>> to cool to be progressed into "the field" or simply some understandable lack
>> of resources?
>>
>
> Right, found the references:
> http://mail.python.org/pipermail/python-dev/2012-December/123028.html
> http://stackoverflow.com/questions/14664620/python-dictionary-details
> http://code.activestate.com/recipes/578375-proof-of-concept-for-a-more-space-efficient-faster/?in=user-178123
>
>
> cheers - chris
>
> --
> Christian Tismer :^)   
> Software Consulting  : Have a break! Take a ride on Python's
> Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
> 14482 Potsdam: PGP key -> http://pgp.uni-mainz.de
> phone +49 173 24 18 776  fax +49 (30) 700143-0023
> PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
>   whom do you want to sponsor today?   http://www.stackless.com/
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

I implemented one for pypy btw (it's parked on a branch for messiness reasons)

Cheers,
fijal
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7.5 baking

2013-05-15 Thread Benjamin Peterson 
2013/5/15 M.-A. Lemburg :
> On 12.05.2013 06:03, Benjamin Peterson wrote:
>> The long anticipated "emergency" 2.7.5 release has now been tagged. It
>> will be publicly announced as binaries arrive.
>>
>> Originally, I was just going to cherrypick regression fixes onto the
>> 2.7.4 release and release those as 2.7.5. I started to this but ran
>> into some conflicts. Since we don't have buildbot testing of release
>> branches, I decided it would be best to just cut from the maintenance
>> branch.
>
> Has the release been postponed ?
>
> I don't see it on http://www.python.org/download/

We're waiting for binaries.

>
> Incidentally, the schedule already lists 2.7.5 as released on
> 2013-05-12 (http://www.python.org/dev/peps/pep-0373/) and
> the release calendar on 2013-05-11:
> https://www.google.com/calendar/feeds/b6v58qvojllt0i6ql654r1v...@group.calendar.google.com/public/basic?orderby=starttime&sortorder=descending
> :-)

In practice, those dates mean when I tag the release.


--
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7.5 baking

2013-05-15 Thread Georg Brandl 
Am 15.05.2013 09:55, schrieb M.-A. Lemburg:
> On 12.05.2013 06:03, Benjamin Peterson wrote:
>> The long anticipated "emergency" 2.7.5 release has now been tagged. It
>> will be publicly announced as binaries arrive.
>> 
>> Originally, I was just going to cherrypick regression fixes onto the
>> 2.7.4 release and release those as 2.7.5. I started to this but ran
>> into some conflicts. Since we don't have buildbot testing of release
>> branches, I decided it would be best to just cut from the maintenance
>> branch.
> 
> Has the release been postponed ?
> 
> I don't see it on http://www.python.org/download/
> 
> Incidentally, the schedule already lists 2.7.5 as released on
> 2013-05-12 (http://www.python.org/dev/peps/pep-0373/) and
> the release calendar on 2013-05-11:
> https://www.google.com/calendar/feeds/b6v58qvojllt0i6ql654r1v...@group.calendar.google.com/public/basic?orderby=starttime&sortorder=descending
> :-)
> 

We're still waiting for the Windows binaries.

I think I will publish the source and Mac releases on the website now
and make a note that Windows is coming shortly.

Has anybody heard from Martin recently?  I hope he's well and just
overworked...

Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7.5 baking

2013-05-15 Thread Brian Curtin 
On Wed, May 15, 2013 at 1:07 PM, Georg Brandl  wrote:
> Am 15.05.2013 09:55, schrieb M.-A. Lemburg:
>> On 12.05.2013 06:03, Benjamin Peterson wrote:
>>> The long anticipated "emergency" 2.7.5 release has now been tagged. It
>>> will be publicly announced as binaries arrive.
>>>
>>> Originally, I was just going to cherrypick regression fixes onto the
>>> 2.7.4 release and release those as 2.7.5. I started to this but ran
>>> into some conflicts. Since we don't have buildbot testing of release
>>> branches, I decided it would be best to just cut from the maintenance
>>> branch.
>>
>> Has the release been postponed ?
>>
>> I don't see it on http://www.python.org/download/
>>
>> Incidentally, the schedule already lists 2.7.5 as released on
>> 2013-05-12 (http://www.python.org/dev/peps/pep-0373/) and
>> the release calendar on 2013-05-11:
>> https://www.google.com/calendar/feeds/b6v58qvojllt0i6ql654r1v...@group.calendar.google.com/public/basic?orderby=starttime&sortorder=descending
>> :-)
>>
>
> We're still waiting for the Windows binaries.
>
> I think I will publish the source and Mac releases on the website now
> and make a note that Windows is coming shortly.

I'm going to get started building the MSIs this evening. I'm looking
into how I can obtain a code signing certificate, otherwise we'd
potentially be shipping unsigned security releases...*ducks*

> Has anybody heard from Martin recently?  I hope he's well and just
> overworked...

I asked some folks on the infrastructure team and the last they heard
from him was 11 April.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7.5 baking

2013-05-15 Thread Zachary Ware 
> I asked some folks on the infrastructure team and the last they heard
> from him was 11 April.

Martin replied on issue17883 on May 10.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7.5 baking

2013-05-15 Thread M.-A. Lemburg 
On 15.05.2013 19:11, Benjamin Peterson wrote:
> 2013/5/15 M.-A. Lemburg :
>> On 12.05.2013 06:03, Benjamin Peterson wrote:
>>> The long anticipated "emergency" 2.7.5 release has now been tagged. It
>>> will be publicly announced as binaries arrive.
>>>
>>> Originally, I was just going to cherrypick regression fixes onto the
>>> 2.7.4 release and release those as 2.7.5. I started to this but ran
>>> into some conflicts. Since we don't have buildbot testing of release
>>> branches, I decided it would be best to just cut from the maintenance
>>> branch.
>>
>> Has the release been postponed ?
>>
>> I don't see it on http://www.python.org/download/
> 
> We're waiting for binaries.

Ah, ok. Thanks for the heads-up.

>> Incidentally, the schedule already lists 2.7.5 as released on
>> 2013-05-12 (http://www.python.org/dev/peps/pep-0373/) and
>> the release calendar on 2013-05-11:
>> https://www.google.com/calendar/feeds/b6v58qvojllt0i6ql654r1v...@group.calendar.google.com/public/basic?orderby=starttime&sortorder=descending
>> :-)
> 
> In practice, those dates mean when I tag the release.

Ok. Was just wondering whether something went wrong with the website.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 15 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...   http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

2013-05-07: Released mxODBC Zope DA 2.1.2 ... http://egenix.com/go46
2013-05-06: Released mxODBC 3.2.3 ... http://egenix.com/go45

: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Mysterious Python pyc file corruption problems

2013-05-15 Thread Barry Warsaw 
I am looking into a particularly vexing Python problem on Ubuntu that
manifests in several different ways.  I think the problem is the same one
described in http://bugs.python.org/issue13146 and I sent a message on the
subject to the ubuntu-devel list:
https://lists.ubuntu.com/archives/ubuntu-devel/2013-May/037129.html

I don't know what's causing the problem and have no way to reproduce it, but
all the clues point to corrupt pyc files in Pythons < 3.3.

The common way this manifests is a traceback on an import statement.  The
actual error can be a "ValueError: bad marshal data (unknown type code)" such
as in http://pad.lv/1010077 or an "EOFError: EOF read where not expected" as
in http://pad.lv/1060842.  We have many more instances of both of these.

Since both error messages come from marshal.c when trying to read the pyc for
a module being imported, I suspect that something is causing the pyc files to
get partially overwritten or corrupted.  The workaround is always to
essentially blow away the .pyc file and re-create it.  (Various different
techniques can be used, but they all boil down to the same thing.)

Another commonality is that this bug -- so far -- has not been observed in any
Python 3.3 code, only 3.2 and earlier, including 2.7 and 2.6.  This
strengthens my hypothesis, since importlib in Python 3.3 included an atomic
rename of the .pyc file whereas older Pythons only do an exclusive open on the
pyc files, but do *not* do an atomic rename AFAICT.

This leads me to hypothesize that the bug is due to an as yet unidentified
race condition during installation of Python source code on Ubuntu, which is
normally when we automatically byte compile the source to .pyc files.  This
can happen at package installation/upgrade time, or during a fresh install.
In each of these cases there *should* be only one process attempting to write
the .pyc, but my guess is that for some reason, multiple processes are trying
to do this, triggering a truncation or other bogus content of .pyc files.
Even in Python < 3.3, it should not be possible to corrupt a .pyc when only a
single process is involved, due to the import lock and/or GIL.  The exclusive
open of the .pyc file is clearly not enough of a protection in a multiprocess
situation, since the bug has already been identified in Python on buildbots
during test_multiprocessing.  See http://bugs.python.org/issue13146

I think the list of errors we've seen is too extensive to chalk up to a
hardware bug, and I think the systems involved are modern enough to not be
subject to file system data loss.  There could be a missing fsync somewhere
though that might be involved.  I think it's doubtful that buggy remote file
systems (e.g. NFSv2) are involved.  I could be wrong about any of that.

I have not succeeded in writing a standalone reproducer using Python 2.7.

So, the mystery is: what process on Ubuntu is exploiting holes in the
exclusive open and causing this problem?

Issue 13146 is closed because the fix was applied to Python 3.3 (see above),
but it was not backported to earlier versions.  I think it would not be that
difficult to backport it, and I would be willing to do so for Python 2.7 and
3.2.  We might include 2.6 in that, but only in Ubuntu since I can't see how
this bug could be exploited as a security vulnerability.

Thoughts?

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How to debug python crashes

2013-05-15 Thread Steve Dower 
> From: Catalin Iacob
> Hi Philippe,
> 
> I don't have access to VS right now but out of my head what you need
> to do is roughly outlined below.
> 
> On Tue, May 14, 2013 at 5:29 PM, Philippe Fremy 
> wrote:
> > But what's the reason for releasing them ? If you need to recompile
> > Python to use them, that would be strange because they are generated as
> > part of the compilation process anyway.
> 
> They can indeed be used like this:
> 
> You should launch the python.exe process that is going to crash,
> attach to it with the Visual Studio debugger and then reproduce the
> crash. This should drop you in the debugger.
> 
> Once you're in the debugger and python.exe is stopped at the point of
> the crash you should see the stack trace of each thread in a VS
> window, the stacktrace will probably have lots of entries of the form
> python27.dll! (no function names because there VS doesn't
> know where to find the PDB files). If you right click one of those
> entries there's an option named "Symbol load information" or similar,
> this will show a window from which you can make VS ask you where on
> disk do you have PDB files. You then tell VS where to find
> python27.pdb and then the stacktrace entries should automatically get
> function names.

Copying the .pdb files to the same directories as the matching DLL/EXE files 
(which may be C:\Windows\System32 or C:\Windows\SysWOW64 for python27.dll) 
should also make this work. VS will always look next to the executable file.

Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mysterious Python pyc file corruption problems

2013-05-15 Thread Brett Cannon 
On Wed, May 15, 2013 at 4:58 PM, Barry Warsaw  wrote:
> I am looking into a particularly vexing Python problem on Ubuntu that
> manifests in several different ways.  I think the problem is the same one
> described in http://bugs.python.org/issue13146 and I sent a message on the
> subject to the ubuntu-devel list:
> https://lists.ubuntu.com/archives/ubuntu-devel/2013-May/037129.html
>
> I don't know what's causing the problem and have no way to reproduce it, but
> all the clues point to corrupt pyc files in Pythons < 3.3.
>
> The common way this manifests is a traceback on an import statement.  The
> actual error can be a "ValueError: bad marshal data (unknown type code)" such
> as in http://pad.lv/1010077 or an "EOFError: EOF read where not expected" as
> in http://pad.lv/1060842.  We have many more instances of both of these.
>
> Since both error messages come from marshal.c when trying to read the pyc for
> a module being imported, I suspect that something is causing the pyc files to
> get partially overwritten or corrupted.  The workaround is always to
> essentially blow away the .pyc file and re-create it.  (Various different
> techniques can be used, but they all boil down to the same thing.)
>
> Another commonality is that this bug -- so far -- has not been observed in any
> Python 3.3 code, only 3.2 and earlier, including 2.7 and 2.6.  This
> strengthens my hypothesis, since importlib in Python 3.3 included an atomic
> rename of the .pyc file whereas older Pythons only do an exclusive open on the
> pyc files, but do *not* do an atomic rename AFAICT.

Just an FYI, the renaming has caught at least one person off-guard:
http://bugs.python.org/issue17222, so you might have to be careful
about considering a backport.

-Brett
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mysterious Python pyc file corruption problems

2013-05-15 Thread Tres Seaver 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 05/15/2013 04:58 PM, Barry Warsaw wrote:
> This leads me to hypothesize that the bug is due to an as yet
> unidentified race condition during installation of Python source code
> on Ubuntu, which is normally when we automatically byte compile the
> source to .pyc files.

Any chance you are using 'detox' or the equivalent to run tests on
mutliple interpreters in parallel?  The only "bad marshall data" errors I
have seen lately seemed to be provoked by that kind of practice.



Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlGUBvkACgkQ+gerLs4ltQ7nCwCcCfcAEGEN26qjQ9sGPaFRx1o4
DhwAoIlNwVU2lcJQ/hs5vQ1PXYT1uUwl
=0s+X
-END PGP SIGNATURE-

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mysterious Python pyc file corruption problems

2013-05-15 Thread Nick Coghlan 
On 16 May 2013 08:11, "Tres Seaver"  wrote:
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 05/15/2013 04:58 PM, Barry Warsaw wrote:
> > This leads me to hypothesize that the bug is due to an as yet
> > unidentified race condition during installation of Python source code
> > on Ubuntu, which is normally when we automatically byte compile the
> > source to .pyc files.
>
> Any chance you are using 'detox' or the equivalent to run tests on
> mutliple interpreters in parallel?  The only "bad marshall data" errors I
> have seen lately seemed to be provoked by that kind of practice.

3.2 shouldn't have a problem with that if the interpreters are different
versions.

Personally, I would be suspicious of developmental web services doing
auto-reloading while an installer is recompiling the world. I don't have
enough context to be sure how plausible that is as a possible explanation,
though.

Cheers,
Nick.

>
>
>
> Tres.
> - --
> ===
> Tres Seaver  +1 540-429-0999  tsea...@palladion.com
> Palladion Software   "Excellence by Design"http://palladion.com
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with undefined - http://www.enigmail.net/
>
> iEYEARECAAYFAlGUBvkACgkQ+gerLs4ltQ7nCwCcCfcAEGEN26qjQ9sGPaFRx1o4
> DhwAoIlNwVU2lcJQ/hs5vQ1PXYT1uUwl
> =0s+X
> -END PGP SIGNATURE-
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7.5 baking

2013-05-15 Thread Martin v. Löwis 
Am 15.05.13 20:07, schrieb Georg Brandl:
> Has anybody heard from Martin recently?  I hope he's well and just
> overworked...

True on both accounts. I was travelling over the weekend, and then
didn't manage to catch up with email. Sorry for the delay.

Regards,
Martin


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [RELEASED] Python 2.7.5

2013-05-15 Thread Benjamin Peterson 
It is my greatest pleasure to announce the release of Python 2.7.5.

2.7.5 is the latest maintenance release in the Python 2.7 series. You may be
surprised to hear from me so soon, as Python 2.7.4 was released slightly more
than a month ago. As it turns out, 2.7.4 had several regressions and
incompatibilities with 2.7.3. Among them were regressions in the zipfile, gzip,
and logging modules. 2.7.5 fixes these. In addition, a data file for testing in
the 2.7.4 tarballs and binaries aroused the suspicion of some virus
checkers. The 2.7.5 release removes this file to resolve that issue.

For details, see the Misc/NEWS file in the distribution or view it at

http://hg.python.org/cpython/file/ab05e7dd2788/Misc/NEWS

Downloads are at

http://python.org/download/releases/2.7.5/

As always, please report bugs to

http://bugs.python.org/

(Thank you to those who reported these bugs in 2.7.4.)

This is a production release.

Happy May,
Benjamin Peterson
2.7 Release Manager
(on behalf of all of Python 2.7's contributors)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASED] Python 2.7.5

2013-05-15 Thread Carlos Nepomuceno 
test_asynchat still hangs! What it does? Should I care?


> Date: Wed, 15 May 2013 23:19:06 -0500
> Subject: [RELEASED] Python 2.7.5
> From: benja...@python.org
> To: python-dev@python.org; python-l...@python.org; 
> python-announce-l...@python.org
>
> It is my greatest pleasure to announce the release of Python 2.7.5.
>
> 2.7.5 is the latest maintenance release in the Python 2.7 series. You may be
> surprised to hear from me so soon, as Python 2.7.4 was released slightly more
> than a month ago. As it turns out, 2.7.4 had several regressions and
> incompatibilities with 2.7.3. Among them were regressions in the zipfile, 
> gzip,
> and logging modules. 2.7.5 fixes these. In addition, a data file for testing 
> in
> the 2.7.4 tarballs and binaries aroused the suspicion of some virus
> checkers. The 2.7.5 release removes this file to resolve that issue.
>
> For details, see the Misc/NEWS file in the distribution or view it at
>
> http://hg.python.org/cpython/file/ab05e7dd2788/Misc/NEWS
>
> Downloads are at
>
> http://python.org/download/releases/2.7.5/
>
> As always, please report bugs to
>
> http://bugs.python.org/
>
> (Thank you to those who reported these bugs in 2.7.4.)
>
> This is a production release.
>
> Happy May,
> Benjamin Peterson
> 2.7 Release Manager
> (on behalf of all of Python 2.7's contributors)
> --
> http://mail.python.org/mailman/listinfo/python-list   
>   
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASED] Python 2.7.5

2013-05-15 Thread Benjamin Peterson 
2013/5/15 Carlos Nepomuceno :
> test_asynchat still hangs! What it does? Should I care?

Is there an issue filed for that?



--
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASED] Python 2.7.5

2013-05-15 Thread Carlos Nepomuceno 
Just filed 17992!

http://bugs.python.org/issue17992


> Date: Wed, 15 May 2013 23:51:00 -0500
> Subject: Re: [Python-Dev] [RELEASED] Python 2.7.5
> From: benja...@python.org
> To: carlosnepomuc...@outlook.com
> CC: python-dev@python.org
>
> 2013/5/15 Carlos Nepomuceno :
>> test_asynchat still hangs! What it does? Should I care?
>
> Is there an issue filed for that?
>
>
>
> --
> Regards,
> Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [RELEASED] Python 3.2.5 and Python 3.3.2

2013-05-15 Thread Georg Brandl 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On behalf of the Python development team, I am pleased to announce the
releases of Python 3.2.5 and 3.3.2.

The releases fix a few regressions in 3.2.4 and 3.3.1 in the zipfile, gzip
and xml.sax modules.  Details can be found in the changelogs:

http://hg.python.org/cpython/file/v3.2.5/Misc/NEWS  and
http://hg.python.org/cpython/file/v3.3.2/Misc/NEWS

To download Python 3.2.5 or Python 3.3.2, visit:

http://www.python.org/download/releases/3.2.5/  or
http://www.python.org/download/releases/3.3.2/

respectively.  As always, please report bugs to

http://bugs.python.org/

(Thank you to those who reported these regressions.)

Enjoy!

- -- 
Georg Brandl, Release Manager
georg at python.org
(on behalf of the entire python-dev team and all contributors)
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.19 (GNU/Linux)

iEYEARECAAYFAlGUbJ4ACgkQN9GcIYhpnLDH8ACdEM4k7bobLJsFmCb49zuwQR3W
EjgAoIWAOFNhJNdTAWEGSWqFWUP20wrb
=YnPr
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASED] Python 2.7.5

2013-05-15 Thread Ben Hoyt 
Thanks, Benjamin -- that's great!

This may not be a python-dev question exactly. But on Windows, is it safe
to update to 2.7.5 on top of 2.7.4 (at C:\Python27) using the .msi
installer? In other words, will it update/add/remove all the files
correctly? What if python.exe is running?

-Ben


On Thu, May 16, 2013 at 4:19 PM, Benjamin Peterson wrote:

> It is my greatest pleasure to announce the release of Python 2.7.5.
>
> 2.7.5 is the latest maintenance release in the Python 2.7 series. You may
> be
> surprised to hear from me so soon, as Python 2.7.4 was released slightly
> more
> than a month ago. As it turns out, 2.7.4 had several regressions and
> incompatibilities with 2.7.3. Among them were regressions in the zipfile,
> gzip,
> and logging modules. 2.7.5 fixes these. In addition, a data file for
> testing in
> the 2.7.4 tarballs and binaries aroused the suspicion of some virus
> checkers. The 2.7.5 release removes this file to resolve that issue.
>
> For details, see the Misc/NEWS file in the distribution or view it at
>
> http://hg.python.org/cpython/file/ab05e7dd2788/Misc/NEWS
>
> Downloads are at
>
> http://python.org/download/releases/2.7.5/
>
> As always, please report bugs to
>
> http://bugs.python.org/
>
> (Thank you to those who reported these bugs in 2.7.4.)
>
> This is a production release.
>
> Happy May,
> Benjamin Peterson
> 2.7 Release Manager
> (on behalf of all of Python 2.7's contributors)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com