Re: [Python-Dev] Module renaming and pickle mechanisms
On 2008-05-17 16:59, Alexandre Vassalotti wrote: On Sat, May 17, 2008 at 5:05 AM, M.-A. Lemburg <[EMAIL PROTECTED]> wrote: I'd like to bring a potential problem to attention that is caused by the recent module renaming approach: Object serialization protocols like e.g. pickle usually store the complete module path to the object class together with the object. Thanks for bringing this up. I was aware of the problem myself, but I hadn't yet worked out a good solution to it. It can also happen in storage setups where Python objects are stored using e.g. pickle, ZODB being a prominent example. As soon as a Python 2.6 application starts writing to such storages, Python 2.5 and lower versions will no longer be able to read back all the data. The opposite problem exists for Python 3.0, too. Pickle streams written by Python 2.x applications will not be readable by Python 3.0. And, one solution to this is to use Python 2.6 to regenerate pickle stream. Another solution would be to write a 2to3 pickle converter using the pickletools module. It is surely not the most elegant or robust solution, but I could work. I'm not really worried much about going from 2.x to 3.x. Breakage is allowed for that transition. However, the case is different for going from 2.5 to 2.6. Breakage should be avoided if at all possible. Now, I think there's a way to solve this puzzle: Instead of renaming the modules (e.g. Queue -> queue), we leave the code in the existing modules and packages and instead add the new module names and package structure with pointers and redirects to the existing 2.5 modules. This would certainly work for simple modules, but what about packages? For packages, you can't use the ``sys.modules[__name__] = Queue`` to preserve module identity. Therefore, pickle will use the new package name when writing its streams. So, we are back to the same problem again. A possible solution could be writing a compatibility layer for the Pickler class, which would map new module names to their old at runtime. Again, this is neither an elegant, nor robust, solution, but it should work in most cases. While it's possible to fix pickle (at least the Python version), this would not help with other serialization formats that rely on the .__module__ attribute mapping to an existing module. It's better to address the problem at the module level. Perhaps I have a misunderstanding of the reasoning behind doing the renaming in the 2.x branch, but it appears that the only reason is to get used to the new names. That's a rather low priority argument in comparison to the breakage the renaming will cause in the 2.x branch. I think it's much better to have 2to3.py do the renaming and only add warnings to the renamed modules in 2.x (without actually applying any renaming). It would also be possible to seed sys.modules with module proxy objects (see e.g. mx.Misc.LazyModule from egenix-mx-base) which only turn into real module object if the module is referenced. This would allow adding a "from __future__ import new_module_names" which then results in loading proxies for all renamed modules (without actually loading the modules until they are used under their new names). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 18 2008) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Module renaming and pickle mechanisms
M.-A. Lemburg wrote: Perhaps I have a misunderstanding of the reasoning behind doing the renaming in the 2.x branch, but it appears that the only reason is to get used to the new names. That's a rather low priority argument in comparison to the breakage the renaming will cause in the 2.x branch. I think this is the key point here. The possibility of breaking pickling compatibility never came up during the PEP 3108 discussions, so wasn't taken into account in deciding whether or not backporting the name changes was a good idea. I think it's pretty clear that the code needs to be moved back into the modules with the old names for 2.6. The only question is whether or not we put any effort into making the new stdlib organisation usable in 2.x, or just rely on 2to3 to fix it (note that the "increasing the common subset" argument doesn't really apply, since you can catch the import errors in order to try both names). Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optimization of Python ASTs: How should we deal with constant values?
On Fri, May 2, 2008 at 10:21 AM, Thomas Lee <[EMAIL PROTECTED]> wrote: > Any Jython folk care to weigh in on this? If there are no major objections I > think I'm going to forge ahead with an independant Const() node. I suspect that having a marker for non-int non-str constants could also be used for some optimization of Jython bytecode emissions as well, though of course it would depend on the details. If the path from raw AST to optimized AST isn't too crazy Jython can just grow the same logic. We have really good AST comparison tests these days, so I bet it won't be a huge problem for us when the time comes. -Frank ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Documentation for ability to execute zipfiles & directories
On Tue, Mar 4, 2008 at 1:36 PM, A.M. Kuchling <[EMAIL PROTECTED]> wrote: > On Tue, Mar 04, 2008 at 08:58:57AM -0500, Steve Holden wrote: >> While I hesitate to suggest a change of such magnitude, there's >> something to recommend the old IBM mainframe approach of separating out >> "Principles of Operation" (which would be the reference manuals, in >> Python's case the Language and Library refs) from "Users' Guide" which >> contains the practical stuff you need to actually make use of a product. > > Good suggestion. Using the debugger and profiler could also be > covered in the User's Guide. > > Would splitting up the docs make them more useful for > IronPython/Jython? For example, Jython could eventually take the 2.6 > language docs as-is, but modify the library reference to remove > unsupported modules and add Jython-specific ones. Speaking for Jython, this would be extremely helpful for us. Once we get caught up, better docs will become one of our most important priorities I think. -Frank ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Visual Studio 2008 compiler option EHsc ?
> Is there any downside to doing this by default? The most obvious one is the (slight) loss of performance. The mere fact that it is an option, and not enabled by default, indicates that there might be problems enabling it. One problem might be that some applications may require support for asynchronous (structured) exceptions and want to specify /EHa, or that they have cases where C functions throw C++ exceptions, so they want to specify /EHs (no c). So yes, unfortunately, you'll have to add the option into your own setup.py, it seems. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Module renaming and pickle mechanisms
On Sun, May 18, 2008 at 6:14 AM, Nick Coghlan <[EMAIL PROTECTED]> wrote: > M.-A. Lemburg wrote: >> >> Perhaps I have a misunderstanding of the reasoning behind >> doing the renaming in the 2.x branch, but it appears that >> the only reason is to get used to the new names. That's a >> rather low priority argument in comparison to the breakage >> the renaming will cause in the 2.x branch. > > I think this is the key point here. The possibility of breaking pickling > compatibility never came up during the PEP 3108 discussions, so wasn't taken > into account in deciding whether or not backporting the name changes was a > good idea. > > I think it's pretty clear that the code needs to be moved back into the > modules with the old names for 2.6. The only question is whether or not we > put any effort into making the new stdlib organisation usable in 2.x, or > just rely on 2to3 to fix it (note that the "increasing the common subset" > argument doesn't really apply, since you can catch the import errors in > order to try both names). Problem with this is it makes forward-porting revisions to 3.0 a PITA. By keeping the module names consistent between the versions merging a revision is just a matter of ``svnmerge merge`` with the usual 3.0-specific changes. Reverting the modules back to the old name will make forward-porting much more difficult as I don't think svn keeps rename information around (and thus map the old name to the new name in terms of diffs). Alexandre's idea of teaching pickle the mapping of old names to new might be the best solution. We could have a flag to pickle that deactivates the renaming. Otherwise we could bump the pickle version number so that the new number doesn't do the mapping while the old versions to the implicit module mapping. And as Greg and Glpyh have pointed out, this is a problem that might need to be addressed in the future with some changes to our serialization method (I have no clue how since I don't deal with pickle very much). -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] buffer interface for C extensions
Hi all,
I've written a small C extension to submit commands to SCSI devices via
Linux's sg_io driver (for a camera hacking project). The extension is
just a wrapper around a couple ioctl()'s with Pythonic exception handling
thrown in. One of my extension methods is called like this from python:
sg.write(fd, command[, data, timeout)
Both command and data are binary strings. I would like to be able to use
either a regular Python string or an array('B', ...) for these read-only
arguments. So I tried to use the "t#" argument specifier to indicate that
these arguments could be either strings or objects that implement the read-
only buffer interface:
if (!PyArg_ParseTuple(args, "it#|t#i:write", &sg_fd, &cmd,
&cmdLen, &buf, &bufLen, &timeout))
return NULL;
Now, this works fine with strings, but when I call it with an array I get
a TypeError:
TypeError: write() argument 2 must be string or read-only character
buffer, not array.array
So, I then tried changing "t#" to "w#" to indicate that the arguments must
implement the /read-write/ buffer interface. Now the array objects work,
but when I try a string argument, I naturally get this error:
TypeError: Cannot use string as modifiable buffer
So here's what I don't understand. Why doesn't the "t#" argument
specifier support read-write buffers as well as read-only buffers? Aren't
read-write buffers a *superset* of read-only buffers?? Is there something
I'm doing wrong or a quick fix to get this to work appropriately?
Thanks for any help!
Dan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Conditional For Statements
I'd like to propose an addition to the for statement's syntax:
for {variable} in {iterable} if {condition}:
{block}
which is equivalent to
for {variable} in {iterable}:
if not {condition}:
continue
{block}
and
for {variable} in filter(lambda: {condition}, iterable):
{block}
This would make the syntax closer to that of generators, which have
'for variable in iterable if condition', and would improve code
clarity by increased brevity and not negating boolean expressions.
Following are examples of current code with what the new code would
be, taken from the Python 3.0a5 tarball.
Demo/tkinter/guido/ss1.py:163:
for (x, y), cell in self.cells.items():
if x <= 0 or y <= 0:
continue
for (x, y), cell in self.cells.items() if x > 0 and y > 0:
Lib/encodings/__init__.py:91:
for modname in modnames:
if not modname or '.' in modname:
continue
for modname in modnames if modname and '.' not in modname:
Lib/idlelib/AutoExpand.py:70:
for w in wafter:
if dict.get(w):
continue
for w in wafter if w not in dict:
Lib/Cookie.py:483:
for K,V in items:
if V == "": continue
if K not in attrs: continue
for K,V in items if V != "" and K not in attrs:
Lib/hashlib.py:108:
for opensslFuncName in filter(lambda n: n.startswith('openssl_'),
dir(_hashlib)):
for opensslFuncName in dir(_hashlib) if opensslFuncName.startswith('openssl_'):
There are many more examples of this in the standard library, and
likely even more in production code.
I am not familiar with LL(1) parsing, so this may impossible under
that constraint.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Conditional For Statements
First, the best list on which to discuss this would be Python-Ideas not
Python-Dev.
Second, this was brought up there a few months ago, although not much
discussion ensued:
http://mail.python.org/pipermail/python-ideas/2008-March/001407.html
- Eli
On Sun, May 18, 2008 at 9:45 PM, Ryan Hitchman <[EMAIL PROTECTED]> wrote:
> I'd like to propose an addition to the for statement's syntax:
>
> for {variable} in {iterable} if {condition}:
>{block}
>
> which is equivalent to
>
> for {variable} in {iterable}:
>if not {condition}:
>continue
>{block}
>
> and
>
> for {variable} in filter(lambda: {condition}, iterable):
>{block}
>
> This would make the syntax closer to that of generators, which have
> 'for variable in iterable if condition', and would improve code
> clarity by increased brevity and not negating boolean expressions.
>
> Following are examples of current code with what the new code would
> be, taken from the Python 3.0a5 tarball.
>
> Demo/tkinter/guido/ss1.py:163:
> for (x, y), cell in self.cells.items():
>if x <= 0 or y <= 0:
>continue
> for (x, y), cell in self.cells.items() if x > 0 and y > 0:
>
> Lib/encodings/__init__.py:91:
> for modname in modnames:
>if not modname or '.' in modname:
>continue
> for modname in modnames if modname and '.' not in modname:
>
> Lib/idlelib/AutoExpand.py:70:
> for w in wafter:
>if dict.get(w):
>continue
> for w in wafter if w not in dict:
>
> Lib/Cookie.py:483:
> for K,V in items:
>if V == "": continue
>if K not in attrs: continue
> for K,V in items if V != "" and K not in attrs:
>
> Lib/hashlib.py:108:
> for opensslFuncName in filter(lambda n: n.startswith('openssl_'),
> dir(_hashlib)):
> for opensslFuncName in dir(_hashlib) if
> opensslFuncName.startswith('openssl_'):
>
> There are many more examples of this in the standard library, and
> likely even more in production code.
>
> I am not familiar with LL(1) parsing, so this may impossible under
> that constraint.
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/eli%40courtwright.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Conditional For Statements
On Sun, 18 May 2008, Ryan Hitchman wrote:
I'd like to propose an addition to the for statement's syntax:
for {variable} in {iterable} if {condition}:
{block}
which is equivalent to
for {variable} in {iterable}:
if not {condition}:
continue
{block}
and
for {variable} in filter(lambda: {condition}, iterable):
{block}
This would make the syntax closer to that of generators, which have
'for variable in iterable if condition', and would improve code
clarity by increased brevity and not negating boolean expressions.
http://mail.python.org/pipermail/python-dev/2006-May/065090.html
This was proposed and rejected before, after which it was again proposed
and rejected. I think it was also proposed and rejected sometime in 2007,
but I can't seem to find it in the archives so my memory may be faulty.
The idea is apparently not dead yet. Hang around a couple of minutes,
though. It won't be long.
Isaac Morland CSCF Web Guru
DC 2554C, x36650WWW Software Specialist
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Conditional For Statements
"Ryan Hitchman" <[EMAIL PROTECTED]> wrote in message
news:[EMAIL PROTECTED]
| I'd like to propose an addition to the for statement's syntax:
|
| for {variable} in {iterable} if {condition}:
|{block}
|
| which is equivalent to
|
| for {variable} in {iterable}:
|if not {condition}:
|continue
|{block}
Is this different from
for var in filter(lambda x: condition(x), iterable)):
?
| and
|
| for {variable} in filter(lambda: {condition}, iterable):
|{block}
IDLE 3.0a5
>>> for i in filter(lambda i: i%2, range(10)):
print(i)
1
3
5
7
9
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
