Re: [Python-Dev] bytes / unicode
Guido van Rossum writes:
> On Thu, Jun 24, 2010 at 1:12 AM, Stephen J. Turnbull
> wrote:
> Understood, but both the majority of str/bytes methods and several
> existing APIs (e.g. many in the os module, like os.listdir()) do it
> this way.
Understood.
> Also, IMO a polymorphic function should *not* accept *mixed*
> bytes/text input -- join('x', b'y') should be rejected.
Agreed.
> But join('x', 'y') -> 'x/y' and join(b'x', b'y') -> b'x/y' make
> sense to me.
>
> So, actually, I *don't* understand what you mean by needing LBYL.
Consider docutils. Some folks assert that URIs *are* bytes and should
be manipulated as such. So base URIs should be bytes. But there are
various ways to refer to a base URI and combine it with relative URI
taken from literal text in reST. That literal text will be
represented as str. So you want to use urljoin, but this usage isn't
polymorphic.
If you forget to do a conversion here, urljoin will raise, of course.
But late conversion may not be appropriate. AIUI Philip at least
wants ways to raise exceptions earlier than that on some code paths.
That's LBYL, no?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytes / unicode
P.J. Eby writes:
> This doesn't have to be in the functions; it can be in the
> *types*. Mixed-type string operations have to do type checking and
> upcasting already, but if the protocol were open, you could make an
> encoded-bytes type that would handle the error checking.
Don't you realize that "encoded-bytes" is equivalent to use of a very
limited profile of ISO 2022 coding extensions? Such as Emacs/MULE
internal encoding or TRON code? It has been tried. It does not work.
I understand how types can do such checking; my point is that the
encoded-bytes type doesn't have enough information to do it in the
cases where you think it is better than converting to str. There are
*no useful operations* that can be done on two encoded-bytes with
different encodings unless you know the ultimate target codec. The
only sensible way to define the concatenation of ('ascii', 'English')
with ('euc-jp','ÆüËܸì') is something like ('ascii', 'English',
'euc-jp','ÆüËܸì'), and *not* ('euc-jp','EnglishÆüËܸì'), because you
don't know that the ultimate target codec is 'euc-jp'-compatible.
Worse, you need to build in all the information about which codecs are
mutually compatible into the encoded-bytes type. For example, if the
ultimate target is known to be 'shift_jis', it's trivially compatible
with 'ascii' and 'euc-jp' requires a conversion, but latin-9 you can't
have.
> (Btw, in some earlier emails, Stephen, you implied that this could be
> fixed with codecs -- but it can't, because the problem isn't with the
> bytes containing invalid Unicode, it's with the Unicode containing
> invalid bytes -- i.e., characters that can't be encoded to the
> ultimate codec target.)
No, the problem is not with the Unicode, it is with the code that
allows characters not encodable with the target codec. If you don't
have a target codec, there are ascii-safe source codecs, such as
'latin-1' or 'ascii' with surrogateescape, that will work any time
that bytes-oriented processing can work.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/24/2010 8:23 PM, James Y Knight wrote: > On Jun 24, 2010, at 5:53 PM, Scott Dial wrote: >> If the package has .so files that aren't compatible with other version >> of python, then what is the motivation for placing that in a shared >> location (since it can't actually be shared) > > Because python looks for .so files in the same place it looks for the > .py files of the same package. My suggestion was that a package that contains .so files should not be shared (e.g., the entire lxml package should be placed in a version-specific path). The motivation for this PEP was to simplify the installation python packages for distros; it was not to reduce the number of .py files on the disk. Placing .so files together does not simplify that install process in any way. You will still have to handle such packages in a special way. You must still compile the package multiple times for each relevant version of python (with special tagging that I imagine distutils can take care of) and, worse yet, you have created a more trick install than merely having multiple search paths (e.g., installing/uninstalling lxml for *one* version of python is actually more difficult in this scheme). Either the motivation for this PEP is inaccurate or I am failing to understand how this is *simpler*. In the case of pure-python, this PEP is clearly a win, but I have not seen an argument that it is a win for .so files. Moreover, the PEP itself is titled "PYC Repository Directories" (not "shared site-packages") and makes no mention of .so files at all. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/24/2010 9:18 PM, Greg Ewing wrote: > Scott Dial wrote: > >> But the only motivation for doing this with .pyc files is that the .py >> files are able to be shared, > > In an application made up of a mixture of pure Python and > extension modules, the .py files are able to be shared too. > Seems to me that a similar motivation exists here as well. > Not exactly the same, but closely related. > If I recall Barry's motivation correctly, the PEP was intended to simplify the installation of packages for multiple versions of Python, although the PEP states that in a less direct way. In the case of pure-python packages, this is merely about avoiding .pyc collisions. But, in the case of packages with .so files, I fail to see how this is simpler (in face, I believe it to be more complicated). So, I am not sure the PEP supports this feature being proposed (since it makes no mention of .so files), and more importantly, I am not sure it actually makes anything better for anyone (still requires multiple compilations and un/install gymnastics). -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "2 or 3" link on python.org
On Fri, Jun 25, 2010 at 5:34 AM, Nick Coghlan wrote: > On Fri, Jun 25, 2010 at 11:18 AM, Terry Reedy wrote: >> I believe there is material on the wiki as well as the two existing pages on >> other sites that were discussed here. So a new page on python.org could >> consist of a few links. Someone just has to write it. > > There's material on the wiki *now* (the Python2orPython3 page), but > there wasn't before the recent discussion started. The whole > Beginner's Guide on the wiki could actually use some TLC to bring it > up to speed with the existence of Python 3.x. > > Cheers, > Nick. > +1, this definitely sounds like a good idea to me. cheers, Laurens ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thoughts on the bytes/string discussion
Ian Bicking writes:
> We've setup a system where we think of text as natively unicode, with
> encodings to put that unicode into a byte form. This is certainly
> appropriate in a lot of cases. But there's a significant class of problems
> where bytes are the native structure. Network protocols are what we've been
> discussing, and are a notable case of that. That is, b'/' is the most
> native sense of a path separator in a URL, or b':' is the most native sense
> of what separates a header name from a header value in HTTP.
IMHO, URIs don't have a native language in this sense. Network
programmers do, however, and it is bytes. Text-handling programmers
also do, and it is str.
> So with this idea in mind it makes more sense to me that *specific pieces of
> text* can be reasonably treated as both bytes and text. All the string
> literals in urllib.parse.urlunspit() for example.
>
> The semantics I imagine are that special('/')+b'x'==b'/x' (i.e., it does not
> become special('/x')) and special('/')+x=='/x' (again it becomes str). This
> avoids some of the cases of unicode or str infecting a system as they did in
> Python 2 (where you might pass in unicode and everything works fine until
> some non-ASCII is introduced).
I think you need to give explicit examples where this actually helps
in terms of "type contagion". I expect that it doesn't help at all,
especially not for the people whose native language for URIs is bytes.
These specials are still going to flip to unicode as soon as it comes
in, and that will be incompatible with the bytes they'll need later.
So they're still going to need to filter out unicode on input.
It looks like it would be useful for programmers of polymorphic
functions, though.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytes / unicode
At 04:49 PM 6/25/2010 +0900, Stephen J. Turnbull wrote:
P.J. Eby writes:
> This doesn't have to be in the functions; it can be in the
> *types*. Mixed-type string operations have to do type checking and
> upcasting already, but if the protocol were open, you could make an
> encoded-bytes type that would handle the error checking.
Don't you realize that "encoded-bytes" is equivalent to use of a very
limited profile of ISO 2022 coding extensions? Such as Emacs/MULE
internal encoding or TRON code? It has been tried. It does not work.
I understand how types can do such checking; my point is that the
encoded-bytes type doesn't have enough information to do it in the
cases where you think it is better than converting to str. There are
*no useful operations* that can be done on two encoded-bytes with
different encodings unless you know the ultimate target codec.
I do know the ultimate target codec -- that's the point.
IOW, I want to be able to do to all my operations by passing
target-encoded strings to polymorphic functions. Then, the moment
something creeps in that won't go to the target codec, I'll be able
to track down the hole in the legacy code that's letting bad data creep in.
The
only sensible way to define the concatenation of ('ascii', 'English')
with ('euc-jp','ÆüËܸì') is something like ('ascii', 'English',
'euc-jp','ÆüËܸì'), and *not* ('euc-jp','EnglishÆüËܸì'), because you
don't know that the ultimate target codec is 'euc-jp'-compatible.
Worse, you need to build in all the information about which codecs are
mutually compatible into the encoded-bytes type. For example, if the
ultimate target is known to be 'shift_jis', it's trivially compatible
with 'ascii' and 'euc-jp' requires a conversion, but latin-9 you can't
have.
The interaction won't be with other encoded bytes, it'll be with
other *unicode* strings. Ones coming from other code, and literals
embedded in the stdlib.
No, the problem is not with the Unicode, it is with the code that
allows characters not encodable with the target codec.
And which code that is, precisely, is the thing that may be very
difficult to find, unless I can identify it at the first point it
enters (and corrupts) my output data. When dealing with a large code
base, this may be a nontrivial problem.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytes / unicode
On Fri, Jun 25, 2010 at 2:05 AM, Stephen J. Turnbull wrote:
> > But join('x', 'y') -> 'x/y' and join(b'x', b'y') -> b'x/y' make
> > sense to me.
> >
> > So, actually, I *don't* understand what you mean by needing LBYL.
>
> Consider docutils. Some folks assert that URIs *are* bytes and should
> be manipulated as such. So base URIs should be bytes.
I don't get what you are arguing against. Are you worried that if we make
URL code polymorphic that this will mean some code will treat URLs as bytes,
and that code will be incompatible with URLs as text? No one is arguing we
remove text support from any of these functions, only that we allow bytes.
--
Ian Bicking | http://blog.ianbicking.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thoughts on the bytes/string discussion
On Fri, Jun 25, 2010 at 5:06 AM, Stephen J. Turnbull wrote:
> > So with this idea in mind it makes more sense to me that *specific
> pieces of
> > text* can be reasonably treated as both bytes and text. All the string
> > literals in urllib.parse.urlunspit() for example.
> >
> > The semantics I imagine are that special('/')+b'x'==b'/x' (i.e., it does
> not
> > become special('/x')) and special('/')+x=='/x' (again it becomes str).
> This
> > avoids some of the cases of unicode or str infecting a system as they
> did in
> > Python 2 (where you might pass in unicode and everything works fine
> until
> > some non-ASCII is introduced).
>
> I think you need to give explicit examples where this actually helps
> in terms of "type contagion". I expect that it doesn't help at all,
> especially not for the people whose native language for URIs is bytes.
> These specials are still going to flip to unicode as soon as it comes
> in, and that will be incompatible with the bytes they'll need later.
> So they're still going to need to filter out unicode on input.
>
> It looks like it would be useful for programmers of polymorphic
> functions, though.
>
I'm proposing these specials would be used in polymorphic functions, like
the functions in urllib.parse. I would not personally use them in my own
code (unless of course I was writing my own polymorphic functions).
This also makes it less important that the objects be a full stand-in for
text, as their use should be isolated to specific functions, they aren't
objects that should be passed around much. So you can easily identify and
quickly detect if you use unsupported operations on those text-like
objects. (This is all a very different use case from bytes+encoding, I
think)
--
Ian Bicking | http://blog.ianbicking.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2010-06-18 - 2010-06-25) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2795 open (+38) / 18104 closed (+14) / 20899 total (+52) Open issues with patches: 1130 Average duration of open issues: 712 days. Median duration of open issues: 503 days. Open Issues Breakdown open 2765 (+38) languishing13 ( +0) pending16 ( +0) Issues Created Or Reopened (55) ___ os.path.normcase documentation/behaviour unclear on Mac OS X 2010-06-25 http://bugs.python.org/issue3485reopened ezio.melotti patch uuid.uuid4() generates non-unique values on OSX2010-06-21 http://bugs.python.org/issue8621reopened skrah patch test_support.run_unittest cmdline options and arguments2010-06-20 http://bugs.python.org/issue9028reopened techtonik errors='replace' works in IDLE, fails at Windows command line. 2010-06-18 http://bugs.python.org/issue9029created jvanpraag ctypes variable limits 2010-06-18 http://bugs.python.org/issue9030created kumma distutils uses invalid "-Wstrict-prototypes" flag when compili 2010-06-18 http://bugs.python.org/issue9031created matteo.vescovi xmlrpc: Transport.request() should also catch socket.error(EPI 2010-06-18 http://bugs.python.org/issue9032created haypo patch cmd module tab misbehavior 2010-06-19 http://bugs.python.org/issue9033created slcott datetime module should use int32_t for date/time components2010-06-20 http://bugs.python.org/issue9034created belopolsky os.path.ismount on windows doesn't support windows mountpoint 2010-06-20 http://bugs.python.org/issue9035created Oren_Held Simplify Py_CHARMASK 2010-06-20 http://bugs.python.org/issue9036created skrah patch, needs review Add explanation as to how to raise a custom exception in the e 2010-06-20 http://bugs.python.org/issue9037created jonathan.underwood patch test_distutils failure 2010-06-20 http://bugs.python.org/issue9038created pitrou IDLE and module Doc2010-06-20 http://bugs.python.org/issue9039created Yoda_Uchiha using MIMEApplication to attach a PDF raises a TypeError excep 2010-06-21 http://bugs.python.org/issue9040created Enrico.Sartori raised exception is misleading 2010-06-21 http://bugs.python.org/issue9041created kumma Gettext cache and classes 2010-06-21 http://bugs.python.org/issue9042created v_peter patch 2to3 doesn't handle byte comparison well 2010-06-21 CLOSED http://bugs.python.org/issue9043created vdupras [optparse] confusion o
[Python-Dev] Schedule for Python 2.6.6
Benjamin is still planning to release Python 2.7 final on 2010-07-03, so it's time for me to work out the release schedule for Python 2.6.6 - likely the last maintenance release for Python 2.6. Because summer schedules are crazy, and I want to leave two weeks between 2.6.6 rc1 and 2.6.6 final, my current schedule looks like: * Python 2.6.6 rc 1 on Monday 2010-08-02 * Python 2.6.6 final on Monday 2010-08-16 This should give folks plenty of time to relax after 2.7 final, and still be able to get those last minute fixes into the 2.6 tree. Let me know if these dates don't work for you. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytes / unicode
P.J. Eby writes: > I do know the ultimate target codec -- that's the point. > > IOW, I want to be able to do to all my operations by passing > target-encoded strings to polymorphic functions. IOW, you *do* have text and (ignoring efficiency issues) could just as well use str. But That Other Code is unreliable, so you need a marker for your own internal strings indicating that they are validated, while other strings are not. This has nothing to do with bytes vs. str as string types, then; it's all about validated (which your architecture indicates by using the bytes type) vs. unvalidated (which your architecture indicates with unicode). Eg, in the case of your USPS vs. ecommerce example, you can't even handle all bytes, so not all possible bytes objects are valid. And other applications might not be able to handle all Japanese, but only a subset, so having valid EUC-JP wouldn't be enough, you'd have to check repertoire -- might as well use str. It seems to me what is wanted here is something like Perl's taint mechanism, for *both* kinds of strings. Am I missing something? But with your architecture, it seems to me that you actually don't want polymorphic functions in the stdlib. You want the stdlib functions to be bytes-oriented if and only if they are reliable. (This is what I was saying to Guido elsewhere.) BTW, this was a little unclear to me: > [Collisions will] be with other *unicode* strings. Ones coming > from other code, and literals embedded in the stdlib. What about the literals in the stdlib? Are you saying they contain invalid code points for your known output encoding? Or are you saying that with non-polymorphic unicode stdlib, you get lots of false positives when combining with your validated bytes? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Schedule for Python 2.6.6
On Jun 25, 2010, at 12:18 PM, Barry Warsaw wrote: >* Python 2.6.6 rc 1 on Monday 2010-08-02 >* Python 2.6.6 final on Monday 2010-08-16 I've also updated the Google calendar of Python releases: http://www.google.com/calendar/ical/b6v58qvojllt0i6ql654r1vh00%40group.calendar.google.com/public/basic.ics -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thoughts on the bytes/string discussion
Ian Bicking writes: > I'm proposing these specials would be used in polymorphic functions, like > the functions in urllib.parse. I would not personally use them in my own > code (unless of course I was writing my own polymorphic functions). > > This also makes it less important that the objects be a full stand-in for > text, as their use should be isolated to specific functions, they aren't > objects that should be passed around much. So you can easily identify and > quickly detect if you use unsupported operations on those text-like > objects. OK. That sounds reasonable to me, but I don't see any need for a builtin type for it. Inclusion in the stdlib is not quite a no-brainer, but given Guido's endorsement of polymorphism, I can't bring myself to go lower than +0.9 . > (This is all a very different use case from bytes+encoding, I think) Very much so. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytes / unicode
Ian Bicking writes: > I don't get what you are arguing against. Are you worried that if > we make URL code polymorphic that this will mean some code will > treat URLs as bytes, and that code will be incompatible with URLs > as text? No one is arguing we remove text support from any of > these functions, only that we allow bytes. No, I understand what Guido means by "polymorphic". I'm arguing that as I understand one of Philip Eby's use cases, "bytes" is a misspelling of "validated" and "unicode" is a misspelling of "unvalidated". In case of some kind of bug, polymorphic stdlib functions would allow propagation of unvalidated/unicode within the validated zone, aka "errors passing silently". Now that I understand that that use case doesn't actually care about bytes vs. unicode *string* semantics at all, the argument becomes moot, I guess. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thoughts on the bytes/string discussion
On Fri, Jun 25, 2010 at 11:30 AM, Stephen J. Turnbull wrote:
> Ian Bicking writes:
>
> > I'm proposing these specials would be used in polymorphic functions,
> like
> > the functions in urllib.parse. I would not personally use them in my
> own
> > code (unless of course I was writing my own polymorphic functions).
> >
> > This also makes it less important that the objects be a full stand-in
> for
> > text, as their use should be isolated to specific functions, they aren't
> > objects that should be passed around much. So you can easily identify
> and
> > quickly detect if you use unsupported operations on those text-like
> > objects.
>
> OK. That sounds reasonable to me, but I don't see any need for
> a builtin type for it. Inclusion in the stdlib is not quite a
> no-brainer, but given Guido's endorsement of polymorphism, I can't
> bring myself to go lower than +0.9 .
>
Agreed on a builtin; I think it would be fine to put something in the
strings module, and then in these examples code that used '/' would instead
use strings.ascii('/') (not sure so sure of what the name should be though).
--
Ian Bicking | http://blog.ianbicking.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] docs - Copy
On 6/24/2010 8:51 PM, Rich Healey wrote: http://docs.python.org/library/copy.html Discussion of the wording of current docs should go to python-list. Py-dev is for development of future Python. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Creating APIs that work as both decorators and context managers
Hello all,
I've put a recipe up on the Python cookbook for creating APIs that work
as both decorators and context managers and wonder if it would be
considered a useful addition to the functools module.
http://code.activestate.com/recipes/577273-decorator-and-context-manager-from-a-single-api/
I wrote this after writing almost identical code the second time for
"patch" in the mock module. (The patch decorator can be used as a
decorator or as a context manager and I was writing a new variant.) Both
py.test and django have similar code in places, so it is not an uncommon
pattern.
It is only 40 odd lines (ignore the ugly Python 2 & 3 compatibility
hack), so I'm fine with it living on the cookbook - but it is at least
slightly fiddly to write and has the added niceness of providing the
optional exception handling semantics of __exit__ for decorators as well.
Example use (really hope email doesn't swallow the whitespace - my
apologies in advance if it does):
from context import Context
class mycontext(Context):
def __init__(self, *args):
"""Normal initialiser"""
def start(self):
"""
Called on entering the with block or starting the decorated
function.
If used in a with statement whatever this method returns will
be the
context manager.
"""
def finish(self, *exc):
"""
Called on exit. Arguments and return value of this method have
the same meaning as the __exit__ method of a normal context
manager.
"""
@mycontext('some', 'args')
def function():
pass
with mycontext('some', 'args') as something:
pass
I'm not entirely happy with the name of the class or the start and
finish methods, so open to suggestions there. start and finish *could*
be __enter__ and __exit__ - but that would make the class you implement
*look* like a normal context manager and I thought it was better to
distinguish them. Perhaps before and after?
All the best,
Michael Foord
--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog
READ CAREFULLY. By accepting and reading this email you agree, on behalf of your
employer, to release me from all obligations and waivers arising from any and all
NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS
AGREEMENTS") that I have entered into with your employer, its partners, licensors,
agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me from any BOGUS AGREEMENTS
on behalf of your employer.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On Fri, Jun 25, 2010 at 01:53, Scott Dial wrote: > On 6/24/2010 8:23 PM, James Y Knight wrote: >> On Jun 24, 2010, at 5:53 PM, Scott Dial wrote: >>> If the package has .so files that aren't compatible with other version >>> of python, then what is the motivation for placing that in a shared >>> location (since it can't actually be shared) >> >> Because python looks for .so files in the same place it looks for the >> .py files of the same package. > > My suggestion was that a package that contains .so files should not be > shared (e.g., the entire lxml package should be placed in a > version-specific path). The motivation for this PEP was to simplify the > installation python packages for distros; it was not to reduce the > number of .py files on the disk. I assume you are talking about PEP 3147. You're right that the PEP was for pyc files and that's it. No one is talking about rewriting the PEP. The motivation Barry is using is an overarching one of distros wanting to use a single directory install location for all installed Python versions. That led to PEP 3147 and now this work. > > Placing .so files together does not simplify that install process in any > way. You will still have to handle such packages in a special way. You > must still compile the package multiple times for each relevant version > of python (with special tagging that I imagine distutils can take care > of) and, worse yet, you have created a more trick install than merely > having multiple search paths (e.g., installing/uninstalling lxml for > *one* version of python is actually more difficult in this scheme). This is meant to be used by distros in a programmatic fashion, so my response is "so what?" Their package management system is going to maintain the directory, not a person. You and I are not going to be using this for anything. This is purely meant for Linux OS vendors (maybe OS X) to manage their installs through their package software. I honestly do not expect human beings to be mucking around with these installs (and I suspect Barry doesn't either). > > Either the motivation for this PEP is inaccurate or I am failing to > understand how this is *simpler*. In the case of pure-python, this PEP > is clearly a win, but I have not seen an argument that it is a win for > .so files. Moreover, the PEP itself is titled "PYC Repository > Directories" (not "shared site-packages") and makes no mention of .so > files at all. You're conflating what is being discussed with PEP 3147. That PEP is independent of this. PEP 3147 just empowered this work to be relevant. -Brett > > -- > Scott Dial > sc...@scottdial.com > scod...@cs.indiana.edu > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/25/2010 2:58 PM, Brett Cannon wrote: > I assume you are talking about PEP 3147. You're right that the PEP was > for pyc files and that's it. No one is talking about rewriting the > PEP. Yes, I am making reference to PEP 3147. I make reference to that PEP because this change is of the same order of magnitude as the .pyc change, and we asked for a PEP for that, and if this .so stuff is an extension of that thought process, then it should either be reflected by that PEP or a new PEP. > The motivation Barry is using is an overarching one of distros > wanting to use a single directory install location for all installed > Python versions. That led to PEP 3147 and now this work. It's unclear to me that that is the correct motivation, which you are divining. As I understand it, the motivation to be to *simplify installation* for distros, which may or may not be achieved by using a single directory. In the case of pure-python packages, a single directory is an obvious win. In the case of mixed-python packages, I remain to be persuaded there is any improvement achieved. > This is meant to be used by distros in a programmatic fashion, so my > response is "so what?" Their package management system is going to > maintain the directory, not a person. Then why is the status quo unacceptable? I have already explained how this will still require programmatic steps of at least the same difficulty as the status quo requires, so why should we change anything? I am skeptical that this is a simple programmatic problem either: take any random package on PyPI and tell me whether or not it has a .so file that must be compiled. If such a .so file exists, then this package must be special-cased and compiled for each version of Python on the system (or will ever be on the system?). Such a package yields an arbitrary number of .so files due to the number of version of Python on the machine, and I can't imagine how it is simpler to manage all of those files than it is to manage multiple site-packages. > You're conflating what is being discussed with PEP 3147. That PEP is > independent of this. PEP 3147 just empowered this work to be relevant. Without a PEP (be it PEP 3147 or some other), what is the justification for doing this? The burden should be on "you" to explain why this is a good idea and not just a clever idea. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Creating APIs that work as both decorators and context managers
On Fri, Jun 25, 2010 at 7:35 PM, Michael Foord wrote: > Hello all, > > I've put a recipe up on the Python cookbook for creating APIs that work as > both decorators and context managers and wonder if it would be considered a > useful addition to the functools module. > http://code.activestate.com/recipes/577273-decorator-and-context-manager-from-a-single-api/ It's an interesting idea. I wanted almost exactly this a little while ago, while doing some experiments to add an IEEE 754-compliance wrapper to the decimal module (for my own use). It seems quite natural that one might want to wrap both functions and blocks in the same way. [1] In case anyone wants the details, this was for a 'delay-exceptions' operation, that allows you to execute some number of arithmetic operations, keeping track of the floating-point signals that they produce but not raising the corresponding exceptions until the end of the block; obviously this idea applies equally well to functions as to blocks. It's one of the recommended exception handling modes from section 8 of IEEE 754-2008. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On Jun 25, 2010, at 4:53 AM, Scott Dial wrote: On 6/24/2010 8:23 PM, James Y Knight wrote: On Jun 24, 2010, at 5:53 PM, Scott Dial wrote: If the package has .so files that aren't compatible with other version of python, then what is the motivation for placing that in a shared location (since it can't actually be shared) Because python looks for .so files in the same place it looks for the .py files of the same package. My suggestion was that a package that contains .so files should not be shared (e.g., the entire lxml package should be placed in a version-specific path). The motivation for this PEP was to simplify the installation python packages for distros; it was not to reduce the number of .py files on the disk. Placing .so files together does not simplify that install process in any way. You will still have to handle such packages in a special way. This is a good point, but I think still falls short of a solution. For a package like lxml, indeed you are correct. Since debian needs to build it once per version, it could just put the entire package (.py files and .so files) into a different per-python-version directory. However, then you have to also consider python packages made up of multiple distro packages -- like twisted or zope. Twisted includes some C extensions in the core package. But then there are other twisted modules (installed under a "twisted.foo" name) which do not include C extensions. If the base twisted package is installed under a version-specific directory, then all of the submodule packages need to also be installed under the same version-specific directory (and thus built for all versions). In the past, it has proven somewhat tricky to coordinate which directory the modules for package "foo" should be installed in, because you need to know whether *any* of the related packages includes a native ".so" file, not just the current package. The converse situation, where a base package did *not* get installed into a version-specific directory because it includes no native code, but a submodule *does* include a ".so" file, is even trickier. James ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "2 or 3" link on python.org
>>> I am extremely keen for this to happen. Does anyone have ownership of this >>> project? There was some discussion of it up-list but the discussion fizzled. >> >> Can you please explain what "this project" is, in the context of your >> message? GSoC? GHOP? > > Oh, I thought this was quite clear. I was specifically meaning the large > "Python 2 or 3" button on python.org. It would help users who want to know > what version of python to use if they had a clear guide as to what version > to download. Ah, ok. No, nobody has taken ownership of that project, and likely, nobody actually will - unless you volunteer. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] docs - Copy
Am 25.06.2010 18:57, schrieb Terry Reedy: > On 6/24/2010 8:51 PM, Rich Healey wrote: >> http://docs.python.org/library/copy.html > > Discussion of the wording of current docs should go to python-list. > Py-dev is for development of future Python. No no no. Mis-worded documentation is a bug, just like any other bug, and deserves being discussed here. Furthermore, a sufficient condition for mis-wording is if a user read it in full, and still managed to misunderstand (as happened here). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] docs - Copy
> My apologies guys, I see now. > > I will see if I can think of a less ambiguous way to word this and submit a > bug. Please don't take out or rephrase the word "shallow", though. This has a long CS tradition of meaning exactly what is meant here. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Schedule for Python 2.6.6
Am 25.06.2010 18:18, schrieb Barry Warsaw: > Benjamin is still planning to release Python 2.7 final on 2010-07-03, so it's > time for me to work out the release schedule for Python 2.6.6 - likely the > last maintenance release for Python 2.6. > > Because summer schedules are crazy, and I want to leave two weeks between > 2.6.6 rc1 and 2.6.6 final, my current schedule looks like: > > * Python 2.6.6 rc 1 on Monday 2010-08-02 > * Python 2.6.6 final on Monday 2010-08-16 That would barely work for me. If schedule slips in any way, we'll have to move the release into end-of-September (but the days as proposed are fine). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thoughts on the bytes/string discussion
On Jun 24, 2010, at 4:59 PM, Guido van Rossum wrote: > Regarding the proposal of a String ABC, I hope this isn't going to > become a backdoor to reintroduce the Python 2 madness of allowing > equivalency between text and bytes for *some* strings of bytes and not > others. For my part, what I want out of a string ABC is simply the ability to do application-specific optimizations. There are many applications where all input and output is text, but _must_ be UTF-8. Even GTK uses UTF-8 as its native text representation, so "output" could just be display. Right now, in Python 3, the only way to be "correct" about this is to copy every byte of input into 4 bytes of output, then copy each code point *back* into a single byte of output. If all your application does is rewrite the occasional XML attribute, for example, this cost can be significant, if not overwhelming. I'd like a version of 'decode' which would give me a type that was, in every respect, unicode, and responded to all protocols exactly as other unicode objects (or "str objects", if you prefer py3 nomenclature ;-)) do, but wouldn't actually copy any of that memory unless it really needed to (for example, to pass to a C API that expected native wide characters), and that would hold on to the original bytes so that it could produce them on demand if encoded to the same encoding again. So, as others in this thread have mentioned, the 'ABC' really implies some stuff about C APIs as well. I'm not sure about the exact performance impact of such a class, which is why I'd like the ability to implement it *outside* of the stdlib and see how it works on a project, and return with a proposal along with some data. There are also different ways to implement this, and other optimizations (like ropes) which might be better. You can almost do this today, but the lack of things like the hypothetical "__rcontains__" does make it impossible to be totally transparent about it.___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On Jun 24, 2010, at 11:37 PM, Éric Araujo wrote: >Your plan seems good. Adding keyword arguments should not create >compatibility issues, and I suspect the impact on the code of build_ext >may be actually quite small. I’ll try to review your patch even though I >don’t know C or compiler oddities, but Tarek will have the best insight >and the final word. The C and configure/Makefile bits are pretty trivial. It basically extends the list of shared library extensions searched for on *nix machines, and allows that to be set on the ./configure command. As for the impact on distutils, with updated tests, it's less than 100 lines of diff. Again there it essentially allows us to pass the extension that build_ext writes to from the setup.py, via the Extension class. Because distutil's default is to use the $SO variable from the system-installed Makefile, with the change to dynload_shlib.c, configure.in, and Makefile.pre.in, we would get distutils writing the versioned .so files for free. I'll note further that if you *don't* specify this to ./configure, nothing much changes[1]. The distutils part of the patch is only there to disable or override the default, and *that's* only there to support proposed semantics that foo.so be used for PEP 384-compliant ABI extension modules. IOW, until PEP 384 is actually implemented, the distutils part of the patch is unnecessary. However, if the other changes are accepted, then I will add a discussion of this issue to PEP 384, and we can figure out the best semantics and implementation at that point. I honestly don't know if I am going to get to work on PEP 384 before 3.2 beta. >In case the time machine’s not available, your suggestion about getting >the filename from the Extension instance instead of passing in a string >can most certainly land in distutils2. Cool. -Barry [1] Well, I now realize you'll get an extra useless stat call, but I will fix that. signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thoughts on the bytes/string discussion
On Fri, Jun 25, 2010 at 1:43 PM, Glyph Lefkowitz wrote: > > On Jun 24, 2010, at 4:59 PM, Guido van Rossum wrote: > > Regarding the proposal of a String ABC, I hope this isn't going to > become a backdoor to reintroduce the Python 2 madness of allowing > equivalency between text and bytes for *some* strings of bytes and not > others. > > For my part, what I want out of a string ABC is simply the ability to do > application-specific optimizations. > There are many applications where all input and output is text, but _must_ > be UTF-8. Even GTK uses UTF-8 as its native text representation, so > "output" could just be display. > Right now, in Python 3, the only way to be "correct" about this is to copy > every byte of input into 4 bytes of output, then copy each code point *back* > into a single byte of output. If all your application does is rewrite the > occasional XML attribute, for example, this cost can be significant, if not > overwhelming. > I'd like a version of 'decode' which would give me a type that was, in every > respect, unicode, and responded to all protocols exactly as other > unicode objects (or "str objects", if you prefer py3 nomenclature ;-)) do, > but wouldn't actually copy any of that memory unless it really needed to > (for example, to pass to a C API that expected native wide characters), and > that would hold on to the original bytes so that it could produce them on > demand if encoded to the same encoding again. So, as others in this thread > have mentioned, the 'ABC' really implies some stuff about C APIs as well. > I'm not sure about the exact performance impact of such a class, which is > why I'd like the ability to implement it *outside* of the stdlib and see how > it works on a project, and return with a proposal along with some data. > There are also different ways to implement this, and other optimizations > (like ropes) which might be better. > You can almost do this today, but the lack of things like the hypothetical > "__rcontains__" does make it impossible to be totally transparent about it. But you'd still have to validate it, right? You wouldn't want to go on using what you thought was wrapped UTF-8 if it wasn't actually valid UTF-8 (or you'd be worse off than in Python 2). So you're really just worried about space consumption. I'd like to see a lot of hard memory profiling data before I got overly worried about that. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Schedule for Python 2.6.6
On Jun 25, 2010, at 10:33 PM, Martin v. Löwis wrote: >Am 25.06.2010 18:18, schrieb Barry Warsaw: >> Benjamin is still planning to release Python 2.7 final on 2010-07-03, so it's >> time for me to work out the release schedule for Python 2.6.6 - likely the >> last maintenance release for Python 2.6. >> >> Because summer schedules are crazy, and I want to leave two weeks between >> 2.6.6 rc1 and 2.6.6 final, my current schedule looks like: >> >> * Python 2.6.6 rc 1 on Monday 2010-08-02 >> * Python 2.6.6 final on Monday 2010-08-16 > >That would barely work for me. If schedule slips in any way, we'll have >to move the release into end-of-September (but the days as proposed are >fine). Would that be bad or good (slipping into September)? I'd like to get a release out as soon after 2.7 final as possible, but it's an entirely self-imposed deadline. There's no reason why we can't push the whole 2.6.6 thing later if that works better for you. OTOH, I can't go much earlier so if September is bad for you, then we'll stick to the above dates. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "2 or 3" link on python.org
On 25/06/2010 21:27, "Martin v. Löwis" wrote: I am extremely keen for this to happen. Does anyone have ownership of this project? There was some discussion of it up-list but the discussion fizzled. Can you please explain what "this project" is, in the context of your message? GSoC? GHOP? Oh, I thought this was quite clear. I was specifically meaning the large "Python 2 or 3" button on python.org. It would help users who want to know what version of python to use if they had a clear guide as to what version to download. Ah, ok. No, nobody has taken ownership of that project, and likely, nobody actually will - unless you volunteer. What page were we suggesting linking to? IIRC someone made a good start in the wiki. I'll move the discussion to pydotorg-www (still need the question about answering) and see if we can get it done. All the best, Michael Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "2 or 3" link on python.org
> What page were we suggesting linking to? I don't think anybody proposed anything specific. Steve Holden suggested it should go to "reasoned discussion of the pros and cons as evinced in this thread". Stephen Thorne didn't propose anything specific but to have a large button. > I'll move the discussion to pydotorg-www I'll predict that this is its death :-( Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Schedule for Python 2.6.6
> Would that be bad or good (slipping into September)? I'd like to get a > release out as soon after 2.7 final as possible, but it's an entirely > self-imposed deadline. There's no reason why we can't push the whole 2.6.6 > thing later if that works better for you. OTOH, I can't go much earlier so if > September is bad for you, then we'll stick to the above dates. I think we can strive for your original proposal. If it slips, we let it slip by a month or two. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Creating APIs that work as both decorators and context managers
On 25/06/2010 19:35, Michael Foord wrote:
Hello all,
I've put a recipe up on the Python cookbook for creating APIs that
work as both decorators and context managers and wonder if it would be
considered a useful addition to the functools module.
http://code.activestate.com/recipes/577273-decorator-and-context-manager-from-a-single-api/
Actually contextlib would be a much more sensible home for it.
Michael
I wrote this after writing almost identical code the second time for
"patch" in the mock module. (The patch decorator can be used as a
decorator or as a context manager and I was writing a new variant.)
Both py.test and django have similar code in places, so it is not an
uncommon pattern.
It is only 40 odd lines (ignore the ugly Python 2 & 3 compatibility
hack), so I'm fine with it living on the cookbook - but it is at least
slightly fiddly to write and has the added niceness of providing the
optional exception handling semantics of __exit__ for decorators as well.
Example use (really hope email doesn't swallow the whitespace - my
apologies in advance if it does):
from context import Context
class mycontext(Context):
def __init__(self, *args):
"""Normal initialiser"""
def start(self):
"""
Called on entering the with block or starting the decorated
function.
If used in a with statement whatever this method returns will
be the
context manager.
"""
def finish(self, *exc):
"""
Called on exit. Arguments and return value of this method have
the same meaning as the __exit__ method of a normal context
manager.
"""
@mycontext('some', 'args')
def function():
pass
with mycontext('some', 'args') as something:
pass
I'm not entirely happy with the name of the class or the start and
finish methods, so open to suggestions there. start and finish *could*
be __enter__ and __exit__ - but that would make the class you
implement *look* like a normal context manager and I thought it was
better to distinguish them. Perhaps before and after?
All the best,
Michael Foord
--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog
READ CAREFULLY. By accepting and reading this email you agree, on behalf of your
employer, to release me from all obligations and waivers arising from any and all
NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS
AGREEMENTS") that I have entered into with your employer, its partners, licensors,
agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me from any BOGUS AGREEMENTS
on behalf of your employer.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog
READ CAREFULLY. By accepting and reading this email you agree, on behalf of your
employer, to release me from all obligations and waivers arising from any and all
NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS
AGREEMENTS") that I have entered into with your employer, its partners, licensors,
agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me from any BOGUS AGREEMENTS
on behalf of your employer.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thoughts on the bytes/string discussion
On Jun 25, 2010, at 5:02 PM, Guido van Rossum wrote: > But you'd still have to validate it, right? You wouldn't want to go on > using what you thought was wrapped UTF-8 if it wasn't actually valid > UTF-8 (or you'd be worse off than in Python 2). So you're really just > worried about space consumption. So, yes, I am mainly worried about memory consumption, but don't underestimate the pure CPU cost of doing all the copying. It's quite a bit faster to simply scan through a string than to scan and while you're scanning, keep faulting out the L2 cache while you're accessing some other area of memory to store the copy. Plus, If I am decoding with the surrogateescape error handler (or its effective equivalent), then no, I don't need to validate it in advance; interpretation can be done lazily as necessary. I realize that this is just GIGO, but I wouldn't be doing this on data that didn't have an explicitly declared or required encoding in the first place. > I'd like to see a lot of hard memory profiling data before I got overly > worried about that. I know of several Python applications that are already constrained by memory. I don't have a lot of hard memory profiling data, but in an environment where you're spawning as many processes as you can in order to consume _all_ the physically available RAM for string processing, it stands to reason that properly decoding everything and thereby exploding everything out into 4x as much data (or 2x, if you're lucky) would result in a commensurate decrease in throughput. I don't think I could even reasonably _propose_ that such a project stop treating textual data as bytes, because there's no optimization strategy once that sort of architecture has been put into place. If your function says "this takes unicode", then you just have to bite the bullet and decode it, or rewrite it again to have a different requirement. So, right now, I don't know where I'd get the data with to make the argument in the first place :). If there were some abstraction in the core's treatment of strings, though, and I could decode things and note their encoding without immediately paying this cost (or alternately, paying the cost to see if it's so bad, but with the option of managing it or optimizing it separately). This is why I'm asking for a way for me to implement my own string type, and not for a change of behavior or an optimization in the stdlib itself: I could be wrong, I don't have a particularly high level of certainty in my performance estimates, but I think that my concerns are realistic enough that I don't want to embark on a big re-architecture of text-handling only to have it become a performance nightmare that needs to be reverted. As Robert Collins pointed out, they already have performance issues related to encoding in Bazaar. I know they've done a lot of profiling in that area, so I hope eventually someone from that project will show up with some data to demonstrate it :). And I've definitely heard many, many anecdotes (some of them in this thread) about people distorting their data structures in various ways to avoid paying decoding cost in the ASCII/latin1 case, whether it's *actually* a significant performance issue or not. I would very much like to tell those people "Just call .decode(), and if it turns out to actually be a performance issue, you can always deal with it later, with a custom string type." I'm confident that in *most* cases, it would not be. Anyway, this may be a serious issue, but I increasingly feel like I'm veering into python-ideas territory, so perhaps I'll just have to burn this bridge when I come to it. Hopefully after the moratorium. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Schedule for Python 2.6.6
On Jun 25, 2010, at 11:16 PM, Martin v. Löwis wrote: >> Would that be bad or good (slipping into September)? I'd like to get a >> release out as soon after 2.7 final as possible, but it's an entirely >> self-imposed deadline. There's no reason why we can't push the whole 2.6.6 >> thing later if that works better for you. OTOH, I can't go much earlier so >> if >> September is bad for you, then we'll stick to the above dates. > >I think we can strive for your original proposal. If it slips, we let it >slip by a month or two. Cool, thanks Martin. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "2 or 3" link on python.org
On 25/06/2010 22:14, "Martin v. Löwis" wrote: What page were we suggesting linking to? I don't think anybody proposed anything specific. Steve Holden suggested it should go to "reasoned discussion of the pros and cons as evinced in this thread". Stephen Thorne didn't propose anything specific but to have a large button. Earlier in this discussion *someone* did start a page on the wiki, with this use case in mind... You forced me to actually look it up: http://wiki.python.org/moin/Python2orPython3 I'll move the discussion to pydotorg-www I'll predict that this is its death :-( Heh. Michael Regards, Martin -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thoughts on the bytes/string discussion
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Guido van Rossum wrote: > But you'd still have to validate it, right? You wouldn't want to go on > using what you thought was wrapped UTF-8 if it wasn't actually valid > UTF-8 (or you'd be worse off than in Python 2). So you're really just > worried about space consumption. I'd like to see a lot of hard memory > profiling data before I got overly worried about that. I do know for a fact that using a UCS2-compiled Python instead of the system's UCS4-compiled Python leads to measurable, noticable drop in memory consumption of long-running webserver processes using Unicode (Zope, repoze.bfg, etc). We routinely build Python from source for deployments precisely because of this fact (in part -- the absurd choices made by packagers to exclude crucial bits on various pltaforms is the other part). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkwlKbQACgkQ+gerLs4ltQ4TfACdHgLXPHeGw42GidhQdzABkQaR +nEAoLE1sd+g1aJuxSn6swvvX0g52EU4 =MSwx -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thoughts on the bytes/string discussion
On Fri, Jun 25, 2010 at 4:02 PM, Guido van Rossum wrote: > On Fri, Jun 25, 2010 at 1:43 PM, Glyph Lefkowitz > > I'd like a version of 'decode' which would give me a type that was, in > every > > respect, unicode, and responded to all protocols exactly as other > > unicode objects (or "str objects", if you prefer py3 nomenclature ;-)) > do, > > but wouldn't actually copy any of that memory unless it really needed to > > (for example, to pass to a C API that expected native wide characters), > and > > that would hold on to the original bytes so that it could produce them on > > demand if encoded to the same encoding again. So, as others in this > thread > > have mentioned, the 'ABC' really implies some stuff about C APIs as well. > > I'm not sure about the exact performance impact of such a class, which is > > why I'd like the ability to implement it *outside* of the stdlib and see > how > > it works on a project, and return with a proposal along with some data. > > There are also different ways to implement this, and other optimizations > > (like ropes) which might be better. > > You can almost do this today, but the lack of things like the > hypothetical > > "__rcontains__" does make it impossible to be totally transparent about > it. > > But you'd still have to validate it, right? You wouldn't want to go on > using what you thought was wrapped UTF-8 if it wasn't actually valid > UTF-8 (or you'd be worse off than in Python 2). So you're really just > worried about space consumption. I'd like to see a lot of hard memory > profiling data before I got overly worried about that. > It wasn't my profiling, but I seem to recall that Fredrik Lundh specifically benchmarked ElementTree with all-unicode and sometimes-ascii-bytes, and found that using Python 2 strs in some cases provided notable advantages. I know Stefan copied ElementTree in this regard in lxml, maybe he also did a benchmark or knows of one? -- Ian Bicking | http://blog.ianbicking.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytes / unicode
At 01:18 AM 6/26/2010 +0900, Stephen J. Turnbull wrote: It seems to me what is wanted here is something like Perl's taint mechanism, for *both* kinds of strings. Am I missing something? You could certainly view it as a kind of tainting. The part where the type would be bytes-based is indeed somewhat incidental to the actual use case -- it's just that if you already have the bytes, and all you want to do is tag them (e.g. the WSGI headers case), the extra encoding step seems pointless. A string coercion protocol (that would be used by .join(), .format(), __contains__, __mod__, etc.) would allow you to do whatever sort of tainted-string or tainted-bytes implementations one might wish to have. I suppose that tainting user inputs (as in Perl) would be just as useful of an application of the same coercion protocol. Actually, I have another use case for this custom string coercion, which is that I once wrote a string subclass whose purpose was to track the original file and line number of some text. Even though only my code was manipulating the strings, it was very difficult to get the tainting to work correctly without extreme care as to the string methods used. (For example, I had to use string addition rather than %-formatting.) But with your architecture, it seems to me that you actually don't want polymorphic functions in the stdlib. You want the stdlib functions to be bytes-oriented if and only if they are reliable. (This is what I was saying to Guido elsewhere.) I'm not sure I follow you. What I want is for the stdlib to create stringlike objects of a type determined by the types of the inputs -- where the logic for deciding this coercion can be controlled by the input objects' types, rather than putting this in the hands of the stdlib function. And of course, this applies to non-stdlib functions, too -- anything that simply manipulates user-defined string classes, should allow the user-defined classes to determine the coercion of the result. BTW, this was a little unclear to me: > [Collisions will] be with other *unicode* strings. Ones coming > from other code, and literals embedded in the stdlib. What about the literals in the stdlib? Are you saying they contain invalid code points for your known output encoding? Or are you saying that with non-polymorphic unicode stdlib, you get lots of false positives when combining with your validated bytes? No, I mean that the current string coercion rules cause everything to be converted to unicode, thereby discarding the tainting information, so to speak. This applies equally to other tainting use cases, and other uses for custom stringlike objects. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Signs of neglect?
I was pretty stunned when I tried this. Remember that the Tools subdirectory is distributed with Windows, so this means we got through almost two releases without anyone realizing that 2to3 does not appear to have touched this code. Yes, I have: http://bugs.python.org/issue9083 When's 3.2 due out? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS:http://holdenweb.eventbrite.com/ "All I want for my birthday is another birthday" - Ian Dury, 1942-2000 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] thoughts on the bytes/string discussion
Guido van Rossum wrote: > On Fri, Jun 25, 2010 at 1:43 PM, Glyph Lefkowitz > wrote: > > > > On Jun 24, 2010, at 4:59 PM, Guido van Rossum wrote: > > > > Regarding the proposal of a String ABC, I hope this isn't going to > > become a backdoor to reintroduce the Python 2 madness of allowing > > equivalency between text and bytes for *some* strings of bytes and not > > others. I never actually replied to this... Absolutely right, which is why you might really want another kind of string, rather than a way to treat some bytes values as strings in some places. Both Python 2 and Python 3 are missing one of the three types. Python 1 and 2 didn't have "bytes", and this caused problems because "str" was pressed into use to hold arbitrary byte sequences. (Python 2 "str" has other problems as well, like losing track of the encoding.) Python 3 doesn't have Python 2's "str" (encoded string), and bytes are being pressed into use for that. Each of these uses is an ad hoc hijack of an inappropriate type, and additional frameworks not directly supported by the Python language are being jury-rigged to try to support the uses. On the other hand, this is all in the eye of the beholder. Both byte sequences and strings are horrible formless things; they remind me of BLISS. You seldom really have a byte sequence; what you have is an XDR float or an encoded string or an IP header or an email message. Similarly for strings; they are really file names or city names or English sentences or URIs or other things with significant semantic constraints not captured by the typical type system. So, yes, there *is* an inescapable equivalency between text and bytes for *some* sequences of bytes (those that represent encoded strings) and not others (those that contain the XDR float, for instance). Creating a separate encoded string type would be one way to keep that straight. > > For my part, what I want out of a string ABC is simply the ability to do > > application-specific optimizations. > > There are many applications where all input and output is text, but _must_ > > be UTF-8. Even GTK uses UTF-8 as its native text representation, so > > "output" could just be display. > > Right now, in Python 3, the only way to be "correct" about this is to copy > > every byte of input into 4 bytes of output, then copy each code point *back* > > into a single byte of output. If all your application does is rewrite the > > occasional XML attribute, for example, this cost can be significant, if not > > overwhelming. > > I'd like a version of 'decode' which would give me a type that was, in every > > respect, unicode, and responded to all protocols exactly as other > > unicode objects (or "str objects", if you prefer py3 nomenclature ;-)) do, > > but wouldn't actually copy any of that memory unless it really needed to > > (for example, to pass to a C API that expected native wide characters), and > > that would hold on to the original bytes so that it could produce them on > > demand if encoded to the same encoding again. So, as others in this thread > > have mentioned, the 'ABC' really implies some stuff about C APIs as well. Seems like it. > > I'm not sure about the exact performance impact of such a class, which is > > why I'd like the ability to implement it *outside* of the stdlib and see how > > it works on a project, and return with a proposal along with some data. Yes, exactly. > > There are also different ways to implement this, and other optimizations > > (like ropes) which might be better. > > You can almost do this today, but the lack of things like the hypothetical > > "__rcontains__" does make it impossible to be totally transparent about it. > > But you'd still have to validate it, right? You wouldn't want to go on > using what you thought was wrapped UTF-8 if it wasn't actually valid > UTF-8 (or you'd be worse off than in Python 2). Yes, but there are different ways to validate it that have different performance impacts. Simply trusting the source of the string, for example, would be appropriate in some cases. > So you're really just worried about space consumption. I'd like to see > a lot of hard memory profiling data before I got overly worried about > that. While I've seen some big Web pages, I think the email folks, who often have to process messages with attachments measuring in the tens of megabytes, have the stronger problems here, and I think speed may be more important than memory. I've built both a Web server and an IMAP server in Python, and the IMAP server is where the issues of storage management really prevail. If you have to convert a 20 MB encoded string into a Unicode string just to look at the headers as strings, you have issues. (The Python email package doesn't do that, by the way.) Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-d
Re: [Python-Dev] thoughts on the bytes/string discussion
Glyph Lefkowitz wrote: > > On Jun 25, 2010, at 5:02 PM, Guido van Rossum wrote: > >> But you'd still have to validate it, right? You wouldn't want to go on >> using what you thought was wrapped UTF-8 if it wasn't actually valid >> UTF-8 (or you'd be worse off than in Python 2). So you're really just >> worried about space consumption. > > So, yes, I am mainly worried about memory consumption, but don't > underestimate the pure CPU cost of doing all the copying. It's quite a > bit faster to simply scan through a string than to scan and while you're > scanning, keep faulting out the L2 cache while you're accessing some > other area of memory to store the copy. > Yes, but you are already talking about optimizations that might be significant for large-ish strings (where large-ish depends on exactly where Moore's Law is currently delivering computational performance) - the amount of cache consumed by a ten-byte string will slip by unnoticed, but at L2 levels megabytes would effectively flush the cache. > Plus, If I am decoding with the surrogateescape error handler (or its > effective equivalent), then no, I don't need to validate it in advance; > interpretation can be done lazily as necessary. I realize that this is > just GIGO, but I wouldn't be doing this on data that didn't have an > explicitly declared or required encoding in the first place. > >> I'd like to see a lot of hard memory profiling data before I got >> overly worried about that. > > I know of several Python applications that are already constrained by > memory. I don't have a lot of hard memory profiling data, but in an > environment where you're spawning as many processes as you can in order > to consume _all_ the physically available RAM for string processing, it > stands to reason that properly decoding everything and thereby exploding > everything out into 4x as much data (or 2x, if you're lucky) would > result in a commensurate decrease in throughput. > Yes, UCS-4's impact does seem like to could be horrible for these use cases. But "knowing of several Python applications that are already constrained by memory" doesn't mean that it's a bad general decision. Most users will never notice the difference, so we should try to accommodate those who do notice a difference without inconveniencing the rest too much. > I don't think I could even reasonably _propose_ that such a project stop > treating textual data as bytes, because there's no optimization strategy > once that sort of architecture has been put into place. If your function > says "this takes unicode", then you just have to bite the bullet and > decode it, or rewrite it again to have a different requirement. > That has always been my understanding. I regard it as a sort of intellectual tax on the United States (and its Western collaborators) for being too dim to realise that eventually they would end up selling computers to people with more than 256 characters in their alphabet). Sorry guys, but your computers are only as fast as you think they are when you only talk to each other. > So, right now, I don't know where I'd get the data with to make the > argument in the first place :). If there were some abstraction in the > core's treatment of strings, though, and I could decode things and note > their encoding without immediately paying this cost (or alternately, > paying the cost to see if it's so bad, but with the option of managing > it or optimizing it separately). This is why I'm asking for a way for > me to implement my own string type, and not for a change of behavior or > an optimization in the stdlib itself: I could be wrong, I don't have a > particularly high level of certainty in my performance estimates, but I > think that my concerns are realistic enough that I don't want to embark > on a big re-architecture of text-handling only to have it become a > performance nightmare that needs to be reverted. > Recent experience with the thoroughness of the Python 3 release preparations leads me to believe that *anything* new needs to prove its worth outside the stdlib for a while. > As Robert Collins pointed out, they already have performance issues > related to encoding in Bazaar. I know they've done a lot of profiling > in that area, so I hope eventually someone from that project will show > up with some data to demonstrate it :). And I've definitely heard many, > many anecdotes (some of them in this thread) about people distorting > their data structures in various ways to avoid paying decoding cost in > the ASCII/latin1 case, whether it's *actually* a significant performance > issue or not. I would very much like to tell those people "Just call > .decode(), and if it turns out to actually be a performance issue, you > can always deal with it later, with a custom string type." I'm > confident that in *most* cases, it would not be. > Well that would be a nice win. > Anyway, this may be a serious issue, but I increasingly feel like I'm > veering into python
Re: [Python-Dev] docs - Copy
Martin v. Löwis wrote: > Am 25.06.2010 18:57, schrieb Terry Reedy: >> On 6/24/2010 8:51 PM, Rich Healey wrote: >>> http://docs.python.org/library/copy.html >> Discussion of the wording of current docs should go to python-list. >> Py-dev is for development of future Python. > > No no no. [...] It isn't always easy to tell, but I think Martin meant "no". regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS:http://holdenweb.eventbrite.com/ "All I want for my birthday is another birthday" - Ian Dury, 1942-2000 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Schedule for Python 2.6.6
Martin v. Löwis wrote: > Am 25.06.2010 18:18, schrieb Barry Warsaw: >> Benjamin is still planning to release Python 2.7 final on 2010-07-03, so it's >> time for me to work out the release schedule for Python 2.6.6 - likely the >> last maintenance release for Python 2.6. >> >> Because summer schedules are crazy, and I want to leave two weeks between >> 2.6.6 rc1 and 2.6.6 final, my current schedule looks like: >> >> * Python 2.6.6 rc 1 on Monday 2010-08-02 >> * Python 2.6.6 final on Monday 2010-08-16 > > That would barely work for me. If schedule slips in any way, we'll have > to move the release into end-of-September (but the days as proposed are > fine). > > Regards, > Martin A six-week slippage wouldn't be good. What's the relevant chaos theory when a one- or two-day hold leads to a six-week delivery slippage? Let's hope things don't slip! regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS:http://holdenweb.eventbrite.com/ "All I want for my birthday is another birthday" - Ian Dury, 1942-2000 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "2 or 3" link on python.org
Martin v. Löwis wrote: I am extremely keen for this to happen. Does anyone have ownership of this project? There was some discussion of it up-list but the discussion fizzled. >>> Can you please explain what "this project" is, in the context of your >>> message? GSoC? GHOP? >> Oh, I thought this was quite clear. I was specifically meaning the large >> "Python 2 or 3" button on python.org. It would help users who want to know >> what version of python to use if they had a clear guide as to what version >> to download. > > Ah, ok. No, nobody has taken ownership of that project, and likely, > nobody actually will - unless you volunteer. > Or perhaps spur the pydotorg community on with some well-placed encouragement. Nobody ever seems to say "thanks" to those guys except the jobs posters - *they* seem pretty happy. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS:http://holdenweb.eventbrite.com/ "All I want for my birthday is another birthday" - Ian Dury, 1942-2000 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Schedule for Python 2.6.6
Martin v. Löwis wrote: >> Would that be bad or good (slipping into September)? I'd like to get a >> release out as soon after 2.7 final as possible, but it's an entirely >> self-imposed deadline. There's no reason why we can't push the whole 2.6.6 >> thing later if that works better for you. OTOH, I can't go much earlier so >> if >> September is bad for you, then we'll stick to the above dates. > > I think we can strive for your original proposal. If it slips, we let it > slip by a month or two. > > Regards, > Martin I suppose for 2..6. it's not really critical. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ UPCOMING EVENTS:http://holdenweb.eventbrite.com/ "All I want for my birthday is another birthday" - Ian Dury, 1942-2000 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signs of neglect?
2010/6/25 Steve Holden : > I was pretty stunned when I tried this. Remember that the Tools > subdirectory is distributed with Windows, so this means we got through > almost two releases without anyone realizing that 2to3 does not appear > to have touched this code. I would call it more a sign of no tests rather than one of neglect and perhaps also an indication of the usefulness of those tools. > > Yes, I have: http://bugs.python.org/issue9083 > > When's 3.2 due out? PEP 392. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyPy 1.3 released
=== PyPy 1.3: Stabilization === Hello. We're please to announce release of PyPy 1.3. This release has two major improvements. First of all, we stabilized the JIT compiler since 1.2 release, answered user issues, fixed bugs, and generally improved speed. We're also pleased to announce alpha support for loading CPython extension modules written in C. While the main purpose of this release is increased stability, this feature is in alpha stage and it is not yet suited for production environments. Highlights of this release == * We introduced support for CPython extension modules written in C. As of now, this support is in alpha, and it's very unlikely unaltered C extensions will work out of the box, due to missing functions or refcounting details. The support is disable by default, so you have to do:: import cpyext before trying to import any .so file. Also, libraries are source-compatible and not binary-compatible. That means you need to recompile binaries, using for example:: python setup.py build Details may vary, depending on your build system. Make sure you include the above line at the beginning of setup.py or put it in your PYTHONSTARTUP. This is alpha feature. It'll likely segfault. You have been warned! * JIT bugfixes. A lot of bugs reported for the JIT have been fixed, and its stability greatly improved since 1.2 release. * Various small improvements have been added to the JIT code, as well as a great speedup of compiling time. Cheers, Maciej Fijalkowski, Armin Rigo, Alex Gaynor, Amaury Forgeot d'Arc and the PyPy team ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On Sat, Jun 26, 2010 at 6:12 AM, James Y Knight wrote: > However, then you have to also consider python packages made up of multiple > distro packages -- like twisted or zope. Twisted includes some C extensions > in the core package. But then there are other twisted modules (installed > under a "twisted.foo" name) which do not include C extensions. If the base > twisted package is installed under a version-specific directory, then all of > the submodule packages need to also be installed under the same > version-specific directory (and thus built for all versions). > > In the past, it has proven somewhat tricky to coordinate which directory the > modules for package "foo" should be installed in, because you need to know > whether *any* of the related packages includes a native ".so" file, not just > the current package. > > The converse situation, where a base package did *not* get installed into a > version-specific directory because it includes no native code, but a > submodule *does* include a ".so" file, is even trickier. I think there are two major ways to tackle this: - allow multiple versions of a .so file within a single directory (i.e Barry's current suggestion) - enhanced namespace packages, allowing a single package to be spread across multiple directories, some of which may be Python version specific (i.e. modifications to PEP 382 to support references to version-specific directories) I think a new PEP is definitely in order, especially to explain why enhancing PEP 382 to support saying "look over here for the .so files for this version" isn't a preferable approach. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "2 or 3" link on python.org
On 2010-06-25, "Martin v. Löwis" wrote: > > What page were we suggesting linking to? > > I don't think anybody proposed anything specific. Steve Holden > suggested it should go to "reasoned discussion of the > pros and cons as evinced in this thread". Stephen Thorne didn't > propose anything specific but to have a large button. I didn't propose anything, I heard a good idea that I'd like to see followed through. -- Regards, Stephen Thorne ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "2 or 3" link on python.org
Am 26.06.2010 02:41, schrieb Stephen Thorne: > On 2010-06-25, "Martin v. Löwis" wrote: >>> What page were we suggesting linking to? >> >> I don't think anybody proposed anything specific. Steve Holden >> suggested it should go to "reasoned discussion of the >> pros and cons as evinced in this thread". Stephen Thorne didn't >> propose anything specific but to have a large button. > > I didn't propose anything, I heard a good idea that I'd like to see followed > through. Ah, ok. I thought "I am extremely keen for this to happen" indicated that you would be willing to volunteer time to make it happen. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signs of neglect?
On Sat, Jun 26, 2010 at 9:23 AM, Benjamin Peterson wrote: > 2010/6/25 Steve Holden : > I would call it more a sign of no tests rather than one of neglect and > perhaps also an indication of the usefulness of those tools. Less than useful tools with no tests probably qualify as neglected... An assessment of the contents of the Py3k tools directory is probably in order, with at least a basic "will it run?" check added for those we decide to keep.. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytes / unicode
P.J. Eby writes: > it's just that if you already have the bytes, and all you want to > do is tag them (e.g. the WSGI headers case), the extra encoding > step seems pointless. Well, I'll have to concede that unless and until I get involved in the WSGI development effort. > >But with your architecture, it seems to me that you actually don't > >want polymorphic functions in the stdlib. You want the stdlib > >functions to be bytes-oriented if and only if they are reliable. (This > >is what I was saying to Guido elsewhere.) > > I'm not sure I follow you. What I'm saying here is that if bytes are the signal of validity, and the stdlib functions preserve validity, then it's better to have the stdlib functions object to unicode data as an argument. Compare the alternative: it returns a unicode object which might get passed around for a while before one of your functions receives it and identifies it as unvalidated data. But you agree that there are better mechanisms for validation (although not available in Python yet), so I don't see this as an potential obstacle to polymorphism now. > What I want is for the stdlib to create stringlike objects of a > type determined by the types of the inputs -- In general this is a hard problem, though. Polymorphism, OK, one-way tainting OK, but in general combining related types is pretty arbitrary, and as in the encoded-bytes case, the result type often varies depending on expectations of callers, not the types of the data. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
