My apologies for hammering on this, but I think it is quite important and
currently Python 3.0 seems confused about UCS-2 versus UTF-16.
-On [20080702 20:47], Guido van Rossum ([EMAIL PROTECTED]) wrote:
>No, Python already is aware of surrogates. I meant applications
>processing non-BMP text shoul
Hi,
> Subsequently doing a: print a[1] to get the 0x942a (鐪) actually requires
> a[2] on the 2-byte Python 3.0.
How is it annoying *in practice*? In actual code the index, instead of
being a constant, will be retrieved through various means such as .find()
or re.search().start()... as you show y
Jeroen Ruigrok van der Werven wrote:
The documentation for len() says:
Return the length (the number of items) of an object.
So what this tells us is that in a UCS-2 build of Python, the "items" in
a unicode string are not, strictly speaking, Unicode code points or
characters. Instead, they a
I think the discussion is going in the wrong direction:
The choice between UCS2 and UCS4 builds is really only meant
to enhance the possibility to interface to native OS or
application APIs, e.g. Windows LIBC and Java use UTF-16, glibc
on Unix uses UCS4.
The problem of slicing Unicode objects is
On Thu, Jul 3, 2008 at 5:39 AM, Nick Coghlan <[EMAIL PROTECTED]> wrote:
> 1. If you are advocating disallowing the use of characters outside the BMP
> in a UCS-2 build, enumerate the advantages of doing so (paying particular
> attention to any advantages which cannot be obtained simply by using an
-On [20080703 15:00], M.-A. Lemburg ([EMAIL PROTECTED]) wrote:
>Unicode if full of combining code points - if you break such a sequence,
>the output will be just as wrong; regardless of UCS2 vs. UCS4.
In my opinion you are confusing two related, but very separated things here.
Com
> For programmers who want to target a 2-byte format (for win32
> compatibility, for example)
As MAL said, this is taking the discussion in the wrong direction.
For people on Windows, win32 isn't a "compatibility" consideration. I
suspect most users of the other platforms MAL mentioned and all o
On 2008-07-03 15:21, Jeroen Ruigrok van der Werven wrote:
-On [20080703 15:00], M.-A. Lemburg ([EMAIL PROTECTED]) wrote:
Unicode if full of combining code points - if you break such a sequence,
the output will be just as wrong; regardless of UCS2 vs. UCS4.
In my opinion you are confusing two
On Thu, Jul 3, 2008 at 3:48 AM, Jeroen Ruigrok van der Werven
<[EMAIL PROTECTED]> wrote:
> My apologies for hammering on this, but I think it is quite important and
> currently Python 3.0 seems confused about UCS-2 versus UTF-16.
[...]
Your seem to be suggesting that len(u"\U00012345") should retu
On Thu, Jul 3, 2008 at 6:42 AM, Mark Hammond <[EMAIL PROTECTED]> wrote:
> For people on Windows, win32 isn't a "compatibility" consideration. I
> suspect most users of the other platforms MAL mentioned and all others with
> their own native unicode implementations would agree.
I'm sorry, but you'
-On [20080703 15:58], Guido van Rossum ([EMAIL PROTECTED]) wrote:
>Your seem to be suggesting that len(u"\U00012345") should return 1 on
>a system that internally uses UTF-16 and hence represents this string
>as a surrogate pair.
From a Unicode and UTF-16 point of view that m
On Thu, Jul 3, 2008 at 7:46 AM, Jeroen Ruigrok van der Werven
<[EMAIL PROTECTED]> wrote:
> -On [20080703 15:58], Guido van Rossum ([EMAIL PROTECTED]) wrote:
>>Your seem to be suggesting that len(u"\U00012345") should return 1 on
>>a system that internally uses U
Hello,
2008/7/3 Guido van Rossum <[EMAIL PROTECTED]>:
> I don't see an answer there to the question of whether the length()
> method of a Java String object containing a single surrogate pair
> returns 1 or 2; I suspect it returns 2. Python 3 supports things like
> chr(0x12345) and ord("\U00012345
On 03/07/2008, Guido van Rossum <[EMAIL PROTECTED]> wrote:
> I don't see an answer there to the question of whether the length()
> method of a Java String object containing a single surrogate pair
> returns 1 or 2; I suspect it returns 2.
It appears you're right:
>type testucs.java
class testucs
Guido van Rossum python.org> writes:
> The one thing that may be missing from Python is things like
> interpretation of surrogates by functions like isalpha() and I'm okay
> with adding that (since those have to loop over the entire string
> anyway).
That and methods to safely iterate and slice s
Paul Moore wrote:
On 03/07/2008, Guido van Rossum <[EMAIL PROTECTED]> wrote:
I don't see an answer there to the question of whether the length()
method of a Java String object containing a single surrogate pair
returns 1 or 2; I suspect it returns 2.
It appears you're right:
type testucs.jav
-On [20080703 17:32], Paul Moore ([EMAIL PROTECTED]) wrote:
>System.out.println(s.length());
I think you want to use codePointCount() to count the Unicode code points.
length() returns Unicode code units.
As http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Character.html explains:
On Thu, Jul 3, 2008 at 9:35 AM, Steve Holden <[EMAIL PROTECTED]> wrote:
> Paul Moore wrote:
>>
>> On 03/07/2008, Guido van Rossum <[EMAIL PROTECTED]> wrote:
>>>
>>> I don't see an answer there to the question of whether the length()
>>> method of a Java String object containing a single surrogate p
-On [20080703 17:03], Guido van Rossum ([EMAIL PROTECTED]) wrote:
>I don't see an answer there to the question of whether the length()
>method of a Java String object containing a single surrogate pair
>returns 1 or 2; I suspect it returns 2.
As
http://java.sun.com/j2se/1.5.0/docs
-On [20080703 18:45], James Y Knight ([EMAIL PROTECTED]) wrote:
>I think this is misguided.
Only trying to at least correct the current situation, which I consider a
bit of a mess, personally. (Although it seems others share my view.)
>I'd like to have 3 levels of access available
On Jul 3, 2008, at 10:46 AM, Jeroen Ruigrok van der Werven wrote:
-On [20080703 15:58], Guido van Rossum ([EMAIL PROTECTED]) wrote:
Your seem to be suggesting that len(u"\U00012345") should return 1 on
a system that internally uses UTF-16 and hence represents this string
as a surr
On Thu, Jul 3, 2008 at 10:01 AM, Jeroen Ruigrok van der Werven
<[EMAIL PROTECTED]> wrote:
> What would the chances for inclusion in Python be if such a PEP + code would
> be presented Guido?
As long as it is clear that the len() function and the basic slicing
and indexing operations on strings con
(sorry for the crossposting)
Do you know what happened with "http://us.pycon.org/";?
Thank you!
--
. Facundo
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python
On Thu, Jul 3, 2008 at 7:57 AM, M.-A. Lemburg <[EMAIL PROTECTED]> wrote:
> On 2008-07-03 15:21, Jeroen Ruigrok van der Werven wrote:
>>
>> -On [20080703 15:00], M.-A. Lemburg ([EMAIL PROTECTED]) wrote:
>>>
>>> Unicode if full of combining code points - if yo
> Basically everything but string forming or string printing seems to be
> broken for surrogate pairs, from what I can tell.
We probably disagree what "it works correctly" means. I think everything
works correctly.
> Also, I think you are confused about slicing in the middle of a surrogate
> pair
On Thu, Jul 3, 2008 at 13:12, Facundo Batista <[EMAIL PROTECTED]> wrote:
> (sorry for the crossposting)
>
> Do you know what happened with "http://us.pycon.org/";?
Not sure. The machine is still up (it serves www.pycon.org as well).
Either something is misconfigured, or a process can't start, or
s
> 1. System is NOT memory limited (i.e. most desktops): use a UCS-4 Python
> build, which is what most Linux distributions do (I'm not sure about the
> pydotorg provided Windows or Mac OS X builds).
The Windows builds must continue to use a two-byte representation, as
otherwise PythonWin will brea
-On [20080703 19:21], Adam Olsen ([EMAIL PROTECTED]) wrote:
>On Thu, Jul 3, 2008 at 7:57 AM, M.-A. Lemburg <[EMAIL PROTECTED]> wrote:
>> Please remember that lone surrogate pair code points are perfectly
>> valid Unicode code points, nevertheless. Just as a lone combining
>
> Please remember that lone surrogate pair code points are perfectly
> valid Unicode code points, nevertheless. Just as a lone combining
> code point is valid on its own.
Actually, I think they aren't (not any more than an invalid codepoint,
or an unassigned codepoint). They are reserved for UTF-1
> I think you want to use codePointCount() to count the Unicode code points.
> length() returns Unicode code units.
>
> As http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Character.html explains:
>
> In the J2SE API documentation, Unicode code point is used for character
> values in the range b
> Surely it's desirable under all circumstances that
>
>len(u) == sum(1 for c in u)
>
> and that
>
>[c for c in u] == [c[i] for i in range(*len(u))]
>
> How would that play under Jeroen's proposed change?
Yes, but I think the argument is about what "c" is -- a character or a
codepoint.
Daniel Arbuckle wrote:
Regardless, as I said before, nothing justifies silently changing the
meaning of a program based on an option that most users don't set for
themselves and are not aware of.
The premise of this thread seems to be that the majority should suffer
for the benefit of a few
On Thu, Jul 3, 2008 at 10:44 AM, Terry Reedy <[EMAIL PROTECTED]> wrote:
> The premise of this thread seems to be that the majority should suffer for
> the benefit of a few. That is not Python's philosophy.
Who are the many here? Who are the few? I'd venture that (at least for
the foreseeable futu
In Montana visiting. Will be back at the hotel in about 4 hours. Looks
like base site include is missing or has wrong permissions.
On 7/3/08, David Goodger <[EMAIL PROTECTED]> wrote:
> On Thu, Jul 3, 2008 at 13:12, Facundo Batista <[EMAIL PROTECTED]>
> wrote:
>> (sorry for the crossposting)
>>
>>
-On [20080703 19:31], "Martin v. Löwis" ([EMAIL PROTECTED]) wrote:
>Yes, but it is two code units. Python's UTF-16 implementation operates
>on code units, not code points.
Thank you, that is the single most important piece of information I got
about this entire thing becau
On Thu, Jul 3, 2008 at 13:32, David Goodger <[EMAIL PROTECTED]> wrote:
> On Thu, Jul 3, 2008 at 13:12, Facundo Batista <[EMAIL PROTECTED]> wrote:
>> (sorry for the crossposting)
>>
>> Do you know what happened with "http://us.pycon.org/";?
>
> Not sure. The machine is still up (it serves www.pycon.
On Thu, Jul 3, 2008 at 11:35 AM, Jeroen Ruigrok van der Werven
<[EMAIL PROTECTED]> wrote:
> -On [20080703 19:21], Adam Olsen ([EMAIL PROTECTED]) wrote:
>>On Thu, Jul 3, 2008 at 7:57 AM, M.-A. Lemburg <[EMAIL PROTECTED]> wrote:
>>> Please remember that lone surrogate
On 2008-07-03 19:21, Adam Olsen wrote:
On Thu, Jul 3, 2008 at 7:57 AM, M.-A. Lemburg <[EMAIL PROTECTED]> wrote:
On 2008-07-03 15:21, Jeroen Ruigrok van der Werven wrote:
-On [20080703 15:00], M.-A. Lemburg ([EMAIL PROTECTED]) wrote:
Unicode if full of combining code points - if you brea
On 2008-07-03 19:35, Jeroen Ruigrok van der Werven wrote:
-On [20080703 19:21], Adam Olsen ([EMAIL PROTECTED]) wrote:
On Thu, Jul 3, 2008 at 7:57 AM, M.-A. Lemburg <[EMAIL PROTECTED]> wrote:
Please remember that lone surrogate pair code points are perfectly
valid Unicode code
2008/7/3 David Goodger <[EMAIL PROTECTED]>:
> Jeff fixed it. URL rewriting was off by mistake.
Thanks! :)
--
. Facundo
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
___
Python-Dev mailing list
Python-Dev@python.org
http://m
On 2008-07-03 19:44, Terry Reedy wrote:
The premise of this thread seems to be that the majority should suffer
for the benefit of a few. That is not Python's philosophy.
In reality, most Unixes ship with UCS4 builds of Python. Windows
and Mac OS X ship with UCS2 builds. Still, anyone is free t
I've grabbed the latest libffi that contains support for the ARM processor.
I then enable FFI_CLOSURES in the arm/ffi.c file.
When I do this, I get compilation errors that it is missing
ffi_prep_closure.
Is ffi.c up to date for supporting the ARM platform?
Not sure if there is a simple
M.-A. Lemburg wrote:
On 2008-07-03 19:44, Terry Reedy wrote:
The premise of this thread seems to be that the majority should suffer
for the benefit of a few. That is not Python's philosophy.
In reality, most Unixes ship with UCS4 builds of Python. Windows
and Mac OS X ship with UCS2 builds. S
Guido van Rossum wrote:
On Thu, Jul 3, 2008 at 10:44 AM, Terry Reedy <[EMAIL PROTECTED]> wrote:
The premise of this thread seems to be that the majority should suffer for
the benefit of a few. That is not Python's philosophy.
The premise is the OP's idea that Python should switch to all UCS
> Thanks for any help.
This list (python-dev) is not for getting help, but for providing it.
So if you have patches that you would like to discuss, please go
ahead. As you are seeking help, please use [EMAIL PROTECTED]
(aka news:comp.lang.python) instead.
Regards,
Martin
_
On Thu, Jul 3, 2008 at 3:01 PM, Terry Reedy <[EMAIL PROTECTED]> wrote:
>
> The premise is the OP's idea that Python should switch to all UCS4 to create
> a more pure ('ideal') situation or the idea that len(s) should count
> codepoints (correct term?) for all builds as a matter of purity even thoug
On Thu, Jul 3, 2008 at 3:00 PM, Adam Olsen <[EMAIL PROTECTED]> wrote:
> On Thu, Jul 3, 2008 at 3:01 PM, Terry Reedy <[EMAIL PROTECTED]> wrote:
>>
>> The premise is the OP's idea that Python should switch to all UCS4 to create
>> a more pure ('ideal') situation or the idea that len(s) should count
>
> Wrong term - code units and code points are equivalent in UTF-16 and
> UTF-32. What you're looking for is unicode scalar values.
How so? Section 2.5, UTF-16 says
"code points in the supplementary planes, in the range
U+1..U+10, are represented as pairs of 16-bit code units."
So clearl
On Thu, Jul 3, 2008 at 4:21 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote:
> On Thu, Jul 3, 2008 at 3:00 PM, Adam Olsen <[EMAIL PROTECTED]> wrote:
>> On Thu, Jul 3, 2008 at 3:01 PM, Terry Reedy <[EMAIL PROTECTED]> wrote:
>>>
>>> The premise is the OP's idea that Python should switch to all UCS4 to
On Thu, Jul 3, 2008 at 4:50 PM, Adam Olsen <[EMAIL PROTECTED]> wrote:
> Clearly, each surrogate is a valid code point, regardless of encoding.
> A surrogate pair simultaneously represents both one code point (the
> scalar value) and two code points (the surrogate code points). To be
> unambiguous
50 matches
Mail list logo
