Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-08 Thread M.-A. Lemburg
Adam Olsen wrote: > On Sun, Sep 20, 2009 at 10:17, Zooko O'Whielacronx wrote: >> On Sun, Sep 20, 2009 at 8:27 AM, Antoine Pitrou wrote: >>> AFAIK, C extensions should fail loading when they have the wrong UCS2/4 >>> setting. >> >> That would be an improvement! Unfortunately we instead get myste

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread Adam Olsen
On Sun, Sep 20, 2009 at 10:17, Zooko O'Whielacronx wrote: > On Sun, Sep 20, 2009 at 8:27 AM, Antoine Pitrou wrote: >> AFAIK, C extensions should fail loading when they have the wrong UCS2/4 >> setting. > > That would be an improvement!  Unfortunately we instead get mysterious > misbehavior of th

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread Neil Hodgson
Ronald Oussoren: > Both Carbon and the modern APIs use UTF-16. If Unicode size standardization is seen as sufficiently beneficial then UTF-16 would be more widely applicable than UTF-32. Unix mostly uses 8-bit APIs which are either explicitly UTF-8 (such as GTK+) or can accept UTF-8 when the l

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread M.-A. Lemburg
Ronald Oussoren wrote: > > On 7 Oct, 2009, at 22:13, M.-A. Lemburg wrote: > >> Ronald Oussoren wrote: >>> >>> On 7 Oct, 2009, at 20:05, M.-A. Lemburg wrote: If we do go for a change, we should use sizeof(wchar_t) as basis for the new default - on all platforms that provid

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread Ronald Oussoren
On 7 Oct, 2009, at 22:13, M.-A. Lemburg wrote: Ronald Oussoren wrote: On 7 Oct, 2009, at 20:05, M.-A. Lemburg wrote: If we do go for a change, we should use sizeof(wchar_t) as basis for the new default - on all platforms that provide a wchar_t type. I'd be -1 on that. Sizeof(wchar_t) is

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread M.-A. Lemburg
Ronald Oussoren wrote: > > On 7 Oct, 2009, at 20:05, M.-A. Lemburg wrote: >> >> >> If we do go for a change, we should use sizeof(wchar_t) >> as basis for the new default - on all platforms that >> provide a wchar_t type. > > I'd be -1 on that. Sizeof(wchar_t) is 4 on OSX, but all non-Unix API's

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread Ronald Oussoren
On 7 Oct, 2009, at 20:05, M.-A. Lemburg wrote: If we do go for a change, we should use sizeof(wchar_t) as basis for the new default - on all platforms that provide a wchar_t type. I'd be -1 on that. Sizeof(wchar_t) is 4 on OSX, but all non-Unix API's that deal with Unicode text use ucs16.

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread Antoine Pitrou
Zooko O'Whielacronx gmail.com> writes: > > I accidentally sent this letter just to MAL when I intended it to > python-dev. Please read it, as it explains why the issue I'm raising > is not just the "we should switch to ucs4 because it is better" issue > that was previously settled by GvR. For

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-07 Thread M.-A. Lemburg
Zooko O'Whielacronx wrote: > Dear MAL and python-dev: > > I failed to explain the problem that users are having. I will try > again, and this time I will omit my ideas about how to improve things > and just focus on describing the problem. > > Some users are having trouble using Python packages

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-05 Thread Antoine Pitrou
Le lundi 05 octobre 2009 à 19:18 +0200, Jan Matejek a écrit : > > I > don't see what is bad about improving compatibility in a place where the > setting doesn't hurt one way or the other. I can't speak for Mandriva, but I'm sure they care more about not breaking user installs when they upgrade to

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-10-05 Thread Jan Matejek
Dne 20.9.2009 18:42, Antoine Pitrou napsal(a): > Le Sun, 20 Sep 2009 10:33:23 -0600, Zooko O'Whielacronx a écrit : >> >> By the way, I was investigating this, and discovered an issue on the >> Mandriva tracker which suggests that they intend to switch to UCS4 in >> the next release in order to av

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-29 Thread Martin v. Löwis
>> I've also encountered this trap multiple times. Obviously, the problem >> is not rebuilding Python which is quick, but to figure out the correct >> configure option to use (--enable-unicode=ucs4). Others have also >> spent some time scratching their heads over the strange >> PyUnicodeUCS4_FromUn

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-29 Thread Zooko O'Whielacronx
Dear MAL and python-dev: I failed to explain the problem that users are having. I will try again, and this time I will omit my ideas about how to improve things and just focus on describing the problem. Some users are having trouble using Python packages containing binary extensions on Linux. I

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-29 Thread Antoine Pitrou
Hello, > I've also encountered this trap multiple times. Obviously, the problem > is not rebuilding Python which is quick, but to figure out the correct > configure option to use (--enable-unicode=ucs4). Others have also > spent some time scratching their heads over the strange > PyUnicodeUCS4_Fro

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-29 Thread Björn Lindqvist
2009/9/28 James Y Knight : >> People building their own Python version will usually also build >> their own extensions, so I don't really believe that the above >> scenario is very common. > > I'd just like to note that I've run into this trap multiple times. I built a > custom python, and expected

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-28 Thread Martin v. Löwis
James Y Knight wrote: > On Sep 28, 2009, at 4:25 AM, M.-A. Lemburg wrote: >> Distributions should really not be put in charge of upstream >> coding design decisions. > > I don't think you can blame distros for this one > > From PEP 0261: > It is also proposed that one day --enable-unicode

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-28 Thread M.-A. Lemburg
James Y Knight wrote: > On Sep 28, 2009, at 4:25 AM, M.-A. Lemburg wrote: >> Distributions should really not be put in charge of upstream >> coding design decisions. > > I don't think you can blame distros for this one > > From PEP 0261: > It is also proposed that one day --enable-unicode

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-28 Thread James Y Knight
On Sep 28, 2009, at 4:25 AM, M.-A. Lemburg wrote: Distributions should really not be put in charge of upstream coding design decisions. I don't think you can blame distros for this one From PEP 0261: It is also proposed that one day --enable-unicode will just default to the width o

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-28 Thread M.-A. Lemburg
M.-A. Lemburg wrote: > Also note that Python will complain loudly when you try to load > a UCS2 extension in a UCS4 build and vice-versa. We've made sure > that any extension using the Python Unicode C API has to be built > for the same UCS version of Python. This is done by using different > names

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-28 Thread M.-A. Lemburg
Zooko O'Whielacronx wrote: > Folks: > > I'm sorry, I think I didn't make my concern clear. My users, and lots > of other users, are having a problem with incompatibility between > Python binary extension modules. One way to improve the situation > would be if the Python devs would use their "bul

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Martin v. Löwis
Zooko O'Whielacronx wrote: > I'm sorry, I should have mentioned that I did read those archives > before I posted my letter. That discussion was all about whether UCS2 > or UCS4 is better. I consider that question to be mostly irrelevant > to this issue, which is about compatibility for people who

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread M.-A. Lemburg
Zooko O'Whielacronx wrote: > On Sun, Sep 20, 2009 at 8:27 AM, Antoine Pitrou wrote: >> >> What "binaries" are you talking about? > > I mean extension modules with native code, which means .so shared > library files on unix. Those will not load unless they are for the right UCS-version of Python.

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Antoine Pitrou
Le Sun, 20 Sep 2009 10:33:23 -0600, Zooko O'Whielacronx a écrit : > > By the way, I was investigating this, and discovered an issue on the > Mandriva tracker which suggests that they intend to switch to UCS4 in > the next release in order to avoid compatibility problems like these. Trying to use

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Antoine Pitrou
Le Sun, 20 Sep 2009 10:17:45 -0600, Zooko O'Whielacronx a écrit : > >> AFAIK, C extensions should fail loading when they have the wrong UCS2/4 >> setting. > > That would be an improvement! Unfortunately we instead get mysterious > misbehavior of the module, e.g.: > > http://bugs.python.org/setu

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Zooko O'Whielacronx
On Sun, Sep 20, 2009 at 8:27 AM, Antoine Pitrou wrote: > For information, all Mandriva versions I've used until now have had their > Python's built with UCS2 (maxunicode == 65535). By the way, I was investigating this, and discovered an issue on the Mandriva tracker which suggests that they inten

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Zooko O'Whielacronx
On Sun, Sep 20, 2009 at 8:27 AM, Antoine Pitrou wrote: > > What "binaries" are you talking about? I mean extension modules with native code, which means .so shared library files on unix. > AFAIK, C extensions should fail loading when they have the wrong UCS2/4 > setting. That would be an impro

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Antoine Pitrou
Zooko O'Whielacronx gmail.com> writes: > > Users occasionally get binaries built for a > compatible Linux and Python version but with a different UCS2-vs-UCS4 setting, > and those users get mysterious memory corruption errors which are hard to > diagnose. What "binaries" are you talking about? A

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Zooko O'Whielacronx
I'm sorry, I should have mentioned that I did read those archives before I posted my letter. That discussion was all about whether UCS2 or UCS4 is better. I consider that question to be mostly irrelevant to this issue, which is about compatibility for people who don't choose to configure that set

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

2009-09-20 Thread Benjamin Peterson
2009/9/20 Zooko O'Whielacronx : > Dear Pythonistas: > > This issue causes serious problems.  Users occasionally get binaries built > for a > compatible Linux and Python version but with a different UCS2-vs-UCS4 setting, > and those users get mysterious memory corruption errors which are hard to >