Re: [Tutor] how can I use unicode in ctypes?
- Original Message - > From: eryksun > To: Albert-Jan Roskam > Cc: Python Mailing List > Sent: Friday, December 7, 2012 7:39 PM > Subject: Re: [Tutor] how can I use unicode in ctypes? > > On Thu, Dec 6, 2012 at 2:39 PM, Albert-Jan Roskam wrote: >> >> http://pastecode.org/index.php/view/29608996 >> >> import ctypes >> s = u'\u0627\u0644\u0633\u0644\u0627\u0645' >> v = ctypes.c_wchar_p(s) >> print v # prints > c_wchar_p(u'\u0627\u0644\u0633\u0644\u0627\u0645') >> v.value # prints > u'\u0627\u0644\u0633\u0644\u0627\u0645' > > Your decorator could end up encoding str or decoding unicode. > Typically this gets routed through the default encoding (i.e. ASCII) > and probably triggers a UnicodeDecodeError or UnicodeEncodeError. I'd > limit encoding to unicode and decoding to bytes/str. Thanks. I modified it. Kinda annoying that the default encoding is ascii, but I read that it could be changed with sys.setdefaultencoding and reload(sys) > On the subject of wchar_t, here's a funny rant: > > http://losingfight.com/blog/2006/07/28/wchar_t-unsafe-at-any-size/ > Interesting article. I experimented with a little in ctypes and also concluded that I couldn't use it for my purposes. Apparently this is still not an entirely mature area of Python. Very useful to know. Thank you very much for this detailed information! > The base type for Unicode in CPython isn't wchar_t on all > platforms/builds. It depends on Py_UNICODE_SIZE (2 or 4 bytes) vs > sizeof(wchar_t) (also on whether wchar_t is unsigned, but that's not > relevant here). 3.3 is in its own flexible universe. > > I recently came across a bug in create_unicode_buffer on Windows > Python 3.3. The new flexible string implementation uses Py_UCS4 > instead of creating surrogate pairs on Windows. However, given that > the size of c_wchar is 2 [bytes] on Windows, create_unicode_buffer > still needs to factor in the surrogate pairs by calculating the target > size = len(init) + sum(ord(c) > 0x for c in init) + 1. Naively it > uses size = len(init) + 1, which fails if the string has multiple > non-BMP characters. > > Here's another ctypes related issue. On a narrow build prior to 3.2, > PyUnicode_AsWideChar returns a wide-character string that may contain > surrogate pairs even if wchar_t is 32-bit. That isn't well-formed > UTF-32. This was fixed in 3.2 as part of fixing a ctypes bug. ctypes > u_set (type 'u' is c_wchar) was modified to use an updated > PyUnicode_AsWideChar and Z_set (type 'Z' is c_wchar_p) was modified to > use the new PyUnicode_AsWideCharString. > > 3.2.3 links: > > u_set: > http://hg.python.org/cpython/file/3d0686d90f55/Modules/_ctypes/cfield.c#l1202 > > Z_set: > http://hg.python.org/cpython/file/3d0686d90f55/Modules/_ctypes/cfield.c#l1401 > > The new PyUnicode_AsWideChar and PyUnicode_AsWideCharString call the > helper function unicode_aswidechar. This was added in 3.2 to handle > the different cases of Py_UNICODE_SIZE more carefully: > > http://hg.python.org/cpython/file/3d0686d90f55/Objects/unicodeobject.c#l1187 > > Py_UNICODE_SIZE == SIZEOF_WCHAR_T > Py_UNICODE_SIZE == 2 && SIZEOF_WCHAR_T == 4 > Py_UNICODE_SIZE == 4 && SIZEOF_WCHAR_T == 2 > > The 2nd case takes advantage of the larger wchar_t to recombine > surrogate pairs. The 3rd case creates surrogate pairs instead of > truncating the character code. (Note: this helper was updated in 3.3 > to use the new function PyUnicode_AsUnicodeAndSize.) > > Prior to 3.2, PyUnicode_AsWideChar wasn't nearly as careful. See the > version in 3.1.5: > > http://hg.python.org/cpython/file/7395330e495e/Objects/unicodeobject.c#l1085 > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] memoize, lookup, or KIS?
(this thread may be closed already but...) while looking for something else, I found another cool way of memoizing: http://effbot.org/zone/default-values.htm It uses an empty dictionary as a function argument. Usually a source of confusion (for me at least!), but in this case I find it quite useful. Regards, Albert-Jan ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] how can I use unicode in ctypes?
On Mon, Dec 10, 2012 at 5:15 AM, Albert-Jan Roskam wrote: > > Thanks. I modified it. Kinda annoying that the default encoding is ascii, but > I read that it could be changed with sys.setdefaultencoding and reload(sys) I wouldn't go against the grain here, especially for library code. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Unexpected results with obj.method().method()
Thanks eryksun, that was the bug. Thanks for pointing out the tabs as well,
they were added by vim's autoindent. I've set expandtab for python files
now.
I decided to change the code such that current_player and turn_number are
hidden behind properties meaning I won't overwrite them accident or
stupidity later.
On 6 December 2012 01:31, eryksun wrote:
> On Wed, Dec 5, 2012 at 1:11 PM, C M Caine wrote:
> >
> > The full code is on pastebin http://pastebin.com/tUh0W5Se
> >
> import game
> S = game.State()
> S1 = S.move_state(1).move_state("SWAP")
> S2 = S.move_state(1)
> S3 = S2.move_state("SWAP")
> S1 == S3
> > False
>
> In lines 156-160 you change players by mutating the object. You need
> to use a local variable such as "next_player" and return State(board,
> self.turn_number + 1, next_player).
>
> Also, you're mixing tabs and 2-space indents. Please choose one or the
> other. The most popular style is 4-space indents, as recommended by
> the PEP 8 style guide.
>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Unexpected results with obj.method().method()
Thanks for the advice. As is often the case with these things, eryksun
pointed out a stupid mistake I'd made (mutating part of an immutable class)
that I should have seen.
On 6 December 2012 00:50, Oscar Benjamin wrote:
> On 5 December 2012 18:11, C M Caine wrote:
> > Dear all,
> >
> > I've written a class State that subclasses tuple. The class has a method
> > move_state that takes a move and returns a new state object representing
> the
> > new state of the game.
> >
> > I would expect S1 and S3 to be equal on the last line here, but they are
> > not.
> >
> import game
> S = game.State()
> S1 = S.move_state(1).move_state("SWAP")
> S2 = S.move_state(1)
> S3 = S2.move_state("SWAP")
> S1 == S3
> > False
> >
> > Printing the two states shows that they have very different internal
> states.
> >
> print S1
> > (8, 8, 8, 8, 8, 8, 0)
> > 1 0
> > (7, 7, 7, 7, 7, 7, 7)
> print S3
> > (7, 7, 7, 7, 7, 7, 7)
> > 0 1
> > (0, 8, 8, 8, 8, 8, 8)
>
> From your description above I have very little idea what you're trying
> to do. You have specified what you were expecting to happen why you're
> not happy with what actually happened, which is good. I still don't
> understand the problem, though. What is the *relevant* code that
> didn't do what you expected?
>
> > If anyone is interested, State represents the state of a 7 7 Kalah board.
>
> I don't know what a Kalah board is.
>
> > The full code is on pastebin http://pastebin.com/tUh0W5Se
>
> You were right not to post this code directly in your email as it's
> too big. For the same reason, though, I'm not prepared to read it
> through and try to understand the problem.
>
> It would be better if you could trim your problem down to a short
> example so that you can then post the full example. An important side
> effect of this process is that you will often discover the cause of
> the problem yourself before completing your email to the list.
>
> > Are my expectations faulty? (I hope not)
> > Have I made some mistake in my code to get these results?
>
> Probably at least one of the above is true, but I can't say much more
> than that. Have a read of http://sscce.org/ for some advice about how
> to post problems to a mailing list. If you follow the advice there you
> will find that
> 1) You will often be able to solve your problem yourself.
> 2) If you do post your problem to a mailing list you will be more
> likely to get a helpful response.
>
>
> Oscar
>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
