On 28/09/2010 12:19, Antoine Pitrou wrote:
On Mon, 27 Sep 2010 23:45:45 -0400
Steve Holden<st...@holdenweb.com> wrote:
On 9/27/2010 11:27 PM, Benjamin Peterson wrote:
2010/9/27 Meador Inge<mead...@gmail.com>:
which, as seen in the trace, is because the 'detect_encoding' function in
'Lib/tokenize.py' searches for 'BOM_UTF8' (a 'bytes' object) in the string
to tokenize 'first' (a 'str' object). It seems to me that strings should
still be able to be tokenized, but maybe I am missing something.
Is the implementation of 'detect_encoding' correct in how it attempts to
determine an encoding or should I open an issue for this?
Tokenize only works on bytes. You can open a feature request if you desire.
Working only on bytes does seem rather perverse.
I agree, the morality of bytes objects could have been better :)
The reason for working with bytes is that source data can only be
correctly decoded to text once the encoding is known. The encoding is
determined by reading the encoding cookie.
I certainly wouldn't be opposed to an API that accepts a string as well
though.
All the best,
Michael
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog
READ CAREFULLY. By accepting and reading this email you agree, on behalf of
your employer, to release me from all obligations and waivers arising from any
and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap,
clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and
acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in perpetuity, without
prejudice to my ongoing rights and privileges. You further represent that you
have the authority to release me from any BOGUS AGREEMENTS on behalf of your
employer.
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
