[issue9969] tokenize: add support for tokenizing 'str' objects

2018-05-17 Thread Thomas Kluyver
Thomas Kluyver added the comment: I've opened a PR for issue #12486, which would make the existing but undocumented 'generate_tokens' function public: https://github.com/python/cpython/pull/6957 I agree that it would be good to design a nicer API for this, but the perfect shouldn't be the en

[issue9969] tokenize: add support for tokenizing 'str' objects

2015-10-04 Thread Martin Panter
Martin Panter added the comment: Actually maybe Issue 12486 is good enough to fix this too. With the patch proposed there, tokenize_basestring("source") would just be equivalent to tokenize(StringIO("source").readline) -- ___ Python tracker

[issue9969] tokenize: add support for tokenizing 'str' objects

2015-10-04 Thread Martin Panter
Martin Panter added the comment: I left some comments. Also, it would be nice to use the new function in the documentation example, which currently suggests tunnelling through UTF-8 but not adding an encoding comment. And see the patch for Issue 12486, which highlights a couple of other places

[issue9969] tokenize: add support for tokenizing 'str' objects

2012-10-15 Thread Serhiy Storchaka
Changes by Serhiy Storchaka : -- versions: +Python 3.4 -Python 3.2, Python 3.3 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue9969] tokenize: add support for tokenizing 'str' objects

2011-09-04 Thread Meador Inge
Meador Inge added the comment: Attached is a first cut at a patch. -- keywords: +patch stage: needs patch -> patch review Added file: http://bugs.python.org/file23099/issue9969.patch ___ Python tracker

[issue9969] tokenize: add support for tokenizing 'str' objects

2011-05-31 Thread Thomas Kluyver
Changes by Thomas Kluyver : -- nosy: +takluyver ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue9969] tokenize: add support for tokenizing 'str' objects

2010-11-20 Thread Nick Coghlan
Nick Coghlan added the comment: The idea is bring the API up a level, and also take care of wrapping the file-like object around the source string/byte sequence. -- ___ Python tracker _

[issue9969] tokenize: add support for tokenizing 'str' objects

2010-11-20 Thread Abhay Saxena
Abhay Saxena added the comment: If the goal is tokenize(...) accepting a text I/O readline, we already have the (undocumented) generate_tokens(readline). -- nosy: +ark3 ___ Python tracker _

[issue9969] tokenize: add support for tokenizing 'str' objects

2010-09-29 Thread Nick Coghlan
Nick Coghlan added the comment: As per Antoine's comment on #9873, requiring a real string via isinstance(source, str) to trigger the string IO version is likely to be cleaner than attempting to duck-type this. Strings are an area where we make so many assumptions about the way their internal

[issue9969] tokenize: add support for tokenizing 'str' objects

2010-09-28 Thread STINNER Victor
STINNER Victor added the comment: See also issue #4626 which introduced PyCF_IGNORE_COOKIE and PyPARSE_IGNORE_COOKIE flags to support unicode string for the builtin compile() function. -- nosy: +haypo ___ Python tracker

[issue9969] tokenize: add support for tokenizing 'str' objects

2010-09-28 Thread Nick Coghlan
Nick Coghlan added the comment: Possible approach (untested): def get_tokens(source): if hasattr(source, "encode"): # Already decoded, so bypass encoding detection return _tokenize(io.StringIO(source).readline, None) # Otherwise attempt to detect the correct encoding

[issue9969] tokenize: add support for tokenizing 'str' objects

2010-09-28 Thread Michael Foord
Michael Foord added the comment: Note from Nick Coghlan from the Python-dev discussion: A very quick scan of _tokenize suggests it is designed to support detect_encoding returning None to indicate the line iterator will return already decoded lines. This is confirmed by the fact the standard li

[issue9969] tokenize: add support for tokenizing 'str' objects

2010-09-28 Thread Michael Foord
Changes by Michael Foord : -- nosy: +michael.foord ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.p

[issue9969] tokenize: add support for tokenizing 'str' objects

2010-09-28 Thread Meador Inge
Changes by Meador Inge : -- components: +Library (Lib) ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://ma

[issue9969] tokenize: add support for tokenizing 'str' objects

2010-09-28 Thread Meador Inge
New submission from Meador Inge : Currently with 'py3k' only 'bytes' objects are accepted for tokenization: >>> import io >>> import tokenize >>> tokenize.tokenize(io.StringIO("1+1").readline) Traceback (most recent call last): File "", line 1, in File "/Users/minge/Code/python/py3k/Lib/tok