[issue12691] tokenize.untokenize is broken

Terry J. Reedy added the comment: If there are, I can't remember. This was one or 7 or 8 untokenize issues with about 5 separate bugs between them. If there are any current untokenize issues not covered by some other open issue, then a new issue should be opened. -- resolution: -> f

[issue12691] tokenize.untokenize is broken

R. David Murray added the comment: Is there anything left to do here? -- nosy: +r.david.murray ___ Python tracker ___ ___ Python-bugs-

[issue12691] tokenize.untokenize is broken

Roundup Robot added the comment: New changeset 0f0e9b7d4f1d by Terry Jan Reedy in branch '2.7': Issue #9974: When untokenizing, use row info to insert backslash+newline. http://hg.python.org/cpython/rev/0f0e9b7d4f1d New changeset 24b4cd5695d9 by Terry Jan Reedy in branch '3.3': Issue #9974: When

[issue12691] tokenize.untokenize is broken

Roundup Robot added the comment: New changeset 8d6dd02a973f by Terry Jan Reedy in branch '3.3': Issue #20750, Enable roundtrip tests for new 5-tuple untokenize. The http://hg.python.org/cpython/rev/8d6dd02a973f -- ___ Python tracker

[issue12691] tokenize.untokenize is broken

Gareth Rees added the comment: Thanks for your work on this, Terry. I apologise for the complexity of my original report, and will try not to do it again. -- ___ Python tracker

[issue12691] tokenize.untokenize is broken

Terry J. Reedy added the comment: I fixed the assert and dropped first iter compat-mode token bugs one-by-one by writing narrow unittests that fail and code that makes them pass. I am now working on the '\' continuation issue. That is the subject of #9974, which has a nearly identical patch. W

[issue12691] tokenize.untokenize is broken

Terry J. Reedy added the comment: The problem of the first iterator pair token being discarded is the subject of #8478. Consider that part of this issue as being closed as a duplicate. The issue of a string being returned if there is no encoding should have been opened as a separate issue, and

[issue12691] tokenize.untokenize is broken

Roundup Robot added the comment: New changeset c896d292080a by Terry Jan Reedy in branch '2.7': Untokenize: An logically incorrect assert tested user input validity. http://hg.python.org/cpython/rev/c896d292080a New changeset 51e5a89afb3b by Terry Jan Reedy in branch '3.3': Untokenize: An logica

[issue12691] tokenize.untokenize is broken

Changes by Terry J. Reedy : -- assignee: docs@python -> terry.reedy nosy: +terry.reedy stage: test needed -> patch review versions: +Python 2.7, Python 3.3 ___ Python tracker ___

[issue12691] tokenize.untokenize is broken

Gareth Rees added the comment: I did some research on the cause of this issue. The assertion was added in this change by Jeremy Hylton in August 2006: (The corresponding Mercurial commit is here:

[issue12691] tokenize.untokenize is broken

Yury Selivanov added the comment: Gareth, Thanks a lot for such a comprehensive writeup and the patch. Please give me a day or two to do the review. -- ___ Python tracker ___ _

[issue12691] tokenize.untokenize is broken

Changes by Gareth Rees : Removed file: http://bugs.python.org/file33919/Issue12691.patch ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue12691] tokenize.untokenize is broken

Changes by Gareth Rees : -- assignee: -> docs@python components: +Documentation, Tests nosy: +docs@python ___ Python tracker ___ ___

[issue12691] tokenize.untokenize is broken

Gareth Rees added the comment: This morning I noticed that I had forgotten to update the library reference, and I also noticed two more problems to add to the list above: 6. Although Lib/test/test_tokenize.py looks like it contains tests for backslash-newline handling, these tests are ineffec

[issue12691] tokenize.untokenize is broken

Changes by Gareth Rees : -- nosy: +benjamin.peterson ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://ma

[issue12691] tokenize.untokenize is broken

Gareth Rees added the comment: Yury, let me see if I can move this issue forward. I clearly haven't done a good job of explaining these problems, how they are related, and why it makes sense to solve them together, so let me have a go now. 1. tokenize.untokenize() raises AssertionError if you pa

[issue12691] tokenize.untokenize is broken

Yury Selivanov added the comment: bump? -- nosy: +yselivanov versions: +Python 3.4 -Python 3.2, Python 3.3 ___ Python tracker ___ ___

[issue12691] tokenize.untokenize is broken

Meador Inge added the comment: I will take a look. As it stands the current patch fixes way too many issues. Patches should fix *one* issue at a time. I will look at fixing the original assert problem and at opening new issues for the others (assuming there aren't already issues for them). --

[issue12691] tokenize.untokenize is broken

Thomas Kluyver added the comment: Is there anything I can do to push this forwards? I'm trying to use tokenize and untokenize in IPython, and for now I'm going to have to maintain our own copies of it (for Python 2 and 3), because I keep running into problems with the standard library module.

[issue12691] tokenize.untokenize is broken

Changes by Meador Inge : -- nosy: +meador.inge ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue12691] tokenize.untokenize is broken

Changes by Serhiy Storchaka : -- nosy: +serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:/

[issue12691] tokenize.untokenize is broken

Gareth Rees added the comment: Thanks Ezio for the review. I've made all the changes you requested, (except for the re-ordering of paragraphs in the documentation, which I don't want to do because that would lead to the "round-trip property" being mentioned before it's defined). Revised patch

[issue12691] tokenize.untokenize is broken

Gareth Rees added the comment: Please find attached a patch containing four bug fixes for untokenize(): * untokenize() now always returns a bytes object, defaulting to UTF-8 if no ENCODING token is found (previously it returned a string in this case). * In compatibility mode, untokenize() succ

[issue12691] tokenize.untokenize is broken

Gareth Rees added the comment: I think I can make these changes independently and issue two patches, one fixing the problems with untokenize listed here, and another improving tokenize. I've just noticed a third bug in untokenize: in full mode, it doesn't handle backslash-continued lines corr

[issue12691] tokenize.untokenize is broken

Changes by Daniel Urban : -- nosy: +durban ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue12691] tokenize.untokenize is broken

Changes by Eric Snow : -- nosy: +ericsnow ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.or

[issue12691] tokenize.untokenize is broken

Sandro Tosi added the comment: The general rule would be to have separate patches. But in this case, if we have interdipendent changes, then those should be "packed" in a single patch (f.e. if changes to tokenize break untokenize, than those parts should be joined). -- _

[issue12691] tokenize.untokenize is broken

Gareth Rees added the comment: See my last paragraph: I propose to deliver a single patch that fixes both this bug and issue12675. I hope this is OK. (If you prefer, I'll try to split the patch in two.) I just noticed another bug in untokenize(): in compatibility mode, if untokenize() is pas

[issue12691] tokenize.untokenize is broken

Sandro Tosi added the comment: Hi Gareth, would you like to provide a patch to fix the bug you spotted and add the relative case into the testsuite? -- nosy: +sandro.tosi ___ Python tracker __

[issue12691] tokenize.untokenize is broken

Changes by Ezio Melotti : -- nosy: +ezio.melotti stage: -> test needed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsub

[issue12691] tokenize.untokenize is broken

New submission from Gareth Rees : tokenize.untokenize is completely broken. Python 3.2.1 (default, Jul 19 2011, 00:09:43) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import tokenize, io >>>