[Python-Dev] A much better tokenize.untokenize function
I am creating this post as a courtesy to anyone interested in python's tokenize module. **tl;dr:** Various posts, linked below, discuss a much better replacement for untokenize. Do with it as you will. This code is very unlikely to be buggy. *Please* let me know if you find problems with it. **About the new untokenize** This post: https://groups.google.com/d/msg/leo-editor/DpZ2cMS03WE/VPqtB9lTEAAJ announces a replacement for the untokenize function in tokenize.py: https://github.com/python/cpython/blob/3.8/Lib/tokenize.py To summarize this post: I have "discovered" a spectacular replacement for Untokenizer.untokenize in python's tokenize library module: - The wretched, buggy, and impossible-to-fix add_whitespace method is gone. - The new code has no significant 'if' statements, and knows almost nothing about tokens! As I see it, the only possible failure modes might involve the zero-length line 0. See the above post for a full discussion. **Testing** This post: https://groups.google.com/d/msg/leo-editor/DpZ2cMS03WE/5X8IDzpgEAAJ discusses testing issues. Imo, the new code should easily pass all existing unit tests. The new code also passes a new unit test for Python issue 38663: https://bugs.python.org/issue38663, something existing tests fail to do, even in "compatibility mode" (2-tuples) . Imo, the way is now clear for proper unit testing of python's Untokenize class. In particular, it is, imo, time to remove compatibility mode. This hack has masked serious issues with untokenize: https://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=untokenize&submit=search&status=-1%2C1%2C2%2C3 **Summary** The new untokenize is the way it is written in The Book. I have done the heavy lifting on issue 38663. Python devs are free to do with it as they like. Your choice will not affect me or Leo in any way. The new code will soon become the foundation of Leo's token-oriented commands. Edward P.S. I would imagine that tokenize.untokenize is pretty much off most dev's radar :-) This Engineering Notebook post:https://groups.google.com/d/msg/leo-editor/aivhFnXW85Q/b2a8GHvEDwAJ discusses (in way too much detail :-) why untokenize is important to me. To summarize that post: Imo, python devs are biased in favor of parse trees in programs involving text manipulations. I assert that the "real" black and fstringify tools would be significantly simpler, clearer and faster if they used python's tokenize module instead of python's ast module. Leo's own "beautify" and "fstringify" commands prove my assertion to my own satisfaction. This opinion will be controversial, so I want to make the strongest possible case. I need to prove that handling tokens can be done simply and correctly in all cases. This is a big ask, because python's tokens are complicated. See the Lexical Analysis section of the Python Language Reference. The new untokenize furnishes the required proof, and does so elegantly. EKR ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AXZRDUUAUMID2CPP5A24SCG45C4ZDR6C/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: A much better tokenize.untokenize function
On Sun, Nov 3, 2019 at 4:14 PM Terry Reedy wrote: > > **tl;dr:** Various posts, linked below, discuss a much better replacement for untokenize. > If that were true, I would be interested. But as explained below, I don't believe it. I do not believe that the tone of my post was in any way objectionable. Yes, there was implied criticism of the code. I see no reason for you or anyone else to take that criticism personally. The post script asserted only that token-based code might be a better choice (for some projects) than ast-based code. This is in no way a criticism of tokenize.py, or of any of its authors. Clearly, the new code would have to be repackaged if it were to be made part of tokenize.py. That's all I would like to say now regarding your comments. Perhaps other devs will pick up the ball. Edward ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/J2ASOTDQRXFGQWQBDQYW4UINY4MUJU5B/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: A much better tokenize.untokenize function
On Sun, Nov 3, 2019 at 7:23 PM Edward K. Ream wrote:
Clearly, the new code would have to be repackaged if it were to be made
> part of tokenize.py. That's all I would like to say now regarding your
> comments. Perhaps other devs will pick up the ball.
>
After sleeping on your comments, I would like to distill them to their gist:
1. untokenize does not take a contents argument.
2. untokenize does not do exactly the same thing as the new code.
These are legitimate complaints, and I'm not quite sure what, if anything,
can be done about them.
However, I am not quite ready dismiss the possibility of using the the new
code in the tokenize module, because...
*Motivations*
For convenience, lets call the proposed new code the *gem *:-)
Let's be clear, there is no reason for python libraries to include every
little code gem. However, I have two motivations adding the gem to the
tokenize module:
1. The gem would be useful for any token-based beautification code, such as
the token-based versions of black or fstringify that Leo uses. Furthermore,
the present untokenize method teaches away
<https://www.google.com/search?client=firefox-b-1-d&q=teaches+away> from
the gem.
2. issue 12691 <https://bugs.python.org/issue12691> is troubling, because
the unit tests failed to detect a code blunder in add_whitespace.
*Strengthening unit tests*
It's not clear how the gem could strengthen unit tests. Indeed, the gem is
always going to pass ;-)
This is frustrating, because full sources *are *available to TestRoundtrip
*.*check_roundtrip. Indeed, that method starts with:
if isinstance(f, str):
code = f.encode('utf-8')
else:
code = f.read()
f.close()
If f is a str, then f *is* "contents". Otherwise, "contents" can be
recreated from the file's bytes.
*Summary*
Your objections are largely valid. Nevertheless, the gem may be worth
further consideration.
There is no need to add every little code gem to python's standard library.
In this case, there are two reasons why adding the gem to tokenize.py might
be a good idea:
1. The gem would be of use to anyone writing a token-based tool. Such tools
*do* have access to full sources.
2. The present untokenize method teaches away from the gem. The gem
corrects the mistaken impression that untokenizing in the presence of full
sources is difficult. In fact, it's a snap.
It is an open question whether it might be possible to strengthen the unit
tests for the tokenize module using the gem. Full source*s are* available
to TestRoundtrip*.*check_roundtrip.
Edward
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/S3SHIMJU4ICG7Q6GS2GT36UG4AB5DOWC/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: A much better tokenize.untokenize function
On Mon, Nov 4, 2019 at 5:42 AM Edward K. Ream wrote: > ...there are two reasons why adding the gem to tokenize.py might be a good idea. Heh. I myself am not convinced that the gem needs "advertising" in tokenize.py. Those interested in how Leo's token-based commands work would be more likely to find the gem in Leo's Untokenize class. Edward ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QJK7DYWZY5MMGL526YCXRJEX423M2EBY/ Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-Dev] [Python-3000] Updated release schedule for 2.6 and 3.0
On Fri, Sep 12, 2008 at 7:54 AM, Barry Warsaw <[EMAIL PROTECTED]> wrote: > We had a lot of discussion recently about changing the release schedule and > splitting Python 2.6 and 3.0. There was general consensus that this was a > good idea, in order to hit our October 1 deadline for Python 2.6 final at > least. Does it matter to anyone besides the you, the Python developers, whether the schedule slips by two weeks, or two months, for that matter? I am underwhelmed by 3.0 b3: sax and 2to3 are/were broken. A b4 release for 3.0 (at least) would seem more prudent. Edward ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Static type analysis
Hello all, I'm wondering whether this is the appropriate place to discuss (global) static type analysis, a topic Guido mentioned around the 28 min mark in his PyCon 2012 keynote, http://pyvideo.org/video/956/keynote-guido-van-rossum This is a topic that has interested me for a long time, and it has important implications for Leo. Just now I have some free time to devote to it. Edward -- Edward K. Ream email: edream...@gmail.com Leo: http://webpages.charter.net/edreamleo/front.html -- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Announcing the python-static-type-checking google group
Hello all, GvR has asked me to announce the python-static-type-checking google group http://groups.google.com/group/python-static-type-checking to python-dev. Consider it announced. Anyone from python-dev who likes may become a member. Here is the "About this group" posting: Q This group got its start as a response to GvR's Keynote address at PyCon 2012, http://pyvideo.org/video/956/keynote-guido-van-rossum specifically, his remarks about static type checking beginning in the 28'th minute. Guido and I have been emailing about this topic for a day or so. These emails will form the first few posts of this group. This group is public, in the sense that anyone can see it, but initial invitations will go only to those who have written significant python tools related to type checking. It will probably be best to have group have a low profile at first. Having said that, I encourage members to invite others who may be able to contribute. Experience with other projects shows that crucial contributions may come from unlikely sources. Because of spam, this will be a moderated group. New members must request permission to post. Q Edward ---------- Edward K. Ream email: edream...@gmail.com Leo: http://webpages.charter.net/edreamleo/front.html -- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
