[issue42687] tokenize module does not recognize Barry as FLUFL

Change by Erik Soma : -- stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___ Python-bugs-list mailing

[issue42687] tokenize module does not recognize Barry as FLUFL

Batuhan Taskaya added the comment: I concur with Terry. -- nosy: +BTaskaya ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue42687] tokenize module does not recognize Barry as FLUFL

Terry J. Reedy added the comment: I strongly disagree. '<>' is not a legal operator any more. It is a parse-time syntax error. Whatever historical artifact is left in the CPython tokenizer, recognizing '<>' is not exposed to Python code. >>> p = ast.parse('a <> b') Traceback (most recent

[issue42687] tokenize module does not recognize Barry as FLUFL

Change by Erik Soma : -- keywords: +patch pull_requests: +22722 stage: -> patch review pull_request: https://github.com/python/cpython/pull/23857 ___ Python tracker ___ __

[issue42687] tokenize module does not recognize Barry as FLUFL

New submission from Erik Soma : '<>' is not recognized by the tokenize module as a single token, instead it is two tokens. ``` $ python -c "import tokenize; import io; import pprint; pprint.pprint(list(tokenize.tokenize(io.BytesIO(b'<>').readline)))" [TokenInfo(type=62 (ENCODING), string='utf