[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Pablo Galindo Salgado added the comment: For instance, consider this: x: ( 3 + 4 + 5) = 3 Once the expression in the annotations is parsed, the value of parser->tok->buf is " +5) = 3" and the tokenizer has thrown away the rest of the lines and the information there. There is n

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Eric V. Smith added the comment: It's true that f-string expressions can't contain newlines. f-strings are definitely easier, because the tokenizer has already tokenized the string from the input, so I'm just remembering pointers inside the tokenized string. I was thinking that maybe you co

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Pablo Galindo Salgado added the comment: > I remember the char* pointer where the expression starts, then I parse the > expression into an AST, then I note the char* pointer where the expression > ended. The text between those is what's output before the equal sign []. This > is how I preser

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Eric V. Smith added the comment: For what it's worth, here's how f-strings with the "=" feature work: I remember the char* pointer where the expression starts, then I parse the expression into an AST, then I note the char* pointer where the expression ended. The text between those is what's

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Guido van Rossum added the comment: Too bad. :-( -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Pablo Galindo Salgado added the comment: After some extra testing, we found that is not enough with handling spaces, unfortunately, as things like the walrus need also the parentheses to be preserved. Given that multi-line strings will complicate the code considerably, I am going to drop thi

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Pablo Galindo Salgado added the comment: > Yeah, we are trying to modify the patch to automatically add spaces between > keywords and other tokens. Once we can see how this looks, we could think if is worth the effort or is better to abandon the idea. One could also argue that having multi-l

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Pablo Galindo Salgado added the comment: > Okay, though aren't there cases where spaces are necessary? E.g. between > names or between a name or keyword and a string or number. Yeah, we are trying to modify the patch to automatically add spaces between keywords and other tokens. > Assuming

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Guido van Rossum added the comment: > the string generated for multi line annotations [...] won't have spaces > between characters. Okay, though aren't there cases where spaces are necessary? E.g. between names or between a name or keyword and a string or number. Assuming this will only hap

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Pablo Galindo Salgado added the comment: > We cannot declare those illegal. They won't be illegal, is just that the string generated for multi line annotations will be a bit uglier because it won't have spaces between characters. -- ___ Python tr

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Guido van Rossum added the comment: Okay, but doesn't *any* way of handling multi-line annotations spoil the wins? We cannot declare those illegal. (Though we can switch to a different way of representing those.) -- ___ Python tracker

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Pablo Galindo Salgado added the comment: > I just wonder why bother changing this unless we reproduce the spaces in the string exactly. Maintenance reasons: it removes over 1000 lines of hand written code. Additionally, makes the compiler a bit faster because it does not need to transform t

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Guido van Rossum added the comment: I know the final result is the same (i.e. the AST is identical and the string differs only in the use of whitespace). I just wonder why bother changing this unless we reproduce the spaces in the string exactly. -- ___

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Pablo Galindo Salgado added the comment: > Hm, this buffer thing is not what I had expected. I think it's a show-stopper > -- if we don't reproduce spaces exactly. Hummm, I think I may have not clarified this in my previous messages: the space issue is an aesthetic one. The AST generated wit

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Batuhan Taskaya added the comment: It is not actually much of a difference; $ cat t.py def func( arg: typing.Callable[ [int, str], str # test ] ): pass import typing print("Annotations: ", func.__annotations__) print("get_type_hints: ", typing.get_type_hints(func))

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Guido van Rossum added the comment: Hm, this buffer thing is not what I had expected. I think it's a show-stopper -- if we don't reproduce spaces exactly. Maybe we should just buffer the entire input always? A way to handle multi-line annotations would be to surround them with parentheses?

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Pablo Galindo Salgado added the comment: > What's the issue exactly with multi-line annotations? The issue is that for grabbing the source, we are inspecting the tokenize buffer and using the information in the expr node of the annotation to know what we need to pick from it. In multi-line

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Guido van Rossum added the comment: What's the issue exactly with multi-line annotations? Is it that they can't be parsed by eval()? Could that be fixed by replacing comments and newlines with spaces? (Either in the parser or in get_type_hints() -- actually ForwardRef._evaluate(), which call

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Batuhan Taskaya added the comment: s/For 516389 annotated arguments/For 516389 arguments/. Example usages; https://search.tree.science/?query=AnnAssign(annotation%20=%20expr(lineno%20=%20not%20~end_lineno)) https://search.tree.science/?query=arg(annotation=expr(lineno%20=%20not%20~end_lineno)

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Batuhan Taskaya added the comment: After scanning 1552 annotated assignments in over 25k python modules (from most populer packages in PyPI) there are only 6 AnnAssign nodes where the annotations are multiline. For 516389 annotated arguments, only 20 of them are multiline annotations. -

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Pablo Galindo Salgado added the comment: On the other hand, multi-line annotations are rare enough we could consider falling back to token reconstruction then (the produced string will miss all whitespace in that case). -- ___ Python tracker

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Pablo Galindo Salgado added the comment: After investigating with BTaskaya, seems that we cannot use the literal string from the tokenizer for multi-line annotations. But we could use the tokens (losing the literal whitespace), which honestly is less exciting because we will lose the literal

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Pablo Galindo Salgado added the comment: I have attached a draft PR for this: https://github.com/python/cpython/pull/22587 I think this may be a considerable win as the diff is +66 −1,013 -- nosy: +gvanrossum, lukasz.langa, lys.nikolaou stage: patch review -> ___

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Change by Pablo Galindo Salgado : -- nosy: +BTaskaya ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

Change by Pablo Galindo Salgado : -- keywords: +patch pull_requests: +21579 stage: -> patch review pull_request: https://github.com/python/cpython/pull/22587 ___ Python tracker __

[issue41967] Handle annotations in the parser to avoid the need for roundtrip

New submission from Pablo Galindo Salgado : For Python 3.10 now that annotations are always returned as strings, we can drop all the unparse logic by obtaining the string in the parser directly (after checking that the annotation is a valid expression). -- messages: 378158 nosy: pablo