[Python-Dev] Declarative imports
This is an idea which has been brought up before, sometimes introduced
as "heresy". But an interesting twist has surfaced now which is
typing.
But firstly, let me present the idea. It is very simple, that Python
should have declarative imports, usable anywhere using a simple
syntax, @.
For example, `some_regex = @re.compile(...)`.
What happens then is that before anything else in that module, that
symbol is imported:
from re import compile as _mangled_re_compile
It must be the very first thing (hoisting) because when else would it
happen? It's been suggested before to have a shorthand syntax which
does a dynamic import at the time of using it but this brings me to
the twist:
We want typing to pick up these imports. And this twist has a second
leg which is that we often need to import symbols simply in order to
type some argument type or return type. This leads to a great many
more imports to type.
(Nevermind that if you want to take typing further, abstract
interfaces really make more sense rather than specific
implementations, but the point is the same.)
A situation where this would come in really handy is in scripting such
as how we use Python in Apache Airflow to let users write out simple
workflows. A workflow definition which could be a 5-liner quickly
becomes a 20-liner – consider for example:
default_args = {
"start_date": @datetime.datetime(...)
}
It's a lot more ergonomic from a user perspective (well perhaps for
some users and for some programs).
Thoughts?
Cheers
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/UX6EJHLJNNLMFPWVPF5ANYHQSHDZK7SV/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Declarative imports
On Fri, 8 Apr 2022 at 08:51, Paul Moore wrote:
> Are you exaggerating for effect here or would this *actually* just expand to
>
> from datetime import datetime
> default_args = {
> "start_date": datetime(...)
> }
Yes – and of course that is just a snippet, an actual complete script
will have lots of such imports.
The point is that in a scripting situation (especially one in which
you have many small scripts) the
top-level import requirement for simple imports like `datetime`
becomes rather verbose.
Thanks
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/FVYRPSN6A7KN73MTQIF2UIU5W3HSUPGV/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Declarative imports
On Fri, 8 Apr 2022 at 10:15, Daniel Pope wrote: > But, your proposed syntax is not usable because it is ambiguous. Exactly what > you propose is already used for decorators in a way that the parser would not > be able to distinguish a decorator from an import expression. Consider: > > @ham.spam() # import expression or decorator? > > def eggs(): > ... > > This currently parses as a decorator and for backwards compatibility that > must not change, which means that import expressions would be usable in some > contexts and not others purely based on what follows them. Perhaps `some_regex = re::compile(r"...")` could work. That is, :: to delineate the import. @breakfast.ham::spam def eggs(): pass Cheers ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VXOSHFQDLPI5KXVNXWKFMRZOZHYJLZL5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Declarative imports
On Fri, 8 Apr 2022 at 16:40, Guido van Rossum wrote: > The interesting idea here seems to make "lazy imports" easier to implement by > making them explicit in the code. So far, most lazy import frameworks for > Python have done hacks with `__getattribute__` overrides. IIRC the Cinder > version even modifies the bytecode and/or the interpreter. Disregarding the > specific notation proposed, *if* people would be willing to mark the points > where they expect lazy imports explicitly, that would make implementation > much simpler. Actually, to me the interesting idea is not so much lazy imports – I think they should not be lazy, at least that was my initial thought. I think they should be immediately resolved before anything else in that module: 1. This would settle any discussion about performance impact (there wouldn't be any). 2. This would enable IDEs, typers and other tooling to know the type using existing import logic. 3. Catch errors early! Notation is hard of course. Would users be willing to use it? I think so, at least in Rust, people do use it and I think just in the right places – typically for imports that are either "canonicalized" such as `re.compile` is in Python, or used just once. Cheers ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NKBWGPJRKYQX4Z4WDDD74CSV4XEBGTFD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Declarative imports
On Sun, 10 Apr 2022 at 09:31, Daniel Pope wrote: > I would like to bid again for (import package.module) as an expression. > Instead of doing the import and assigning package to a variable package it > would evaluate to the module object package.module. I like this proposal and I agree with previous posters about hoisting being undesirable. And if lazy importing is slow then that is probably something that can be optimized away; of course in tight loops or performance-sensitive code you wouldn't use this functionality. But perhaps using "from" makes the behavior more clear (and it is shorter): start_date = from datetime.datetime(2012, 1, 6) I think the behavior could be to simply keep importing until we have a non-module value, rather than rely on putting the import name in parentheses. Cheers ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2NLRKDG3TZCQSALAJVHOM3VKZH45ZYWH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Decreasing refcount for locals before popping frame
Consider this example code: def test(): a = A() test() Currently, the locals (i.e. `a`) are cleared only after the function has returned: If we attach a finalizer to `a` immediately after the declaration then the frame stack available via `sys._getframe()` inside the finalizer function does not include the frame used to evaluate the function (i.e. with the code object of the `test` function). The nearest frame is that of the top-level module (where we make the call to the function). This is in practical terms no different than: def test(): return A() test() There's no way to distinguish between the two cases even though in the second example, the object is dropped only after the frame (used to evaluate the function) has been cleared. The effect I am trying to achieve is: def test(): a = A() del a Here's a use-case to motivate this need: In Airflow, we're considering introducing some "magic" to help users write: with DAG(...): # some code here That is, without declaring a top-level variable such as `dag`. However, we can't detect the following situation: def create(): with DAG(...) as dag: # some code here create() The DAG is not returned from the function but nevertheless, we can't distinguish between this code and the correct version: def create(): with DAG(...) as dag: # some code here return dag In this case, calling `create` will then "return" the DAG and of course, without a variable assignment, the finalizer will be called – but now we can detect this. I'm thinking that it ought to be possible to clear out `frame->localsplus` before leaving the function frame. I played around with "ceval.c" and only got segfaults. It's complicated machinery :-) Thoughts? ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/D5HCLMN42SIRRUHWPU566R7YYAVLCAEN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Decreasing refcount for locals before popping frame
Dennis Sweeney wrote: > I don't know if there's anything specifically stopping this, but from what I > understand, the precise moment that a finalizer gets called is unspecified, > so relying on any sort of behavior there is undefined and non-portable. > Implementations like PyPy don't always use reference counting, so their > garbage collection might get called some unspecified amount of time later. It's unspecified of course for the language as such, but in the specific case of CPython (which we're targeting), I think the refcounting logic is here to stay and generally speaking, can be relied on. Of course some version may come along to break expectations and I suppose we might cross that bridge when we get to it. > I'm not familiar with Airflow, but would you be able to decorate the create() > function to check for good return values? We could but for the most part, people don't define DAGs inside functions – it happens, but it is not the most simple usage pattern. It's not so much about the function itself, but about being able to determine if a DAG was dropped at the top-level of the module. If the frame clearing behavior was changed so that locals were reclaimed before popping the frame, I think the line number (i.e. `f_lineno`) would have to be that of the function definition, i.e. `def test():` in the examples above. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FWRP3RPCGXXDQT2IVO7HQBCUQFHGTCRM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Decreasing refcount for locals before popping frame
Pablo Galindo Salgado wrote: > As it has been mentioned there is no guarantee that your variable will even > be finalized (or even destroyed) after the frame finishes. For example, if > your variable goes into a reference cycle for whatever reason it may not be > cleared until a GC run happens (and in some situations it may not even be > cleared at any point). I think there is a reasonable guarantee in CPython that it will happen exactly when you leave the frame, assuming there are no cycles or other references to the object. There's always the future, but I don't see a very near future where this will change fundamentally. Relying too much on CPython's behavior is a bad thing, but I think there are cases where it makes sense and can be a pragmatic choice. Certainly lots of programs have successfully relied on `sys._getframe` over the years. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BVO7RMMZ2LJFEG4GRNNTYZU3Q4P3DHV3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Decreasing refcount for locals before popping frame
On Fri, 29 Apr 2022 at 06:38, Thomas Grainger wrote:
> Can you show a run-able example of the successful and unsuccessful usage of
> `with DAG(): ... `?
from airflow import DAG
# correct:
dag = DAG("my_dag")
# incorrect:
DAG("my_dag")
The with construct really has nothing to do with it, but it is a
common source of confusion:
# incorrect
with DAG("my_dag"):
...
It is less obvious (to some) in this way that the entire DAG will not
be picked up. You will in fact have to write:
# correct
with DAG("my_dag") as dag:
...
This way, you're capturing the DAG in the top-level scope which is the
requirement.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/HREOTTGPB5JMLGYMIQL4VR2DFI6GBG5J/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Decreasing refcount for locals before popping frame
On Fri, 29 Apr 2022 at 06:50, Thomas Grainger wrote: > You can use a `__del__` method that warns on collection - like an unawaited > coroutine > > Also if you're in control of importing the dagfile you can record all created > dags and report any that are missing from the globals of the module Yes and I think this is the best we can do given how frames are being cleared. We can notify the user that a DAG was instantiated and not exposed at the top-level which is almost guaranteed to be a mistake. There's probably no good way currently to do better (for some value of "better"). Thanks ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/E4IU26RL4I72FMACQLNTIPT5DN5XTE3S/ Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-Dev] Proposing PEP 386 for addition
On 12/8/09 6:16 PM, Tarek Ziadé wrote: I believe that the current situation is as close to consensus as we will get on distutils-sig, and in the interests of avoiding months of further discussion which won't take things any further, I propose to allow final comments from python-dev and then look for a final decision. Great work, Tarek. I think you've managed to establish a good body of knowledge on this and the proposal seems sound. That said, I think the terms ``LooseVersion`` and ``StrictVersion`` are less than optimal. Really, what's meant is ``LexicalVersion`` and ``ChronologicalVersion`` (or ``NumberedVersion``). It's not about strictness or looseness. Also, the word "rational" is not familiar to me in the context of versions; is this term known outside of this proposal? I couldn't find any reference to it. \malthe ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposing PEP 386 for addition
2009/12/10 Darren Dale : > Those aren't new proposals, though, they already exist in distutils. I see. Thanks for clarifying –– maybe the PEP should better explain this. \malthe ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Syntax error in python2.6
Lennart Regebro wrote: 2. Using **kw in the argument and looking for noth "with" and "with_", that way, which will be backwards compatible. +1 \malthe ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Transformation with ``parser`` ast
The ``compiler.ast`` module makes parsing Python source-code and AST manipulation relatively painless and it's straight-forward to implement a transformer class. However, I find that the ``compiler.pycodegen`` module imposes a hard limit on the length of functions since it calculates jump points in a recursion. I'm using this module to compile an XML dynamic template into Python code (akin to "Mako") and functions may grow to a rather large size. Now it seems that the ``parser`` module, which parses source code into ``parser.st`` trees does not have the same limitations, however, I could not find a transformer class compatible with its tree structure. What's the recommended way of working with the AST tree from the ``parser`` module? \malthe ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
