Re: Async wrappers in contrib packages

[Carlton Gibson]

No plan ever survives first contact with the enemy :) but, this seems
reasonable to me Jon, yes.

Can I suggest we move further discussion on details to the Forum's async
category? It's nicer, better contained, and we more likely to catch
interested eyes.

Thanks for pursuing this! 🎁

On Thu, 9 Feb 2023 at 23:38, Jon Janzen  wrote:

> Hey Carlton,
>
> Thanks for your thoughtful comments, a few things come to mind:
>
> A. It sounds like we’re in agreement about the utility and severability of
> Phase 1 (just creating an async_to_sync-based wrapper around the auth
> interface). I want to make sure I don’t cause extra work on the bug
> tracker, so does this sound like it’s ready to be filed as a ticket there?
> I can file the ticket and get started on writing tests/docs if so!
>
> B. In reply to your "Is there a best order here, or can we just chip away
> bit-by-bit? 🤔” comment, I think there is a reasonable order: we mirror the
> dependencies between django internals. By that I mean I think we should
> asyncify the necessary parts of sessions or signals before asyncifying the
> parts of auth that depend on them. That reduces the need to
> insert “temporary” sync/async boundaries and reduces churn.
>
> C. Based on your responses and my own thoughts in (2) I think it’s
> reasonable to revise the overall order into this:
>
> 1. In any order: asyncify the auth API (Phase 1), asyncify signals
> (ticket 32172), and asyncify sessions (TODO: discussion/ticket)
> 2. After all 3 parts to (1), asyncify auth backends and internals of the
> auth API (Phase 2)
> 3. Resolve ticket 31920
> 4. Asyncify the auth middleware (Phase 3)
>
> D. I’m admittedly very new to Django and contributing code to it, but
> Phase 1 is certainly within my area of expertise considering the 2 PRs that
> I have contributed 😉:
>
> https://github.com/django/django/pull/16256
> https://github.com/django/django/pull/16252
>
> After that, I’d be interested in helping nudge this along over time. I’d
> like to build my experience beyond just adding async wrappers 😛, so maybe
> I can pick up https://github.com/django/django/pull/13651 and hack away
> at your idea about categorizing during registration. If that all goes well
> I can start a discussion around asyncifying sessions (which, on first
> glance seems like a LARGE project of its own)
>
> Thanks,
>
> Jon
>
>
> On Feb 7, 2023, at 09:25, Carlton Gibson  wrote:
>
> Hi Jon.
>
> Thanks for this.
>
> I think your use-case is reasonable, and that you're basically on the
> right track.
> If you were to add test cases to your PoC, there's certainly a case for
> looking seriously at it.
> It should be reasonable to keep pushing the interfaces down one layer at a
> time. (See comment on Sessions below.)
>
> I'm not sure a priori how Phase 2 plays out in advance. It's likely
> something we have to think about as we go.
> Targeted posts to the Async category of the Forum might help drive
> discussion.
> https://forum.djangoproject.com/c/users/async-channels/23
>
> Let me inline some quick thoughts on your …
>
> ## Open Questions
>
> Based on my stab at the above PoC implementation I came away with several
> questions:
>
> 1. Are there proven strategies for reducing code duplication between sync
> and async versions of functionality in Django or in Python broadly? I’m not
> aware of any guidance here but I’m eager for resources to consider. The
> implementation is verbose (doubles file size in some cases) and fragile
> (what if a bug fix is only applied to the sync version and not the async
> version?) right now.
>
> Yes… you end up writing it twice... 🤔 Maybe in some ideal end state we
> end up with an async core with a thin sync wrapper around that, but we are
> a long way from there: I'm not sure it bears thinking about.
>
> Part of the task on the decorators work is to try not to duplicate
> everything. e.g. a process_request only middleware doesn't necessarily need
> to consume the `get_response()` callable, so often times can make use of
> `markcoroutinefunction()` rather than needed to duplicate the whole
> middleware function. (And so on.) — But those patterns aren't 100% clear
> just yet — we'll find them as we go through in this next phase.
>
>
> 2. The auth app obviously fires several signals, and Django signals
> currently has a sync-only interface. I see there is an issue on the bug
> tracker (https://code.djangoproject.com/ticket/32172) but it is currently
> blocked for performance reasons. Is it fine to just add more sync_to_async
> wrappers here? (I did so in the above PoC implementation). Another idea is
> to just add a async wrapper around the `send` method and defer the rest of
> that ticket until the perf question can be resolved.
>
> So, yes... — my waiting-for-time-to-try-it-idea™ there is to sort signals
> into two groups on registration and call any async ones in a group
> concurrently, with only the single switch into the concurrent context.
> **Something** along those lines should be p

Async QuerySet with Prefetch

['Barry Johnson' via Django developers (Contributions to Django itself)]

For those working on async code, I just opened a question on the Django 
Internals Async forum regarding query sets with prefetches, which aren't 
currently supported in the alpha build of 4.2.  A solution seemed so 
surprisingly easy, I'm wondering what the real experts in the area would 
think.

https://forum.djangoproject.com/t/release-4-2a-async-querysets-with-prefetch-related/18767

baj

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d99adab6-a20a-42be-9ada-d90adf19d20an%40googlegroups.com.

Re: Async wrappers in contrib packages

[Jon Janzen]

Hey Carlton,

Sorry I skipped over the suggestion in one of your earlier messages to post on 
the Forum.

I’ve done so now: 
https://forum.djangoproject.com/t/asyncifying-django-contrib-auth-and-signals-and-maybe-sessions/18770

I think we can consider this thread deprecated in favor of the above Forum 
thread.

Cheers,

Jon

> On Feb 10, 2023, at 03:54, Carlton Gibson  wrote:
> 
> No plan ever survives first contact with the enemy :) but, this seems 
> reasonable to me Jon, yes. 
> 
> Can I suggest we move further discussion on details to the Forum's async 
> category? It's nicer, better contained, and we more likely to catch 
> interested eyes. 
> 
> Thanks for pursuing this! 🎁
> 
> On Thu, 9 Feb 2023 at 23:38, Jon Janzen  > wrote:
>> Hey Carlton, 
>> 
>> Thanks for your thoughtful comments, a few things come to mind:
>> 
>> A. It sounds like we’re in agreement about the utility and severability of 
>> Phase 1 (just creating an async_to_sync-based wrapper around the auth 
>> interface). I want to make sure I don’t cause extra work on the bug tracker, 
>> so does this sound like it’s ready to be filed as a ticket there? I can file 
>> the ticket and get started on writing tests/docs if so!
>> 
>> B. In reply to your "Is there a best order here, or can we just chip away 
>> bit-by-bit? 🤔” comment, I think there is a reasonable order: we mirror the 
>> dependencies between django internals. By that I mean I think we should 
>> asyncify the necessary parts of sessions or signals before asyncifying the 
>> parts of auth that depend on them. That reduces the need to insert 
>> “temporary” sync/async boundaries and reduces churn.
>> 
>> C. Based on your responses and my own thoughts in (2) I think it’s 
>> reasonable to revise the overall order into this:
>> 
>> 1. In any order: asyncify the auth API (Phase 1), asyncify signals (ticket 
>> 32172), and asyncify sessions (TODO: discussion/ticket)
>> 2. After all 3 parts to (1), asyncify auth backends and internals of the 
>> auth API (Phase 2)
>> 3. Resolve ticket 31920
>> 4. Asyncify the auth middleware (Phase 3)
>> 
>> D. I’m admittedly very new to Django and contributing code to it, but Phase 
>> 1 is certainly within my area of expertise considering the 2 PRs that I have 
>> contributed 😉:
>> 
>> https://github.com/django/django/pull/16256
>> https://github.com/django/django/pull/16252
>> 
>> After that, I’d be interested in helping nudge this along over time. I’d 
>> like to build my experience beyond just adding async wrappers 😛, so maybe I 
>> can pick up https://github.com/django/django/pull/13651 and hack away at 
>> your idea about categorizing during registration. If that all goes well I 
>> can start a discussion around asyncifying sessions (which, on first glance 
>> seems like a LARGE project of its own)
>> 
>> Thanks,
>> 
>> Jon
>> 
>> 
>>> On Feb 7, 2023, at 09:25, Carlton Gibson >> > wrote:
>>> 
>>> Hi Jon. 
>>> 
>>> Thanks for this. 
>>> 
>>> I think your use-case is reasonable, and that you're basically on the right 
>>> track. 
>>> If you were to add test cases to your PoC, there's certainly a case for 
>>> looking seriously at it. 
>>> It should be reasonable to keep pushing the interfaces down one layer at a 
>>> time. (See comment on Sessions below.) 
>>> 
>>> I'm not sure a priori how Phase 2 plays out in advance. It's likely 
>>> something we have to think about as we go. 
>>> Targeted posts to the Async category of the Forum might help drive 
>>> discussion. 
>>> https://forum.djangoproject.com/c/users/async-channels/23
>>> 
>>> Let me inline some quick thoughts on your …
>>> 
>>> ## Open Questions
>>> 
>>> Based on my stab at the above PoC implementation I came away with several 
>>> questions:
>>> 
>>> 1. Are there proven strategies for reducing code duplication between sync 
>>> and async versions of functionality in Django or in Python broadly? I’m not 
>>> aware of any guidance here but I’m eager for resources to consider. The 
>>> implementation is verbose (doubles file size in some cases) and fragile 
>>> (what if a bug fix is only applied to the sync version and not the async 
>>> version?) right now.
>>> 
>>> Yes… you end up writing it twice... 🤔 Maybe in some ideal end state we end 
>>> up with an async core with a thin sync wrapper around that, but we are a 
>>> long way from there: I'm not sure it bears thinking about.
>>> 
>>> Part of the task on the decorators work is to try not to duplicate 
>>> everything. e.g. a process_request only middleware doesn't necessarily need 
>>> to consume the `get_response()` callable, so often times can make use of 
>>> `markcoroutinefunction()` rather than needed to duplicate the whole 
>>> middleware function. (And so on.) — But those patterns aren't 100% clear 
>>> just yet — we'll find them as we go through in this next phase. 
>>> 
>>> 2. The auth app obviously fires several signals, and Django signals 
>>> currently has a sync

Re: Async wrappers in contrib packages

[Carlton Gibson]

Great. Thanks Jon. I will follow up there next week.

On Saturday, 11 February 2023, Jon Janzen  wrote:

> Hey Carlton,
>
> Sorry I skipped over the suggestion in one of your earlier messages to
> post on the Forum.
>
> I’ve done so now: https://forum.djangoproject.com/t/
> asyncifying-django-contrib-auth-and-signals-and-maybe-sessions/18770
>
> I think we can consider this thread deprecated in favor of the above Forum
> thread.
>
> Cheers,
>
> Jon
>
> On Feb 10, 2023, at 03:54, Carlton Gibson 
> wrote:
>
> No plan ever survives first contact with the enemy :) but, this seems
> reasonable to me Jon, yes.
>
> Can I suggest we move further discussion on details to the Forum's async
> category? It's nicer, better contained, and we more likely to catch
> interested eyes.
>
> Thanks for pursuing this! 🎁
>
> On Thu, 9 Feb 2023 at 23:38, Jon Janzen  wrote:
>
>> Hey Carlton,
>>
>> Thanks for your thoughtful comments, a few things come to mind:
>>
>> A. It sounds like we’re in agreement about the utility and severability
>> of Phase 1 (just creating an async_to_sync-based wrapper around the auth
>> interface). I want to make sure I don’t cause extra work on the bug
>> tracker, so does this sound like it’s ready to be filed as a ticket there?
>> I can file the ticket and get started on writing tests/docs if so!
>>
>> B. In reply to your "Is there a best order here, or can we just chip
>> away bit-by-bit? 🤔” comment, I think there is a reasonable order: we
>> mirror the dependencies between django internals. By that I mean I think we
>> should asyncify the necessary parts of sessions or signals before
>> asyncifying the parts of auth that depend on them. That reduces the need to
>> insert “temporary” sync/async boundaries and reduces churn.
>>
>> C. Based on your responses and my own thoughts in (2) I think it’s
>> reasonable to revise the overall order into this:
>>
>> 1. In any order: asyncify the auth API (Phase 1), asyncify signals
>> (ticket 32172), and asyncify sessions (TODO: discussion/ticket)
>> 2. After all 3 parts to (1), asyncify auth backends and internals of the
>> auth API (Phase 2)
>> 3. Resolve ticket 31920
>> 4. Asyncify the auth middleware (Phase 3)
>>
>> D. I’m admittedly very new to Django and contributing code to it, but
>> Phase 1 is certainly within my area of expertise considering the 2 PRs that
>> I have contributed 😉:
>>
>> https://github.com/django/django/pull/16256
>> https://github.com/django/django/pull/16252
>>
>> After that, I’d be interested in helping nudge this along over time. I’d
>> like to build my experience beyond just adding async wrappers 😛, so maybe
>> I can pick up https://github.com/django/django/pull/13651 and hack away
>> at your idea about categorizing during registration. If that all goes well
>> I can start a discussion around asyncifying sessions (which, on first
>> glance seems like a LARGE project of its own)
>>
>> Thanks,
>>
>> Jon
>>
>>
>> On Feb 7, 2023, at 09:25, Carlton Gibson 
>> wrote:
>>
>> Hi Jon.
>>
>> Thanks for this.
>>
>> I think your use-case is reasonable, and that you're basically on the
>> right track.
>> If you were to add test cases to your PoC, there's certainly a case for
>> looking seriously at it.
>> It should be reasonable to keep pushing the interfaces down one layer at
>> a time. (See comment on Sessions below.)
>>
>> I'm not sure a priori how Phase 2 plays out in advance. It's likely
>> something we have to think about as we go.
>> Targeted posts to the Async category of the Forum might help drive
>> discussion.
>> https://forum.djangoproject.com/c/users/async-channels/23
>>
>> Let me inline some quick thoughts on your …
>>
>> ## Open Questions
>>
>> Based on my stab at the above PoC implementation I came away with several
>> questions:
>>
>> 1. Are there proven strategies for reducing code duplication between sync
>> and async versions of functionality in Django or in Python broadly? I’m not
>> aware of any guidance here but I’m eager for resources to consider. The
>> implementation is verbose (doubles file size in some cases) and fragile
>> (what if a bug fix is only applied to the sync version and not the async
>> version?) right now.
>>
>> Yes… you end up writing it twice... 🤔 Maybe in some ideal end state we
>> end up with an async core with a thin sync wrapper around that, but we are
>> a long way from there: I'm not sure it bears thinking about.
>>
>> Part of the task on the decorators work is to try not to duplicate
>> everything. e.g. a process_request only middleware doesn't necessarily need
>> to consume the `get_response()` callable, so often times can make use of
>> `markcoroutinefunction()` rather than needed to duplicate the whole
>> middleware function. (And so on.) — But those patterns aren't 100% clear
>> just yet — we'll find them as we go through in this next phase.
>>
>>
>> 2. The auth app obviously fires several signals, and Django signals
>> currently has a sync-only interface. I see there is an issue on the bug
>>