Async Caching

[Andrew Wang]

Hey guys, I'd like to contribute to the effort to make Django more async 
capable. I've started to write an aioredis based cache backend based on 
django-redis, but I noticed the BaseCache in Django is still all 
synchronous basically.

I was wondering which backends I should make async capable and how would I 
go about it?

I was thinking instead of creating a new class, we could just add the async 
methods (e.g. add, get) to BaseClass. And for the FileBased, Dummy, and 
LocMem backends, the plan would be the same? Problem with those are 
Python's local file IO is synchronous...

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/68307bf9-e6f4-4dc3-9e0f-d6e660075a85o%40googlegroups.com.

Re: Async Caching

[Andrew Wang]

Hey Adam and Andrew,

I can definitely make the naming scheme something like get_async() rather 
than just get().

> This section specifically says that the default implementations will back 
onto the sync methods by default, so built-in cache backends won't need to 
all be converted (at once?).

I'm slightly confused by what you mean (and what the dep means). Are you 
saying in case someone is using a built-in cache backend like LocMem for 
local development and then an async third-party cache backend for 
production, then the code is like:

class BaseCache:
def get(...):
NotImplementedError
async def get_async(...):
NotImplementedError

class LocMemCache:
def get(...):
...
async def get_async(...):
return await asgiref.sync_to_async(self.get)(*args, **kwargs)

>  It could be informative to have a working backend using this interface, 
and if you encounter any edge cases whilst creating it.

You can view my progress (once midterms are over :P) over here. 
<https://github.com/Andrew-Chen-Wang/django-async-redis> I'm planning on 
making the structure very similar to django-redis to make any migration 
process to an async Django easier. When I compared redis-py to aioredis, 
there weren't that many differences besides file structure (and passing the 
event loop around), so hopefully nothing weird pops up. But I will 
definitely post in this thread or in a ticket (if I do make a PR for 
BaseCache) if something odd does pop up.

> The main work to do here is to work how quite how possible it is to offer 
all of them and if everything can be made to work regardless of what mode 
(sync or async) it's in

In terms of the package or the other built-in backends or just everything 
in general like the ORM? If in terms of the package, most if not all 
methods would have to be awaited... I could also just be confusing all 
these words, so sorry about uhh me I guess!

Also just saw https://forum.djangoproject.com/. Any preference for where 
future talks should be held: Google or that forum?

Thanks for the suggestions! Have fun with the ORM!
Andrew
On Saturday, September 26, 2020 at 8:38:07 PM UTC-4 Andrew Godwin wrote:

> Agreed - there's no work on caching inside Django yet, since the ORM is my 
> next focus, but I would definitely suggest writing a new pluggable 
> third-party backend that somehow provides async versions of the methods.
>
> The main work to do here is to work how quite how possible it is to offer 
> all of them and if everything can be made to work regardless of what mode 
> (sync or async) it's in; hopefully there's a lot less long-lived-connection 
> issues than in the ORM, say.
>
> Andrew
>
> On Sat, Sep 26, 2020, at 3:56 PM, Adam Johnson wrote:
>
> Hi Andrew
>
> I don't believe any work has started on async caching in core. However 
> there is a plan in DEP 9, the document that outlines how asynchronous 
> caching will work, using suffixed methods like get_async() etc. See 
> https://github.com/django/deps/blob/master/accepted/0009-async.rst#caching 
> . This section specifically says that the default implementations will back 
> onto the sync methods by default, so built-in cache backends won't need to 
> all be converted (at once?).
>
> I think a good start would be making your aioredis-based backend in a 
> third party package, using the future naming scheme `*_async`. It could be 
> informative to have a working backend using this interface, and if you 
> encounter any edge cases whilst creating it.
>
> Thanks,
>
> Adam
>
> On Sat, 26 Sep 2020 at 05:55, Andrew Wang  wrote:
>
> Hey guys, I'd like to contribute to the effort to make Django more async 
> capable. I've started to write an aioredis based cache backend based on 
> django-redis, but I noticed the BaseCache in Django is still all 
> synchronous basically.
>
> I was wondering which backends I should make async capable and how would I 
> go about it?
>
> I was thinking instead of creating a new class, we could just add the 
> async methods (e.g. add, get) to BaseClass. And for the FileBased, Dummy, 
> and LocMem backends, the plan would be the same? Problem with those are 
> Python's local file IO is synchronous...
>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-develop...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/68307bf9-e6f4-4dc3-9e0f-d6e660075a85o%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/django-developers/68307bf9-e6f4-4dc3-9e0f-d6e660075a85

Re: Async Caching

[Andrew Wang]

Cool, thanks for the clarification. One thing I noticed while programming 
the test cases (so WIP is 
here: https://github.com/Andrew-Chen-Wang/django-async-redis) is that it's 
not really fun having to type out get_async since autocomplete. If a dev 
knows that they'll be using an await, then I think it's best if everything 
is prefixed with async_* rather than suffixed. Thoughts?

As a side note, the only other change that would need to happen is in 
django.core.cache close_caches AFAIK.

Andrew 

On Sunday, September 27, 2020 at 3:30:56 AM UTC-4 Adam Johnson wrote:

> I'm slightly confused by what you mean (and what the dep means). 
>
>
> I understand the DEP as saying that we'll implement the async methods in 
> the BaseCache class as backing onto the sync versions:
>
> class BaseCache
> async def get_async(...):
> return await asgiref.sync_to_async(self.get)(*args, **kwargs)
>
> This means that existing cache backends don't all need to be converted in 
> one go.
>
> Also just saw https://forum.djangoproject.com/. Any preference for where 
>> future talks should be held: Google or that forum?
>>
>
> Essentially both are equivalent. I think there remain more readers on the 
> mailing list though, which can help if you're discussing bigger picture 
> things.
>
> On Sun, 27 Sep 2020 at 04:30, Andrew Wang  wrote:
>
>> Hey Adam and Andrew,
>>
>> I can definitely make the naming scheme something like get_async() rather 
>> than just get().
>>
>> > This section specifically says that the default implementations will 
>> back onto the sync methods by default, so built-in cache backends won't 
>> need to all be converted (at once?).
>>
>> I'm slightly confused by what you mean (and what the dep means). Are you 
>> saying in case someone is using a built-in cache backend like LocMem for 
>> local development and then an async third-party cache backend for 
>> production, then the code is like:
>>
>> class BaseCache:
>> def get(...):
>> NotImplementedError
>> async def get_async(...):
>> NotImplementedError
>>
>> class LocMemCache:
>> def get(...):
>> ...
>> async def get_async(...):
>> return await asgiref.sync_to_async(self.get)(*args, **kwargs)
>>
>> >  It could be informative to have a working backend using this 
>> interface, and if you encounter any edge cases whilst creating it.
>>
>> You can view my progress (once midterms are over :P) over here. 
>> <https://github.com/Andrew-Chen-Wang/django-async-redis> I'm planning on 
>> making the structure very similar to django-redis to make any migration 
>> process to an async Django easier. When I compared redis-py to aioredis, 
>> there weren't that many differences besides file structure (and passing the 
>> event loop around), so hopefully nothing weird pops up. But I will 
>> definitely post in this thread or in a ticket (if I do make a PR for 
>> BaseCache) if something odd does pop up.
>>
>> > The main work to do here is to work how quite how possible it is to 
>> offer all of them and if everything can be made to work regardless of what 
>> mode (sync or async) it's in
>>
>> In terms of the package or the other built-in backends or just everything 
>> in general like the ORM? If in terms of the package, most if not all 
>> methods would have to be awaited... I could also just be confusing all 
>> these words, so sorry about uhh me I guess!
>>
>> Also just saw https://forum.djangoproject.com/. Any preference for where 
>> future talks should be held: Google or that forum?
>>
>> Thanks for the suggestions! Have fun with the ORM!
>> Andrew
>> On Saturday, September 26, 2020 at 8:38:07 PM UTC-4 Andrew Godwin wrote:
>>
>>> Agreed - there's no work on caching inside Django yet, since the ORM is 
>>> my next focus, but I would definitely suggest writing a new pluggable 
>>> third-party backend that somehow provides async versions of the methods.
>>>
>>> The main work to do here is to work how quite how possible it is to 
>>> offer all of them and if everything can be made to work regardless of what 
>>> mode (sync or async) it's in; hopefully there's a lot less 
>>> long-lived-connection issues than in the ORM, say.
>>>
>>> Andrew
>>>
>>> On Sat, Sep 26, 2020, at 3:56 PM, Adam Johnson wrote:
>>>
>>> Hi Andrew
>>>
>>> I don't believe any work has started on async caching in

Async Django Cache - Redis Complete

[Andrew Wang]

Hi all,

1.5 weeks ago, I posted that I was interested in making Django's cache 
async. Just here to report that the package for django-async-redis is on 
pypi and the repo is 
here: https://github.com/Andrew-Chen-Wang/django-async-redis. Please test 
it out. I have only tested it on a local Django project with uvicorn as a 
test server.

Regarding edge cases from our previous conversation, there were none that I 
could see. I've stuck with the naming scheme get_async rather than 
async_get. I think I'll leave it like that for the foreseeable future (I 
didn't feel a difference in typing speed, either).

My only concern is cache.close() which can be fixed in a PR for adding 
cache.close_async in BaseCache: 
https://github.com/django/django/blob/999cddd58d30469f3ee85278985313fdf528323d/django/core/cache/__init__.py#L116-L121

In other words, I don't think Django's caching framework needs to change... 
for most of it besides those aforementioned lines. That's all I've got for 
now (midterms are tomorrow). Let me know if there are any critical failures.

Cheers,
Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9eabc712-a33a-4d93-a326-3a683d11c62bn%40googlegroups.com.

Re: Generate JWTs with Django

[Andrew Wang]

Hi all, I know this is an old thread, but I have a solution for SPAs using 
httpOnly session cookies. Here is a demo with React and Django WITHOUT JWTs 
with httpOnly cookies for session and csrf: 
https://acwpython.pythonanywhere.com/authenticated/. The tutorial and open 
source repository is found here: 
https://github.com/Andrew-Chen-Wang/SPA-with-httponly-sessions .

The original purpose of this thread was for SPA development, not really for 
JWTs. I'm a maintainer at SimpleJWT, a repository that almost all tutorials 
use to show React/SPA/JS Frameworks and Django integration. I also agree 
with the security concerns for JWT usage on the browser. So I made this 
demo and tutorial overnight to make sure everyone stops using JWTs instead 
of sessions.

Thanks for taking a look. Please spread the word to get people to stop 
using JWTs instead of sessions.
Cheers 

On Monday, May 11, 2020 at 7:19:33 PM UTC-4 dans...@gmail.com wrote:

> The place where JWT begins to get useful and important is when federated 
> login capabilities end-up in your app. That sort of thing seems more 
> the domain of python-social-auth packages like social-auth-core and 
> social-auth-app-django.  Generating an authentication cookie doesn't 
> require JWT - Django already does that.
>
> On Mon, May 11, 2020 at 9:37 AM Derek Adair  wrote:
>
>> Maybe we can update the docs to show how you you would might use some of 
>>> the signing primitives instead of JWTs, but this also sounds a bit 
>>> dangerous 🤷‍♂️ 
>>>
>>
>> As someone hoodwinked into believing JWT was the way... I'd absolutely 
>> LOVE a clear and concise write up on how I might get my single page js apps 
>> to communicate securely with projects like Django Rest. 
>>
>> Thanks for closing the door on JWT for me James.
>>
>> -- 
>>
> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/adc7a8eb-6100-4639-af98-4bca9afaad0b%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2ba4d87b-594a-4736-8889-47941facdaa3n%40googlegroups.com.

Re: Generate JWTs with Django

[Andrew Wang]

Hey Carlton,

Thanks! There's actually already a package (django-webpack-loader) designed
to handle exactly what I did, but being a maintainer of SimpleJWT made me
realize most Django devs weren't using it. I basically almost completely
blame all the tutorials that keep using SimpleJWT to do the task. This
article: https://www.valentinog.com/blog/webpack-django/ talks about two
reasons (besides people bandwagoning off SimpleJWT tutorials and making
their own): webpack maintainability and in general the JS bundles growing
too big and thus slowing dev time. I think *even if* there is wide adoption
for webpack, because of the JS bundles continuously growing (being a huge
turn-off once you have a semi-production-grade SPA repository), I proposed
a moderate idea:

To fix this, I'm going to develop a middleware as described in issue #3
here:
https://github.com/Andrew-Chen-Wang/SPA-with-httponly-sessions/issues/3
It'll use JWT when you're delivering on Node and SessionMiddleware when
delivering your bundles via staticfiles. That way, a webpack loader is not
needed and development time is extremely fast. Deploy your staticfiles on
GitHub pages with the provided action; use React hot-reloading during
development so you don't need a webpack loader, split chunking, etc..

But thanks for taking a look! I'll update this thread once more when I
develop a package or just create that middleware.

Cheers,
Andrew

On Tue, Jan 19, 2021 at 9:55 AM Carlton Gibson 
wrote:

> Hi Andrew,
>
> Thanks for updating — that's really interesting to look at. I think a few
> more examples like this around the community would be handy to break the
> "Must use JWT" presumption that I think there is.
>
> Kind Regards,
>
> Carlton
>
>
> On Saturday, 16 January 2021 at 16:46:35 UTC+1 Andrew Wang wrote:
>
>> Hi all, I know this is an old thread, but I have a solution for SPAs
>> using httpOnly session cookies. Here is a demo with React and Django
>> WITHOUT JWTs with httpOnly cookies for session and csrf:
>> https://acwpython.pythonanywhere.com/authenticated/. The tutorial and
>> open source repository is found here:
>> https://github.com/Andrew-Chen-Wang/SPA-with-httponly-sessions .
>>
>> The original purpose of this thread was for SPA development, not really
>> for JWTs. I'm a maintainer at SimpleJWT, a repository that almost all
>> tutorials use to show React/SPA/JS Frameworks and Django integration. I
>> also agree with the security concerns for JWT usage on the browser. So I
>> made this demo and tutorial overnight to make sure everyone stops using
>> JWTs instead of sessions.
>>
>> Thanks for taking a look. Please spread the word to get people to stop
>> using JWTs instead of sessions.
>> Cheers
>>
>> On Monday, May 11, 2020 at 7:19:33 PM UTC-4 dans...@gmail.com wrote:
>>
>>> The place where JWT begins to get useful and important is when federated
>>> login capabilities end-up in your app. That sort of thing seems more
>>> the domain of python-social-auth packages like social-auth-core and
>>> social-auth-app-django.  Generating an authentication cookie doesn't
>>> require JWT - Django already does that.
>>>
>>> On Mon, May 11, 2020 at 9:37 AM Derek Adair  wrote:
>>>
>>>> Maybe we can update the docs to show how you you would might use some
>>>>> of the signing primitives instead of JWTs, but this also sounds a bit
>>>>> dangerous 🤷‍♂️
>>>>>
>>>>
>>>> As someone hoodwinked into believing JWT was the way... I'd absolutely
>>>> LOVE a clear and concise write up on how I might get my single page js apps
>>>> to communicate securely with projects like Django Rest.
>>>>
>>>> Thanks for closing the door on JWT for me James.
>>>>
>>>> --
>>>>
>>> You received this message because you are subscribed to the Google
>>>> Groups "Django developers (Contributions to Django itself)" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to django-develop...@googlegroups.com.
>>>>
>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/django-developers/adc7a8eb-6100-4639-af98-4bca9afaad0b%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/django-developers/adc7a8eb-6100-4639-af98-4bca9afaad0b%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developer

Current Development of async ORM?

[Andrew Wang]

Happy new year y'all!

I'm wondering if anyone else has a fork that started to chip away at an 
async ORM and advice for how it's going so far.

For instance, where did you start? For me, I started with implementing an 
aiosqlite backend, moving on to RawQuery and Query, then SQLCompiler. How 
often are you using asgiref.sync rather than implementing an async 
compatible class like Signal or atomic? If you implemented a db backend, 
did you implement the DatabaseCreation class? Importantly, are you having 
to duplicate a lot of class method code due to the number of awaits used?

Cheers,
Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/860f65e5-4f91-4219-9ab9-87fc7e1cf435n%40googlegroups.com.

DATABASES setting for async usage

[Andrew Wang]

Hi, I'm intrigued in helping develop the async ORM engines (link to random 
PR  with aiosqlite and base 
async engine). The biggest road block is having test cases for async 
engines only; for instance, introspection/test_async.py can't run because 
the test suite needs to first migrate using a sync engine as the default 
database connection then switch the default alias to using an async engine; 
additionally, for most of the test suite, it uses the sync engine. The 
current methods I'm thinking of:

1. Each async engine would get its own CI worker. I think it's a waste of 
resources and because it wouldn't help Django users who also need to write 
test cases with both an async and sync (for migrations) engine.
2. Async and sync engines for the same database are run at the same time. 
Every async engine requires a parallel sync engine, an alias to be 
specified in the engine's DATABASES options. This way, while testing, 
migrations can be performed with a designated sync engine. This is great 
for the end user who may be async-centric; this doesn't really resolve the 
current problem. Most of the Django test suite is designed for synchronous 
db engines, so the default alias database engine will switch around a bunch 
of times.
3. Implement a test decorator that switches the default alias connection.

Lemme know if that's confusing.

Thanks,
Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4ca69f37-cc9e-40c3-a73b-cc9a125ebaecn%40googlegroups.com.

Re: DATABASES setting for async usage

[Andrew Wang]

I'm thinking that for our purposes:

   1. Because several Django projects need a migration from sync to async 
   which requires time and resources, sync engines can specify a new key 
   called ASYNC_DATABASE_ALIAS whose value is a string that is a database 
   alias in the DATABASES dict. The purpose of this is two-fold: A) ease 
   Django project migrations to async and B) allow for testing on Django 
   internal test suite and third-party packages. This also helps with 
   backwards compatibility for the templating system such that when we 
   call Model.objects.acreate, because of that a prefix, we can implicitly use 
   the async engine.
   2. Projects that start off as async or have their default engine as 
   async need an option called SYNC_DATABASE_ALIAS that is exactly like 
   the ASYNC_DATABASE_ALIAS option. The purpose for this is to fill in the gap 
   of DEP 9 by having a migration and schema engine changer since async 
   engines aren't supposed to. In the database wrapper, we would grab the sync 
   database's database wrapper's Schema class.

It'll look like this:
DATABASES = { 'default': { 'ENGINE': 'django.db.backends.aiosqlite', 
'SYNC_DATABASE_ALIAS': 'sync' }, 'sync': { 'ENGINE': 
'django.db.backends.sqlite3', }, } 

or
DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 
'ASYNC_DATABASE_ALIAS': 'async' }, 'async': { 'ENGINE': 
'django.db.backends.aiosqlite', }, } 

I'm not sure if either of these options are possible due to sqlite being 
tested in memory though.

On Monday, January 31, 2022 at 6:48:13 PM UTC-5 Adam Johnson wrote:

> Hi Andrew,
>
> I'm afraid I don't know much about async, but I can point you at some 
> recent changes. Andrew Godwin created a PR with the draft of the async ORM 
> API. Carlton recently asked for tests: 
> https://twitter.com/carltongibson/status/1486281689265545221 . Perhaps 
> check out those PR's and see if you can contribute further?
>
> Adam
>
> On Sat, Jan 29, 2022 at 1:42 AM Andrew Wang  wrote:
>
>> Hi, I'm intrigued in helping develop the async ORM engines (link to 
>> random PR <https://github.com/django/django/pull/15357> with aiosqlite 
>> and base async engine). The biggest road block is having test cases for 
>> async engines only; for instance, introspection/test_async.py can't run 
>> because the test suite needs to first migrate using a sync engine as the 
>> default database connection then switch the default alias to using an async 
>> engine; additionally, for most of the test suite, it uses the sync engine. 
>> The current methods I'm thinking of:
>>
>> 1. Each async engine would get its own CI worker. I think it's a waste of 
>> resources and because it wouldn't help Django users who also need to write 
>> test cases with both an async and sync (for migrations) engine.
>> 2. Async and sync engines for the same database are run at the same time. 
>> Every async engine requires a parallel sync engine, an alias to be 
>> specified in the engine's DATABASES options. This way, while testing, 
>> migrations can be performed with a designated sync engine. This is great 
>> for the end user who may be async-centric; this doesn't really resolve the 
>> current problem. Most of the Django test suite is designed for synchronous 
>> db engines, so the default alias database engine will switch around a bunch 
>> of times.
>> 3. Implement a test decorator that switches the default alias connection.
>>
>> Lemme know if that's confusing.
>>
>> Thanks,
>> Andrew
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/4ca69f37-cc9e-40c3-a73b-cc9a125ebaecn%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/django-developers/4ca69f37-cc9e-40c3-a73b-cc9a125ebaecn%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/70cf36e9-2c67-4bb8-ace9-a06d2ef6ee38n%40googlegroups.com.