Re: Django Async DEP

[Pkl]

Hello,

I'm a little late to the party, thanks for the big overview on this complex 
matter.

There are lots of thinks I still don't understand though, for example 
regarding "what it unlocks". What asyncio structures would allow to run 
several DB queries concurrently safely and easily, that can't be achieved 
with some threadpool-like mechanism in nowadays' cod ?

Most importantly, I'm not getting why gevent/eventlet-style solutions are 
systematically being dismissed in favor of asyncio. And the more I 
read/talk about it, the less I understand what all that hype around asyncio 
is. This is a whole new language, which forces people to trashbin half of 
the Python ecosystem, and remake it with similar code but filled with 
incompatible async/await statements. 

For sure, having so obstrusive keywords everywhere can be handy when 
writing highly concurrent code without mutexes. But who writes highly 
concurrent code in webservers ? The very principle of webservers is imho 
precisely to never interfere with other requests, never modify 
process-global structures, and at worst delegate some heavy or concurrent 
tasks to some dedicated executor.

Greenlet-style parallelism would bring long-running requests and high 
parallelism to Django without having to touch the bulk of the code code, 
just the I/O parts (DB connections...) ; and thread-local is already 
greenified by Gevent for exemple, according to docs. Why would it bring 
"much higher risk of race conditions and deadlocks without careful 
programming" ? Preemptive threads are much more dangerous than the implicit 
but deterministic context-switching which occurs when greenlets reach 
socket/disk/sleep operations, and yet race conditions seem to be the least 
of the problems of the huge majority of Django programmers.

I'd love to be wrong, but I have the feeling that with a fraction of the 
work required by django-asyncio, it would be possible to greenify the whole 
of Django, fix lots of current limitations and corner-cases of gevent (lack 
of builtin executor to offload tasks to real threads, lack of support in 
python package X/Y/Z...), and without having to recode any of the "middle" 
parts of existing modules. How can we say "third-party support for this 
style of concurrency is much weaker", whereas through monkey-patching, 
about ANY python module (except those using blocking C extensions) can be 
used in a Gevent project ?

Granted, I have little experience with Geven and Asyncio, but all 
experience feedbacks I've read so far mainly insist on minor limitations of 
greenlets, and on the fact that "people are mainly going with asyncio" (a 
self-fulfilling prophecy?). Considered the dramatic difference in workload 
between the two, I'd really love to understand what killer-features justify 
to go for the "recode everything" solution (or what greenlet limitations 
would be show-stoppers on the long term). 

Thanks in advance everyone for your feedback on this issue.

regards,
Pascal Chambon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/605d78af-f24a-46eb-9cac-4ad75547b604%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Andrew Godwin]

On Thu, Jun 6, 2019 at 3:20 PM Pkl  wrote:

> Hello,
>
> I'm a little late to the party, thanks for the big overview on this
> complex matter.
>
> There are lots of thinks I still don't understand though, for example
> regarding "what it unlocks". What asyncio structures would allow to run
> several DB queries concurrently safely and easily, that can't be achieved
> with some threadpool-like mechanism in nowadays' cod ?
>

Python threads perform badly, as they force context-switches even when no
work is pending; you can't sensibly run more than 20/30 threads before the
overheads seriously start eating into your performance.


>
> Most importantly, I'm not getting why gevent/eventlet-style solutions are
> systematically being dismissed in favor of asyncio. And the more I
> read/talk about it, the less I understand what all that hype around asyncio
> is. This is a whole new language, which forces people to trashbin half of
> the Python ecosystem, and remake it with similar code but filled with
> incompatible async/await statements.
>
> ...
>
> Granted, I have little experience with Geven and Asyncio, but all
> experience feedbacks I've read so far mainly insist on minor limitations of
> greenlets, and on the fact that "people are mainly going with asyncio" (a
> self-fulfilling prophecy?). Considered the dramatic difference in workload
> between the two, I'd really love to understand what killer-features justify
> to go for the "recode everything" solution (or what greenlet limitations
> would be show-stoppers on the long term).
>

I can't give you a full answer about why they are being dismissed, but the
move is clearly systematic. My own personal experience is that writing with
gevent in particular ends up being very difficult, as it is not very
explicit about what is async, what causes a context switch, and so you end
up wrapping a lot of your code in locks to even try and get code that isn't
susceptible to nasty race conditions.

I suspect asyncio fits much more with the Zen of Python - you know exactly
when a context switch might occur (when you see an await), and modules
explicitly add support for it rather than having it monkey-patched in. In
addition, most people I know in the Python community who are actively
working on async libraries are doing so against asyncio.

You could make the same argument with trio - it's arguably a better,
cleaner async implementation. But, again, it's not where Python is at. This
change is already big enough that it's very important we keep consistency
with Python core to lower the workload. If Python core turned around and
blessed greenlets and gevent as the chosen async solution, I'd change my
mind, but I haven't seen any evidence of that over many years.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1uqgUUwL_%3DXS%2Be6tzieycn%3DjDZK6wFCKzHKxRtfukacfQw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.