#33497: Database persistent connections do not work with ASGI in 4.0
-------------------------------------+-------------------------------------
Reporter: Stenkar | Owner: (none)
Type: New feature | Status: new
Component: Database layer | Version: 4.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: ASGI, Database, | Triage Stage: Accepted
async |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Florian Apolloner):
I think there is a bit of a missunderstanding about this ticket and what
the connection pool can do and how.
First off, let me start with the initial description of the ticket, namely
that persistent connections do not work with ASGI. Yes this is correct, is
by design and will stay that way. The reason for this is that persistent
connections as implemented in Django are basically persistent per thread
identifier. With ASGI you basically get a new thread identifier per
request and as such a new connection. The old connections are not getting
cleaned up at all quickly resulting in an exhaustion of the connection
limits.
With the connection pool you are actually able to reuse those connections
because they don't have a thread affinity anymore. If not this is yet
another bug that should be fixed, but I think this should work and so far
noone has shown any logs or an example project to the contrary and I will
also not waste my time if there is no effort in helping by providing logs
etc…
What will not work though is that the connection pool will make your
server magically be massively concurrently. The way connection handling in
Django currently works is that the connection is checked out at the first
SQL operation in a request and returned at the **end** of the request. So
the max size of the pool basically sets a hard limit on how many request
you can handle in parallel holding a connection. This is basically
equivalent to using no connection pool inside Django and pgbouncer in
`pool_mode = session`. If you have only have a handful of views that are
accessing the database and also doing a lot of work (ie the response time
is long) then you probably can manually close the connection mid view
which will return the connection to the pool allowing for more concurrent
requests.
A small example of this might look like this ([syntax] errors to be
expected):
{{{#!python
from django.db import connection
async def my_long_running_view(request):
my_tasks = list(Task.objects.all()) # Force the query to execute now
connection.close() # Close the connection and as such release back to
the pool
for task in tasks: # Note: I know that this could be done in parallel
as well, but that is besides the point here
await some_long_running_operation(task)
# Queries here will checkout a new connection from the pool
}}}
So yes, I do think that the connection pool in Django does solve the
problem of persistent connections not working in ASGI, what it doesn't do
is providing a mode that behaves like pgbouncers `pool_mode =
transaction`. This would certainly be a worthwile improvement but given
the limited feedback on pooling so far I never bothered implementing it.
--
Ticket URL: <https://code.djangoproject.com/ticket/33497#comment:51>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/django-updates/010701958e45ec68-0fdcaf4d-02a0-4515-aefb-36f1dc457d90-000000%40eu-central-1.amazonses.com.
