Re: Performance does not scale when increasing file upload sizes

2022-05-17 Thread 'Noxx' via Django developers (Contributions to Django itself)
Hi Carlton, thanks for your response. I've opened up a PR: https://github.com/django/django/pull/15704 I've decided to moved the "ASGIStream" wrapper code into the Django code and get rid of the dependency as it is only a small part of it. My version of the code was nearly the same as the one fr

Re: Performance does not scale when increasing file upload sizes

2022-05-17 Thread Carlton Gibson
Hi Noxx. This is interesting, thanks for the write-up. As a bit of history, initially ASGIRequests would read the whole body into memory, which was fine for an initial version, but led quickly enough to resource concerns. Andrew followed up with the SpooledTemporaryFile approach, as a nice next-s

Re: Performance does not scale when increasing file upload sizes

2022-05-17 Thread 'Noxx' via Django developers (Contributions to Django itself)
Hi all, I've played around a bit and would like to give you a heads up: WSGI and ASGI have only differences in how they parse incoming requests. As WSGI is synchronous the body is already provided as a ByteIO stream which is set directly on the HTTPRequest (wrapped as a LazyStream). On ASGI the

Re: Performance does not scale when increasing file upload sizes

2022-05-11 Thread 'Adam Johnson' via Django developers (Contributions to Django itself)
Ferran - No, this is the right place. Noxx is suggesting a change to Django. Noxx - it seems reasonable to improve Django here. You mentioned ASGI - did you check what the WSGI paths do? It may be that the newer ASGI code is simply less optimal. Would it be possible to reduce the workflow to a si

Re: Performance does not scale when increasing file upload sizes

2022-05-11 Thread Ferran Jovell
This is still the wrong place. Try django-users mailing list. El dc., 11 de maig 2022, 17:53, 'Noxx' via Django developers (Contributions to Django itself) va escriure: > Story > > Our application supports many file uploads. > In normal circumstances, those files are only a few megabytes to of a

Performance does not scale when increasing file upload sizes

2022-05-11 Thread 'Noxx' via Django developers (Contributions to Django itself)
Story Our application supports many file uploads. In normal circumstances, those files are only a few megabytes to of around 100MB. However, some uploads can be several GB in size (10-15GB and up). I know, that file uploads with such a size can be handled differently, e.g. using a seperate endpo