CSRF middleware calculates new tokens that are never used
It was requested of me that I post this here to get some more eyes on it:
Django ticket: https://code.djangoproject.com/ticket/24696
Pull request: https://github.com/django/django/pull/4550
I noticed that the csrf middleware will always calculate a new csrf token
for any request that does not include a csrf cookie. This is rather
wasteful because in many (most?) cases this token is never sent back to the
client. I have found that avoiding this wasteful computation can speed up
requests by about 40%.
For example take the following simple view:
def json(request):
response = {
"message": "Hello, World!"
}
return HttpResponse(uj_dumps(response), content_type="application/json")
Using siege or openload to request above view with csrf middleware enabled:
Stock django 1.9: 876 transactions/sec
With above pull request applied: 1410 transactions/sec
The above pull request (https://github.com/django/django/pull/4550) causes
auth_tests.test_views.LoginTest. test_login_csrf_rotate test to fail. This
is because the test is manually manipulating the
request.META["CSRF_COOKIE_USED"] variable.
Do we need to support the case where apps or other middleware change that
variable directly?
Should I change the test, or should I change the patch to support the case
of CSRF_COOKIE_USED getting manipulated outside of the csrf middleware?
Thanks,
Jay Cox
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/5b4f572f-23c5-4bea-b31e-fe65dd62305b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: CSRF middleware calculates new tokens that are never used
Hi Jay,
On 04/24/2015 12:29 PM, jay...@linear3d.com wrote:
> It was requested of me that I post this here to get some more eyes on it:
>
> Django ticket: https://code.djangoproject.com/ticket/24696
> Pull request: https://github.com/django/django/pull/4550
>
> I noticed that the csrf middleware will always calculate a new csrf
> token for any request that does not include a csrf cookie. This is
> rather wasteful because in many (most?) cases this token is never sent
> back to the client. I have found that avoiding this wasteful
> computation can speed up requests by about 40%.
>
> For example take the following simple view:
>
> def json(request):
>
> response = {
>
> "message": "Hello, World!"
>
> }
>
> return HttpResponse(uj_dumps(response), content_type="application/json")
>
>
>
> Using siege or openload to request above view with csrf middleware enabled:
> Stock django 1.9: 876 transactions/sec
> With above pull request applied: 1410 transactions/sec
Wow. I'm surprised that generating a CSRF token is that expensive, but
it doesn't really matter; if it doesn't need to be done for a given
request, there's no reason to do it.
> The above pull request
> (https://github.com/django/django/pull/4550) causes
> auth_tests.test_views.LoginTest. test_login_csrf_rotate test to fail.
> This is because the test is manually manipulating the
> request.META["CSRF_COOKIE_USED"] variable.
>
> Do we need to support the case where apps or other middleware change
> that variable directly?
META['CSRF_COOKIE_USED'] is never mentioned in the documentation, so I
don't think there is any need to maintain backwards-compatibility with it.
Carl
> Should I change the test, or should I change the patch to support the
> case of CSRF_COOKIE_USED getting manipulated outside of the csrf middleware?
>
>
> Thanks,
>
> Jay Cox
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-developers+unsubscr...@googlegroups.com
> .
> To post to this group, send email to django-developers@googlegroups.com
> .
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/5b4f572f-23c5-4bea-b31e-fe65dd62305b%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/553A96E0.7010505%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.
signature.asc
Description: OpenPGP digital signature
System check is using the default database connection for all the models
Hello, Wasn't sure if this is a bug or not, and I couldn't find any related tickets on https://code.djangoproject.com We have the following database set-up: - the default database is MySQL - two additional databases are PostGIS The system check framework (either `manage.py check` or `manage.py runserver`) is using the default database (mysql) to validate `django.contrib.gis.db.models.Model`-based models, and the commands are blowing up with the following error: AttributeError: 'DatabaseOperations' object has no attribute 'geo_db_type'. You can find the detailed stack trace here: https://gist.github.com/delinhabit/587d6f26afb4bbc559cb We set-up database routers for the PostGIS models to use the proper connection, but the check framework seems to ignore them. Are we doing something wrong? Is this a known pitfall of the system check framework around multiple databases that we are unaware of? Or is this a bug? Thanks, Ion -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4050cb91-25ce-4447-a6fd-e3a1335faf61%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: CSRF middleware calculates new tokens that are never used
On Friday, April 24, 2015 at 12:18:08 PM UTC-7, Carl Meyer wrote: > > > Wow. I'm surprised that generating a CSRF token is that expensive, but > it doesn't really matter; if it doesn't need to be done for a given > request, there's no reason to do it. > > Most of that time is spent in posix.urandom > > The above pull request > > (https://github.com/django/django/pull/4550) causes > > auth_tests.test_views.LoginTest. test_login_csrf_rotate test to fail. > > This is because the test is manually manipulating the > > request.META["CSRF_COOKIE_USED"] variable. > > > > Do we need to support the case where apps or other middleware change > > that variable directly? > > META['CSRF_COOKIE_USED'] is never mentioned in the documentation, so I > don't think there is any need to maintain backwards-compatibility with it. > > Thanks. I have updated the pull request to fix that test. Jay Cox -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6e2f633a-3b49-4925-88d4-664c9ca4bd5f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Potentially inconsistent behavior between test client and normal WSGI requests
Hi all, As I was fixing an issue in our API related to url encodings, I noticed a problem should have been caught by a test that was somehow passing. (Remember, make sure your test can fail!) If you had some path /some/path/Spam%20Ham, and a URL pattern to capture /some/path/(?P.+)$, it's not unreasonable to think to think your named capture would pick up "Spam Ham" (with an actual space) and send that to your view. And indeed it does exactly that, when you make that request through the Django test client. This is because there's an explicit call to unquote in django.test.client.RequestFactory before proceeding to build a WSGIRequest. (The behavior of the development server is similar to the test client, although I haven't investigated for what exact reason.) But when an actual WSGI server makes the same call, WSGIHandler doesn't make that same call to unquote; it passes the exact URL through to WSGIRequest. This leads to a scenario where, in the above example, views will see "Spam Ham" as the value of foo in unit tests, but will see "Spam%20Ham" when run in production. This strikes me as bug worthy. I don't have *particularly *strong feelings on which is the correct behavior, but both behaviors should be the same. I'd be willing to take on the patch once I get some input from others. Thoughts? -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/aecc395b-eb4f-4046-96bf-46ea74b4e68d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Potentially inconsistent behavior between test client and normal WSGI requests
On Friday, April 24, 2015 at 10:51:12 PM UTC+2, Matt Hooks wrote: > > (The behavior of the development server is similar to the test client, > although I haven't investigated for what exact reason.) > To be honest, even on my production machines I have this behavior -- so the question is which "production" server you are using and which versions of $stuff. Cheers, Florian -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/c7db3c8e-24c7-4979-befc-bd8ed4d32cce%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Potentially inconsistent behavior between test client and normal WSGI requests
Took me a few minutes to narrow down the right environment to recreate this. I'm seeing this "issue" with Gunicorn (latest version ,19.3.0), and only when using the gaiohttp async worker (again latest version of aiohttp, 0.15.3). I've tracked it down to the troublesome line in aiohttp, but I imagine that isn't terrible relevent here. I'm not really familiar with the design ideals of the Django devs or the WSGI spec. From what I can tell, the spec doesn't specify whether the url should be unquoted before passing it to the application. This leaves us with the possibility of changing behavior when moving among different WSGI servers. While any decent developer should know there will be differences when they reconfigure their stack, it might make sense to ensure consistency for this particular detail. The only concern would be backwards compatibility, but from what I can tell, most of the gunicorn worker types currently already behave this way**, along with uwsgi. I imagine the majority of people are using one of those, so I doubt any signficant number are relying on this behavior, but I dont have much to back that up with. I havent checked any other implementations, like mod_wsgi. I suppose this boils down to whether or not Django should be normalizing the url path (PATH_INFO) from the WSGI server, or should just go with whatever it is provided. (** All the ones I checked, specifically sync, eventlet and tornado. The gevent ones dont play nice with my python3 install.) On Friday, April 24, 2015 at 5:07:59 PM UTC-4, Florian Apolloner wrote: > > > > On Friday, April 24, 2015 at 10:51:12 PM UTC+2, Matt Hooks wrote: >> >> (The behavior of the development server is similar to the test client, >> although I haven't investigated for what exact reason.) >> > > To be honest, even on my production machines I have this behavior -- so > the question is which "production" server you are using and which versions > of $stuff. > > Cheers, > Florian > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/271b15b5-2ff0-49b2-a5eb-c87ed99a5b04%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
