Since your use case seems to be "avoid blocking the server when one pageview uses unexpectedly many resources", and you don't want to fix or optimise the actual application code, why not set these limits at a level where it makes sense?
For example: * http://uwsgi-docs.readthedocs.org/en/latest/Options.html#limit-as sets a memory allocation limit on the UWSGI worker processes, and terminates & restarts the worker if it uses more memory. This will protect against not just big querysets, but also big plain python lists, loading huge image or xml files, memory leaks in C extensions and any other memory-consuming issues. * http://uwsgi-docs.readthedocs.org/en/latest/FAQ.html#what-is-harakiri-mode sets a timeout on request handling, and terminates & restarts the worker if it takes more than X seconds to serve the request. This protects against not just big querysets, but also any other slow code that ties up the worker, for example forgetting to set a timeout on calls to external APIs, or a plain old infinite loop. * http://stackoverflow.com/questions/5421776/how-can-i-limit-database-query-time-during-web-requests You can set a Postgresql query timeout in seconds for the queries run by the web app user. This makes far more sense than a limit on the result size, as complex JOINs can be very resource-intensive but only return a few rows in the result. There should be similar configuration options for other app servers and databases as well. Django's ORM is just the wrong level in the software stack for these limits, since there are hundreds of other ways to kill the performance of a server, and the number of results in a queryset is a poor indicator of performance issues. On Sunday, 23 November 2014 23:53:26 UTC+2, Rick van Hattem wrote: > > > On 23 Nov 2014 22:13, "Christophe Pettus" <x...@thebuild.com <javascript:>> > wrote: > > > > > > On Nov 23, 2014, at 1:07 PM, Rick van Hattem <wo...@wol.ph <javascript:>> > wrote: > > > > > > Not really, cause psycopg already fetched everything. > > > > > > Not if Django limits it by default :) > > > > Unfortunately, that's not how it works. There are three things that > take up memory as the result of a query result: > > > > 1. The Django objects. These are created in units of 100 at a time. > > 2. The psycopg2 Python objects from the result. These are already > limited to a certain number (I believe to 100) at a time. > > 3. The results from libpq. These are not limited, and there is no way > of limiting them without creating a named cursor, which is a significant > change to how Django interacts with the database. > > > > In short, without substantial, application-breaking changes, you can't > limit the amount of memory a query returns unless you add a LIMIT clause to > it. However, adding a LIMIT clause can often cause performance issues all > by itself: > > > > http://thebuild.com/blog/2014/11/18/when-limit-attacks/ > > > > There's no clean fix that wouldn't have significant effects on > unsuspecting applications. > > Very true, that's a fair point. That's why I'm opting for a configurable > option. Patching this within Django has saved me in quite a few cases but > it can have drawbacks. > > > -- > > -- Christophe Pettus > > x...@thebuild.com <javascript:> > > > > -- > > You received this message because you are subscribed to a topic in the > Google Groups "Django developers (Contributions to Django itself)" group. > > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/django-developers/aYwPykvLaMU/unsubscribe > . > > To unsubscribe from this group and all its topics, send an email to > django-develop...@googlegroups.com <javascript:>. > > To post to this group, send email to django-d...@googlegroups.com > <javascript:>. > > Visit this group at http://groups.google.com/group/django-developers. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/CD10DACF-4D0A-458D-BB85-0F7BB8BFF4C0%40thebuild.com > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/e17fb633-b3d6-422c-9f6d-dc7fddae1aab%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
