On Nov 24, 2014, at 11:16 AM, Rick van Hattem <wo...@wol.ph> wrote: > It seems you are misunderstanding what I am trying to do here. The 10,000 (or > whatever, that should be configurable) is a number large enough not to bother > anyone but small enough not to trigger the OOM system.
There are really only four options that could be implemented at the framework level: 1. You add a LIMIT to every query. As noted, this can change the performance of a query, sometimes quite radically, often in a bad way. An OOM error is clear and explicit; a query suddenly running 100x slower is a lot stranger to track down. (Also, unless you decide that returning the entire LIMIT amount is an error, you can't really know if the query would have returned *more* than that.) 2. You don't add the LIMIT. At that point, you have no guard against excessive memory consumption. 3. You do two queries, one a count(*), one to get the result. This is a huge protocol change, and quite inefficient (and without wrapping it a transaction at an elevated isolation mode, not reliable). 4. You use a named cursor. This is also a huge protocol change. Those are your choices. It's not that the *problem* is unusual; it's that being unable to fix the problem in the applcication is unusual. I've encountered this problem, sure, and what I did was fix the code; it took like two minutes. Indeed, I would *strongly* advise that if you are issuing queries that you expect to get 100 results back and are getting a memory-crushing result back, you fundamentally don't understand something about your data, and need to address that promptly. Running properly on large datasets is a big job; just a patch like this isn't going to solve all the issues. In your particular case, where you have the relatively unusual situation that: 1. You have this problem, and, 2. You can't fix the code to solve this problem. ... you probably have the right answer is having a local patch for Django. -- -- Christophe Pettus x...@thebuild.com -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5224637B-A37E-4B9F-A6F9-B95B405A82BF%40thebuild.com. For more options, visit https://groups.google.com/d/optout.
