It seems you are misunderstanding what I am trying to do here. The 10,000 (or whatever, that should be configurable) is a number large enough not to bother anyone but small enough not to trigger the OOM system. In that case it works perfectly.
Due note that the proposed solution is not a hypothetical one, it's a tried and proven one that has saved me a lot of headache in the past. It's definitely helped a lot when I was trying 3rd party apps at Fashiolista.com, one of the larger Django based websites. On 24 November 2014 at 19:39, Christophe Pettus <x...@thebuild.com> wrote: > > On Nov 24, 2014, at 3:36 AM, Rick van Hattem <wo...@wol.ph> wrote: > > If you fetch N+1 items you know if there are over N items in your list. > > Let's stop there. Unfortunately, because of the way libpq works, just > sending the query and checking the result set size won't solve your > problem, except for an even smaller edge case. > > Using the standard (non-named-curosr) protocol, when you get the first > result back from libpq, *every result* is sent over the wire, and takes up > space on the client. Thus, no limitation on the client side (number of > Django objects created, number of Python objects created by psycopg2) will > prevent an out-of-memory error. In my experience (and I've seen this > problem a lot), the OOM occurs on the libpq results, not on the other parts. > > Thus, the proposal only solves the relatively narrow case where the libpq > result does *not* create an OOM, but creating the Django and psycopg2 > objects does. > > I'll note also that it's not that server that dies in this case; the > particular thread doing the request gets an exception. > > -- > -- Christophe Pettus > x...@thebuild.com > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Django developers (Contributions to Django itself)" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/django-developers/aYwPykvLaMU/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/81A38D7E-E355-46D1-9C21-1C7D43BCC648%40thebuild.com > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAJAwA%3Dx4X3CJ%3DEXfFz9xTvNzfRYYmq2r3VL4OPdSRteJ6P5pXQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
