I've had to implement workarounds for such problems in many Django projects I created. For example I wrote code to determine which models have too many instances for dumping them all into a HTML dropdown and automatically add this model to raw_id_fields in related models.
The main difficulty I'm foreseeing is to provide a generic and useful set of restrictions. Even within specific projects, it was hard to come up with one-size-fits-all-models solutions. If we want to implement something in Django itself, the problem is much worse. Having workarounds for 70% of the places where Django handles large tables ungracefully isn't much better than 0%. It creates a false feeling of security. -- Aymeric. 2014-11-18 15:17 GMT+01:00 Rick van Hattem <wolp...@gmail.com>: > The framework is indeed generic which is why it should also take scenarios > into account where there is a lot more data. There is obviously no > one-size-fits-all solution which is why I propose configurable options. The > needed patches within the Django core are small and have very limited > impact but the debugging it can save when working with larger querysets is > enormous. > > Believe me when I tell you that debugging why your entire Django instance > is hanging at random times due to some erroneous query with an unfortunate > filter is painful at best. Especially since Django won't ever generate any > useful debugging output in those cases, it will simply kill your machine > and die. > > On Tuesday, November 18, 2014 3:09:27 PM UTC+1, Michael Manfre wrote: >> >> Django is a generic framework. Those who use it to implement >> functionality are the ones who kill either the browser or their entire >> application. Applications must be written to meet functional and >> performance requirements of the project. When dealing with large tables, >> more effort is certainly needed when using the Admin. E.g. write custom >> ModelAdmins or admin filters that restrict the resultset size. >> >> I don't think Django should automatically limit these queries. >> >> Regards, >> Michael Manfre >> >> On Tue, Nov 18, 2014 at 7:58 AM, Rick van Hattem <wol...@gmail.com> >> wrote: >> >>> Hi guys, >>> >>> As it is right now Django has the tendency to kill either your browser >>> (if you're lucky) or the entire application server when confronted with a >>> large database. For example, the admin always does counts for pagination >>> and a count over a table with many rows (say, in the order of 100M) can >>> really stall your app and database servers. >>> >>> In the past I've simply patched Django to include a few "security" >>> features for cases like these so someone can't accidently kill the site by >>> doing something silly, but I was wondering if it would be a good idea to >>> include some of these configurable options in the core. >>> >>> Examples: >>> - If COUNT_LIMIT is set than wrap counts in a subquery and limit the >>> results: SELECT COUNT(*) FROM (SELECT pk FROM table LIMIT <COUNT_LIMIT>) >>> - When someone does "if queryset" automatically limit it to a default >>> slice size so someone doesn't accidently fetch all rows in the queryset. >>> >>> ~wolph >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers (Contributions to Django itself)" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to django-develop...@googlegroups.com. >>> To post to this group, send email to django-d...@googlegroups.com. >>> Visit this group at http://groups.google.com/group/django-developers. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/django-developers/9ec3e5e6-0681-4e5b-bb8e- >>> c450ce38c96a%40googlegroups.com >>> <https://groups.google.com/d/msgid/django-developers/9ec3e5e6-0681-4e5b-bb8e-c450ce38c96a%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/97dce1de-29a6-41d9-969a-a7df0f4f217e%40googlegroups.com > <https://groups.google.com/d/msgid/django-developers/97dce1de-29a6-41d9-969a-a7df0f4f217e%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Aymeric. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANE-7mV-2_i3U8mGsaFcwhpDK_aM551TLU_LVpUnDg0O21kmdg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
