On Sunday, March 12, 2017 at 4:53:25 AM UTC+5:30, James Schneider wrote: > > > > On Mar 11, 2017 12:01 PM, "Vijay Khemlani" <[email protected] > <javascript:>> wrote: > > Am I the only one who thinks that generating a report over a set of > just 10.000 records could be done in 10 - 20 secs unless there are > some serious computations going on with that data? > > For a report I have to query around 200.000 records, with > aggregations, and it takes less than a minute using the ORM. > > > The OP never mentioned a time interval that I can find in this thread, > only CPU utilization. I can only imagine that the query is taking long > enough to notice the CPU utilization, which would be at least a few seconds. > > Querying and aggregating 200K records within the DB is not comparable to > pulling 10K individual records and performing processing on each one. An > ORM call with aggregation will perform a large majority of the work in the > DB, and the ORM simply wraps the response accordingly. >
I agree. The task was mostly to pull data from DB and generate the CSV. The model structure was as follows: Main Model A A has foreignkey to model D A has foriegnkey to model E F has foreignkey to model D Model B having manyTomany relationship with A through B1 Model C having manyTomany relationship with A through C1 It was mostly DB fetches which was the complex part. Something like fetch the latest of B related to each record of A. Fetch the latest of F related to the record of D which is related to each record of A. There were no CPU intensive calculations done. Though the overall DB fetches were complex, as mentioned in my response to Vijay, I just broke down the issue and tried the single queryset with select_related and prefetch_related and also "defer" and "only" to make the query lighter. But weren't helping. In fact the prefetch related on B and C were making the overall query heavier and hence, increasing the response time further. Please note the above example was just the subset of the overall DB relations and models. > -James > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/b169cc33-8120-441c-847d-6977d0e0712a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
