Re: QuerySet.iterator together with prefetch_related because of chunk_size

2020-05-21 Thread Adam Johnson
After over a year, it's probably reasonable to create a new PR. You can use Co-Authored-By to credit the original author: https://github.blog/2018-01-29-commit-together-with-co-authors/ On Fri, 22 May 2020 at 05:43, 'Taylor Hakes' via Django developers (Contributions to Django itself) wrote: > T

Re: QuerySet.iterator together with prefetch_related because of chunk_size

2020-05-21 Thread 'Taylor Hakes' via Django developers (Contributions to Django itself)
That makes sense to me. It is a little weird, but it’s the best option. I will post a comment on the PR. I am happy to update the code if he’s not available. On Wed, May 20, 2020 at 9:59 PM charettes wrote: > Taylor, I think that 2. is the way forward. > > It looks like the linked PR doesn't en

Re: QuerySet.iterator together with prefetch_related because of chunk_size

2020-05-20 Thread charettes
Taylor, I think that 2. is the way forward. It looks like the linked PR doesn't ensure a chunk_size is specified to turn on the feature though. I like Adam's idea to first warn and don't do a prefetch if chunk_size is not specified and eventually turn that into an exception. What do you think o

Re: QuerySet.iterator together with prefetch_related because of chunk_size

2020-05-20 Thread 'Taylor' via Django developers (Contributions to Django itself)
Checking in here. What are the next steps to make a decision? On Wednesday, February 13, 2019 at 2:06:29 PM UTC-5, Taylor wrote: > > I agree that 2 makes sense and isn't a huge usability issue. It looks like > the PR is ready to go whenever this is decided > https://github.com/django/django/pull

Re: QuerySet.iterator together with prefetch_related because of chunk_size

2019-02-13 Thread Taylor
I agree that 2 makes sense and isn't a huge usability issue. It looks like the PR is ready to go whenever this is decided https://github.com/django/django/pull/10707/files On Monday, January 14, 2019 at 5:29:18 PM UTC-5, Adam Johnson wrote: > > I agree with your reasoning, and also favour option

Re: QuerySet.iterator together with prefetch_related because of chunk_size

2019-01-14 Thread Adam Johnson
I agree with your reasoning, and also favour option 2 now, especially since the default can break on sqlite. It would also be possible to go through a deprecation period to first raise a warning and not prefetch, before a later version raises an exception, which is probably kinder since previously

Re: QuerySet.iterator together with prefetch_related because of chunk_size

2019-01-14 Thread charettes
> This seems reasonable, but would you do in the case chunk_size isn't explicitly defined - throw an exception? That's would be the plan. > Currently it silently fails to prefetch which means N + 1 queries, so even prefetching for the default chunk_size of 2000 would be a huge gain in cases wh

Re: QuerySet.iterator together with prefetch_related because of chunk_size

2019-01-14 Thread Adam Johnson
> > ...what if we required chunk_size to be explicitly specified when the > queryset has prefetch lookups? This seems reasonable, but would you do in the case chunk_size isn't explicitly defined - throw an exception? Currently it silently fails to prefetch which means N + 1 queries, so even prefe

Re: QuerySet.iterator together with prefetch_related because of chunk_size

2019-01-12 Thread charettes
Replying to my concerns about the P * C queries. Throwing that idea out there but what if we required chunk_size to be explicitly specified when the queryset has prefetch lookups? That would at least force the developers to consider how many prefetching queries iterating through the results wou

Re: QuerySet.iterator together with prefetch_related because of chunk_size

2018-11-23 Thread Tim Graham
https://code.djangoproject.com/ticket/29984 suggests to support prefetch_related() with QuerySet.iterator(). I accepted the ticket to do something and linked back to this discussion. On Friday, October 26, 2018 at 8:12:02 PM UTC-4, charettes wrote: > > Josh, I agree that silently not working is

Re: QuerySet.iterator together with prefetch_related because of chunk_size

2018-10-26 Thread charettes
Josh, I agree that silently not working is problematic but it has been this way since prefetch_related() was introduced. Something to keep in mind as well is that silently turning it on would also perform P * C extra queries where P is the number of prefetches requested through prefetch_related

Re: QuerySet.iterator together with prefetch_related because of chunk_size

2018-10-26 Thread Josh Smeaton
I tend to agree with Tobi. Prefetching silently not working on iterator can be quite confusing, unless you have a good understanding of both APIs. It might be possible to do what you're asking, but it'd mean that django is now actually caching the result when it explicitly says it isn't - even i

Re: QuerySet.iterator together with prefetch_related because of chunk_size

2018-10-15 Thread tobias . kroenke
Thank you for your feedback. I would like to answer some statements to either convince you or make it more clear, where my idea stems from: The fundamental reason why iterator() cannot be used with > prefetch_related() is that the latter requires a set of model instance to > be materialized to

Re: QuerySet.iterator together with prefetch_related because of chunk_size

2018-10-11 Thread Curtis Maloney
On 10/12/18 10:51 AM, charettes wrote: Hello Tobias, From my understanding the introduction of chunk_size doest't help here. The fundamental reason why iterator() cannot be used with prefetch_related() is that the latter requires a set of model instance to be materialized to work appropriate

Re: QuerySet.iterator together with prefetch_related because of chunk_size

2018-10-11 Thread charettes
Hello Tobias, >From my understanding the introduction of chunk_size doest't help here. The fundamental reason why iterator() cannot be used with prefetch_related() is that the latter requires a set of model instance to be materialized to work appropriately which chunk_size doesn't control at a

QuerySet.iterator together with prefetch_related because of chunk_size

2018-10-11 Thread tobias . kroenke
Hi everyone! The docs (https://docs.djangoproject.com/en/2.1/ref/models/querysets/#iterator) state that the use of iterator() causes previous prefetch_related() calls to be ignored > since these two optimizations do not make sense together. I am wondering, if this is still true with the int