Hello Tobias,
>From my understanding the introduction of chunk_size doest't help here.
The fundamental reason why iterator() cannot be used with
prefetch_related() is that the latter requires a set of model instance to
be materialized to work appropriately which chunk_size doesn't control at
all.
In other words chunk_size only controls how many rows should be fetched
from the database cursor and kept into memory at a time. Even when this
parameter is used, iterator() will only materialize a single model instance
per yield.
Now, something you could do if you really want to be using prefetch_related
with iterator() is to materialize chunks and use the now public
prefetch_related_objects[0] helper to prefetch relationships.
A somewhat complete implementation would look like
def prefetch_related_iterator(queryset, *related_lookups, chunk_size=100):
iterator = queryset.iterator(chunk_size=chunk_size)
while True:
chunk = list(itertools.islice(iterator, chunk_size))
if not chunk:
return
prefetch_related_objects(chunk, *related_lookups)
yield from chunk
Given that iterator() always ignored prefetch related lookups instead of
erroring out when they were specified make me feel like turning such a
feature on by default could be problematic as it could balloon the memory
usage which is the main reason why iterator is useful anyway.
Cheers,
Simon
[0]
https://docs.djangoproject.com/en/2.1/ref/models/querysets/#django.db.models.prefetch_related_objects
Le jeudi 11 octobre 2018 15:44:06 UTC-4, tobias....@truffls.com a écrit :
>
> Hi everyone!
>
> The docs (
> https://docs.djangoproject.com/en/2.1/ref/models/querysets/#iterator)
> state that the
>
> use of iterator() causes previous prefetch_related() calls to be ignored
>> since these two optimizations do not make sense together.
>
>
> I am wondering, if this is still true with the introduction of the
> *chunk_size* parameter. Having only one additional query per chunk could
> heavily reduce the amount of queries per batch.
>
> Best regards
> Tobi
>
>
>
> *Truffls GmbH*
> Chausseestraße 86, 10115 Berlin
> <https://www.google.com/maps/place/Truffls+GmbH/@52.5375244,13.3715239,17z/data=!3m1!4b1!4m5!3m4!1s0x47a851ece418ed87:0xdab2d728c831f006!8m2!3d52.5375212!4d13.3737126>
> https://truffls.de
>
> Vertretungsberechtigte Geschäftsführer: Clemens Dittrich, Matthes Dohmeyer
> Amtsgericht Charlottenburg, HRB 160036B
>
> Diese Nachricht ist vertraulich. Sie darf ausschließlich durch den
> vorgesehenen Empfänger und Adressaten gelesen, kopiert oder genutzt werden.
> Sollten Sie diese Nachricht versehentlich erhalten haben, bitten wir, den
> Absender (durch Antwort-E-Mail) hiervon unverzüglich zu informieren und die
> Nachricht zu löschen. Jede Nutzung oder Weitergabe des Inhalts dieser
> Nachricht ist unzulässig.
> This message (including any attachments) is confidential and may
> be privileged. It may be read, copied and used only by the intended
> recipient. If you have received it in error please contact the sender (by
> return e-mail) immediately and delete this message. Any unauthorized use or
> dissemination of this message in whole or in part is strictly prohibited.
>
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/afe9c247-45ac-4f63-8673-2b8b4c337d54%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
