#34067: django.core.Paginator wrong query slicing
-------------------------------------+-------------------------------------
Reporter: Hristo Trendafilov | Owner: nobody
Type: Bug | Status: closed
Component: Core (Other) | Version: 3.2
Severity: Normal | Resolution: needsinfo
Keywords: | Triage Stage:
Paginator,slice,queryset | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Hristo Trendafilov:
Old description:
> A have a ListView defined like so:
>
> {{{
> class AllPracticesListView(LoginRequiredMixin, PermissionRequiredMixin,
> ListView, ToolBoxFunctions):
> template_name = 'practice_list.html'
> model = Practice
> paginate_by = 6
> permission_required = 'customauth.access_practices'
>
> def get_queryset(self):
> qs = self.get_all_practices_by_user().order_by(
> 'company_branch__pk',
> 'practice_profile__personal_profile__last_name').prefetch_related('practice_profile')
>
> # method < get_all_practices_by_user > is a toolbox method that
> returns:
> #
> Practice.objects.filter(company_branch__owner=self.request.user).order_by('pk')
>
> return qs
>
> }}}
>
> Models are defined like so:
> {{{
>
> class Practice(models.Model):
> company_branch = models.ForeignKey(
> CompanyBranch,
> on_delete=models.PROTECT,
> related_name='practices',
> )
> # CompanyBranch model has an owner field which ForeignKey
> to the user model
>
> practice_profile = models.ForeignKey(
> PracticeProfile,
> on_delete=models.PROTECT,
> related_name='practices',
> )
>
> class PracticeProfile(models.Model):
> personal_profile = models.ForeignKey(
> PersonalProfile,
> on_delete=models.PROTECT,
> blank=True,
> null=True,
> )
> # PersonalProfile model has a field called <
> last_name >
> }}}
> Or schematically:
> {{{
> Practice object relations:
> -> CompanyBranch
> -> PracticeProfile
> -> PersonalProfile /*NULLABLE RELATION/
>
> }}}
>
> When the code is run, the view does not display results correctly.
>
> I did like so to get the WRONG result:
> Added a breakpoint on {{{Paginator._get_page}}} return statement
>
> Queryset is passed okay and is available in
> {{{Paginator.object_list}}}
>
> But in the args of {{{Paginator._get_page}}} the queryset is totally
> different than expected / should be a slice from the first six elements
> of {{{Paginator.object_list}}} /
>
> I have tried to add a breakpoint like so
> [[Image(https://imgur.com/SdaQUt6)]]
> Then I get this result [[Image(https://imgur.com/5zzLNV0)]].
>
> As you can see, PKs 1632, 1624, etc. are missing, objects are
> different/marked red and purple/, and are actually removed and never
> shown in the view.
>
> If you run slice manually at this point - everything is again fine
> [[Image(https://imgur.com/yvn8KMO)]]
> This happens only on NULL PersonalProfile objects.
>
> I did like so to get OKAY results:
>
> 1. Added a simple breakpoint is added on {{{Paginator.page}}} like so
> [[Image(https://imgur.com/d7J5BMZ)]] and that breakpoint is just run then
> the result is okay [[Image(https://imgur.com/ILDHAMN)]]
>
> 2. changed paginate_by to 10
>
> 3. changed {{{order_by}}} in {{{get_queryset}}} to {{{-pk}}}
>
> 4. call again the {{{page = paginator.page(page_number)}}}
>
> P.S. That is also happening in django 2.2
New description:
A have a ListView defined like so:
{{{
class AllPracticesListView(LoginRequiredMixin, PermissionRequiredMixin,
ListView, ToolBoxFunctions):
template_name = 'practice_list.html'
model = Practice
paginate_by = 6
permission_required = 'customauth.access_practices'
def get_queryset(self):
qs = self.get_all_practices_by_user().order_by(
'company_branch__pk',
'practice_profile__personal_profile__last_name').prefetch_related('practice_profile')
# method < get_all_practices_by_user > is a toolbox method that
returns:
#
Practice.objects.filter(company_branch__owner=self.request.user).order_by('pk')
return qs
}}}
Models are defined like so:
{{{
class Practice(models.Model):
company_branch = models.ForeignKey(
CompanyBranch,
on_delete=models.PROTECT,
related_name='practices',
)
# CompanyBranch model has an owner field which ForeignKey
to the user model
practice_profile = models.ForeignKey(
PracticeProfile,
on_delete=models.PROTECT,
related_name='practices',
)
class PracticeProfile(models.Model):
personal_profile = models.ForeignKey(
PersonalProfile,
on_delete=models.PROTECT,
blank=True,
null=True,
)
# PersonalProfile model has a field called < last_name
>
}}}
Or schematically:
{{{
Practice object relations:
-> CompanyBranch
-> PracticeProfile
-> PersonalProfile /*NULLABLE RELATION/
}}}
When the code is run, the view does not display results correctly.
I did like so to get the WRONG result:
Added a breakpoint on {{{Paginator._get_page}}} return statement
Queryset is passed okay and is available in
{{{Paginator.object_list}}}
But in the args of {{{Paginator._get_page}}} the queryset is totally
different than expected / should be a slice from the first six elements of
{{{Paginator.object_list}}} /
I have tried to add a breakpoint like so
[[Image(https://imgur.com/SdaQUt6)]]
Then I get this result [[Image(https://imgur.com/5zzLNV0)]].
As you can see, PKs 1632, 1624, etc. are missing, objects are
different/marked red and purple/, and are actually removed and never shown
in the view.
If you run slice manually at this point - everything is again fine
[[Image(https://imgur.com/yvn8KMO)]]
This happens only on NULL PersonalProfile objects.
I did like so to get OKAY results:
1. Added a simple breakpoint is added on {{{Paginator.page}}} like so
[[Image(https://imgur.com/d7J5BMZ)]] and that breakpoint is just run then
the result is okay [[Image(https://imgur.com/ILDHAMN)]]
2. changed paginate_by to 10
3. changed {{{order_by}}} in {{{get_queryset}}} to {{{-pk}}}
4. call again the {{{page = paginator.page(page_number)}}}
It is Django fault for those reasons:
* `order_by` might be complicated but is supported by the framework
* Chained `order_by` is also supported
* queryset slicing is a feature of Django queryset API.
* queryset slicing should act like list and string slicing and namely to
provide you with results `[ from : to ]` indexes provided and not to
change or lose data in between. Or in other words - what is passed in the
`Paginator.object_list` should be just sliced `[ from : to ]`. `Paginator`
should never care for the queryset, what is the ordering of the queryset,
what is excluded and so on - it should just work with it.
* changing `paginate_by` to `10` or greater fixes the problem.
`paginate_by` is a standard and documented Django ListView setting.
* changing `order_by` to some other `order_by` also fixes the problem.
`order_by` should work the same or if any limitations those should be well
documented
* removing `order_by` also fixes the problem.
P.S. That is also happening in Django 2.2
--
--
Ticket URL: <https://code.djangoproject.com/ticket/34067#comment:6>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/010701839c961d5d-95ef6060-da73-4015-9bf2-24c9ecca8180-000000%40eu-central-1.amazonses.com.
