#35408: Optimize post-migrate permission creation
-------------------------------------+-------------------------------------
Reporter: Adam Johnson | Owner: Adam
Type: | Johnson
Cleanup/optimization | Status: assigned
Component: contrib.auth | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Description changed by Adam Johnson:
Old description:
> I have often seen `django.contrib.auth.management.create_permissions()`
> take a significant amount of time in test run profiles. It can be
> optimized by batching more of its operations, including making
> `ContentTypeManager.get_for_models()` use batch creation.
>
> For a comparison, I profiled 1518 of Django’s 1518 tests in modules
> called “models”:
>
> {{{
> $ python -m cProfile -o profile runtests.py --parallel 1 *model*
>
> $ python -m pstats profile <<< 'sort cumtime
> stats 10000' | less
> }}}
>
> Before optimization stats:
>
> * Total 11,938,857 function calls taking 5.349 seconds.
> * 88 calls to `create_permissions()` take 456ms, ~8.5% of the total time.
>
> After optimization stats:
>
> * Total 11,359,071 function calls taking 5.035 seconds.
> * 88 calls to `create_permissions()` now take 239ms, ~4.7% of the toal
> time.
> * 217ms and 579,786 function calls saved.
>
> Optimization is limited because the `post_migrate` signal runs once per
> migrated app config, so there’s no chance to bulk create *all* content
> types and permissions at once. If we introduced a new “all migrated apps”
> signal, that could reduce runtime further by batching all creation.
New description:
I have often seen `django.contrib.auth.management.create_permissions()`
take a significant amount of time in test run profiles. It can be
optimized by batching more of its operations, including making
`ContentTypeManager.get_for_models()` use batch creation.
For a comparison, I profiled 1518 of Django’s tests in modules called
“models”:
{{{
$ python -m cProfile -o profile runtests.py --parallel 1 *model*
$ python -m pstats profile <<< 'sort cumtime
stats 10000' | less
}}}
Before optimization stats:
* Total 11,938,857 function calls taking 5.349 seconds.
* 88 calls to `create_permissions()` take 456ms, ~8.5% of the total time.
After optimization stats:
* Total 11,359,071 function calls taking 5.035 seconds.
* 88 calls to `create_permissions()` now take 239ms, ~4.7% of the toal
time.
* 217ms and 579,786 function calls saved.
Optimization is limited because the `post_migrate` signal runs once per
migrated app config, so there’s no chance to bulk create *all* content
types and permissions at once. If we introduced a new “all migrated apps”
signal, that could reduce runtime further by batching all creation.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/35408#comment:5>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/0107018f622fbded-e615ad31-74a2-4e3a-8cda-e59d13132830-000000%40eu-central-1.amazonses.com.
