Improving django.setup() and app loading performance

[Rich Jones]

(Originally posted this as a ticket, but we can have discussion here and 
leave the ticket  for just 
more specific discussion.)

I imagine that this is an area that hasn't really been given much 
consideration with regards to optimization, because it isn't relevant to 
normal Django deployments. However, with "serverless" deployments (those 
without any permanent infrastructure), this becomes quite relevant as we 
have to call setup() every request. 


So, I'd love to discuss ideas for performance optimizations to Django's 
setup method. (A sample output for profile is available here: ​
https://github.com/Miserlou/django-zappa/issues/24 )


For starters - can we load apps in parallel? 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f85a73b4-a4a3-40a0-9035-d17a609ded16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Tim Graham]

I'm not sure we can load apps in parallel -- they are loaded in the order 
in which they appear in INSTALLED_APPS so the process is consistent. Maybe 
it would work for some setups, but I guess it would introduce subtle bugs 
in others.

https://docs.djangoproject.com/es/stable/ref/applications/#how-applications-are-loaded

On Friday, February 26, 2016 at 10:36:57 AM UTC-5, Rich Jones wrote:
>
> (Originally posted this as a ticket, but we can have discussion here and 
> leave the ticket  for just 
> more specific discussion.)
>
> I imagine that this is an area that hasn't really been given much 
> consideration with regards to optimization, because it isn't relevant to 
> normal Django deployments. However, with "serverless" deployments (those 
> without any permanent infrastructure), this becomes quite relevant as we 
> have to call setup() every request. 
>
>
> So, I'd love to discuss ideas for performance optimizations to Django's 
> setup method. (A sample output for profile is available here: ​
> https://github.com/Miserlou/django-zappa/issues/24 )
>
>
> For starters - can we load apps in parallel? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8fcafc86-4ae0-4fc8-a68b-02a2d911d3c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Rich Jones]

That might be true in theory.. but in practice?

Do any of the core/contrib/top100 apps actually depend on loading orders? 
I've never encountered that problem before, but I don't know.

It seems like we could easily get a 10x performance improvement by doing 
this in parallel by default, and have anything that _does_ have a 
dependency have some kind of DEPENDS_ON in the package init, so that the 
loader loop wouldn't complete until everything is ready. This would be a 
major benefit, and I don't think it would introduce all that much 
complexity.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f20ac9bc-2492-4b84-874f-6c1635eb15ab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Aymeric Augustin]

Hi Rich,

On 26 févr. 2016, at 16:36, Rich Jones  wrote:

> I'd love to discuss ideas for performance optimizations to Django's setup 
> method. 


In my opinion, the first concrete step would be to measure how much time is 
spent executing Django code rather than importing Python modules. That’s tricky 
because the two are intertwined: django.setup() mostly imports all the modules 
required by the project. If 90% of the time is spent importing modules and 10% 
doing other things, then there’s not much to gain by optimizing these other 
things!

While performance improvements are always good, I’m against changes that would 
re-introduce non-determism in the app-loading process.

Django 1.7 eliminated a whole class of bugs, included some that randomly 
occurred with low probability when restarting a production server under load. 
These bugs aren’t fun to track down.

The global import lock was replaced by per-module locks in Python 3.3. This 
implies you could import things in threads on Python 3.3+ and it may be faster. 
However I’m afraid any thread-based solution will re-introduce non-determism.

I remember trying to simplify:

- building Options classes (aka. Model._meta)
- building relations between models

and failing multiple times, due to the difficulty of preserving 
backwards-compatibility. I think it’s doable but it will take someone smarter, 
more persistent and/or more familiar with the app-loading process than I am. 
And I’m not even sure it would have an effect on performance!

Best regards,

-- 
Aymeric.


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/10BD8633-CF0F-48BE-A655-DB418EEEC418%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Cristiano Coelho]

If with "serverless" you are talking deployments such as Amazon Lambda or 
similar, I don't think setup is called on every request, at least for AWS 
Lambda, the enviorment is cached so it will only happen once each time it 
needs to scale up. Are there any other issues?

El viernes, 26 de febrero de 2016, 12:36:57 (UTC-3), Rich Jones escribió:
>
> (Originally posted this as a ticket, but we can have discussion here and 
> leave the ticket  for just 
> more specific discussion.)
>
> I imagine that this is an area that hasn't really been given much 
> consideration with regards to optimization, because it isn't relevant to 
> normal Django deployments. However, with "serverless" deployments (those 
> without any permanent infrastructure), this becomes quite relevant as we 
> have to call setup() every request. 
>
>
> So, I'd love to discuss ideas for performance optimizations to Django's 
> setup method. (A sample output for profile is available here: ​
> https://github.com/Miserlou/django-zappa/issues/24 )
>
>
> For starters - can we load apps in parallel? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/92beeb53-b347-40b1-928f-3998929a3334%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Rich Jones]

@Aymeric
> In my opinion, the first concrete step would be to measure how much time 
is spent executing Django code rather than importing Python modules.

You can find a complete profile of a Django request as it goes through the 
complete request/response loop here: 
https://github.com/Miserlou/django-zappa/files/141600/profile.txt

Over 70% of the total request time is spent in django.setup() - so you can 
see why we have an incentive to improve this! 


@ Cristiano - 
> If with "serverless" you are talking deployments such as Amazon Lambda or 
similar, I don't think setup is called on every request, at least for AWS 
Lambda, the enviorment is cached so it will only happen once each time it 
needs to scale up. Are there any other issues?

You're halfway there, but the process is more complicated than that. The 
code is cached, not the internal state of the machine. You can follow our 
progress here: https://github.com/Miserlou/django-zappa/

But - another type caching could be another possibility. Can anybody think 
of any arrangements where we could perhaps call setup() with 
"pre-loaded"/cached applications? 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a7d47b1d-7805-4085-a0d3-b7270c5966d3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Florian Apolloner]

On Friday, February 26, 2016 at 9:53:00 PM UTC+1, Rich Jones wrote:
>
> @Aymeric
> > In my opinion, the first concrete step would be to measure how much time 
> is spent executing Django code rather than importing Python modules.
>
> Over 70% of the total request time is spent in django.setup() - so you can 
> see why we have an incentive to improve this! 
>

That profile looks somewhat worthless imo, please provide it in a usable 
form, ie cProfile output. As Aymeric pointed out, most of the time in setup 
is most likely spent in importing python modules, which is nothing we can 
do about.
 

> You're halfway there, but the process is more complicated than that. The 
> code is cached, not the internal state of the machine. You can follow our 
> progress here: https://github.com/Miserlou/django-zappa/
>

What does this even mean? "code is cached"?! "internal state"!? which 
internal state?
 

> But - another type caching could be another possibility. Can anybody think 
> of any arrangements where we could perhaps call setup() with 
> "pre-loaded"/cached applications? 
>

Use a sane application stack? :D

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c666857e-cfe9-4b1d-a191-33204ec25e56%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Cristiano Coelho]

Rich,

I believe you know a lot way more than me about AWS Lambda since you have 
made such a great project, and I'm really interested to know how it really 
works since theyr documentation is a bit superficial.

On their FAQ this is what they state:

*Q: Will AWS Lambda reuse function instances?*

To improve performance, AWS Lambda may choose to retain an instance of your 
function and reuse it to serve a subsequent request, rather than creating a 
new copy. Your code should not assume that this will always happen.


I always thought, and with very simple testing seen, that your code is 
basically "frozen" between each request, and for example, every import that 
is done in the main module is always done once (so your whole program is 
only initialized once) so this means django would only initialize once for 
quite a while (until your instance of the code is discarded, and a new 
request will basically generate all modules to be imported again). So 
technically if you do the right imports to have django call setup(), this 
should be only done once. What is really happening that makes it always 
call setup() on every request?


With the above said, if that's really the case. Python is known to able to 
serialize classes in a very interesting way (pickling) where you can even 
send a class with its methods over the wire and on the other side the 
person can execute every method defined in there and the class also keeps 
any state. Would it be possible to store the state with this somehow?





El viernes, 26 de febrero de 2016, 17:53:00 (UTC-3), Rich Jones escribió:
>
> @Aymeric
> > In my opinion, the first concrete step would be to measure how much time 
> is spent executing Django code rather than importing Python modules.
>
> You can find a complete profile of a Django request as it goes through the 
> complete request/response loop here: 
> https://github.com/Miserlou/django-zappa/files/141600/profile.txt
>
> Over 70% of the total request time is spent in django.setup() - so you can 
> see why we have an incentive to improve this! 
>
>
> @ Cristiano - 
> > If with "serverless" you are talking deployments such as Amazon Lambda 
> or similar, I don't think setup is called on every request, at least for 
> AWS Lambda, the enviorment is cached so it will only happen once each time 
> it needs to scale up. Are there any other issues?
>
> You're halfway there, but the process is more complicated than that. The 
> code is cached, not the internal state of the machine. You can follow our 
> progress here: https://github.com/Miserlou/django-zappa/
>
> But - another type caching could be another possibility. Can anybody think 
> of any arrangements where we could perhaps call setup() with 
> "pre-loaded"/cached applications? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5c84c0a5-0ecd-4c80-8560-919cdd7b0879%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Collin Green]

Ugh. As a strong advocate of both django and zappa, I'd love it if we could 
keep the conversation on target without degenerating into stack attacks. If 
you don't want to weigh in, please feel no obligation to do so.

I do agree that we could pare down the profile into more actionable 
sections. I'll see if I can get something a bit more preprocessed in the 
near future. Perhaps that will help keep this thread on target.


On Friday, February 26, 2016 at 1:34:46 PM UTC-8, Florian Apolloner wrote:
>
>
>
> On Friday, February 26, 2016 at 9:53:00 PM UTC+1, Rich Jones wrote:
>>
>> @Aymeric
>> > In my opinion, the first concrete step would be to measure how much 
>> time is spent executing Django code rather than importing Python modules.
>>
>> Over 70% of the total request time is spent in django.setup() - so you 
>> can see why we have an incentive to improve this! 
>>
>
> That profile looks somewhat worthless imo, please provide it in a usable 
> form, ie cProfile output. As Aymeric pointed out, most of the time in setup 
> is most likely spent in importing python modules, which is nothing we can 
> do about.
>  
>
>> You're halfway there, but the process is more complicated than that. The 
>> code is cached, not the internal state of the machine. You can follow our 
>> progress here: https://github.com/Miserlou/django-zappa/
>>
>
> What does this even mean? "code is cached"?! "internal state"!? which 
> internal state?
>  
>
>> But - another type caching could be another possibility. Can anybody 
>> think of any arrangements where we could perhaps call setup() with 
>> "pre-loaded"/cached applications? 
>>
>
> Use a sane application stack? :D
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0093ead4-21e7-417a-b9e4-e331307dd312%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Cleaning up admin CSS

[Martin Whitehouse]

Hi all,

I'd like to give massive thanks to everyone for the Django project, it's 
made my life so incredibly simple it's unreal!

There are a few things that I'd like to help improve though.

To start with, I'd like to clean up the admin CSS to make it valid CSS.

Could someone please provide me with the basics on how to make changes to 
the actual source of the project and submit them for inclusion in an 
official release.

Many thanks and looking forward to helping as much as I can.

Martin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/64792e47-b52d-46eb-bb07-803ea7d826f9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Florian Apolloner]

Hi Collin,

On Friday, February 26, 2016 at 10:56:33 PM UTC+1, Collin Green wrote:
>
> Ugh. As a strong advocate of both django and zappa, I'd love it if we 
> could keep the conversation on target without degenerating into stack 
> attacks. If you don't want to weigh in, please feel no obligation to do so.
>

Sorry that wasn't ment as attack (maybe I should have written it out a 
little bit better). Please understand that most of us have no experience 
with Lambda or whatever you are currently using. I really have no idea what 
"cached code" or "internal state" in that sense means. As for the 
application stack: Since Python is slow to start up and on every bigger 
project you are going to need to implement loads of modules, you have a 
long startup time -- stuff like this prompted the switch from CGI to FCGI 
ages ago (this was what prompted me to make the sane application stack 
"joke"). Do not get me wrong, I love Django, but if you are going back to a 
CGI like environment, I am not convinced that it is the best tool (though 
if you can clear up what "cached code", "internal state", "serverless 
infrastructure" etc actually means instead of throwing them around like 
buzzwords and assuming we all know what you mean that would be helpful too).
 

> I do agree that we could pare down the profile into more actionable 
> sections. I'll see if I can get something a bit more preprocessed in the 
> near future. Perhaps that will help keep this thread on target.
>

Please do so, especially how much time is spent in importing and 
configuring python modules -- then substract those from your timings and 
see what else is left.

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/353200e9-1992-4eb0-a34b-eace95bbe9ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Cleaning up admin CSS

[Tim Graham]

Here is the contributing documentation: 
https://docs.djangoproject.com/en/dev/internals/contributing/

If you have further questions about contributing feel free to ask in 
#django-dev IRC or on the django-core-mentorship mailing list.

Thanks and I look forward to your contributions.

On Friday, February 26, 2016 at 4:56:34 PM UTC-5, Martin Whitehouse wrote:
>
> Hi all,
>
> I'd like to give massive thanks to everyone for the Django project, it's 
> made my life so incredibly simple it's unreal!
>
> There are a few things that I'd like to help improve though.
>
> To start with, I'd like to clean up the admin CSS to make it valid CSS.
>
> Could someone please provide me with the basics on how to make changes to 
> the actual source of the project and submit them for inclusion in an 
> official release.
>
> Many thanks and looking forward to helping as much as I can.
>
> Martin
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/52e05e42-0c88-4797-8685-f5dd05f61d65%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: should manage.py test run system checks?

[Tim Graham]

It looks like about 250 milliseconds for Django's test suite. It's quicker 
in the case of running an individual test since the checks will be 
restricted to the application that contains that test. I guess it's fine 
with me if we proceed with this change. After all, no one complained about 
in Django 1.7.

On Sunday, January 31, 2016 at 6:28:35 AM UTC-5, Žan Anderle wrote:
>
> Same for us. It takes slightly longer because of some custom checks, but 
> it's still under a second.
>
> Dne nedelja, 31. januar 2016 10.23.30 UTC+1 je oseba Adam Johnson napisala:
>>
>> Y'all know my position (original author). How long are the checks taking 
>> for people? We have a very large project with >100 models, ~30 apps, and it 
>> still takes less than a second.
>>
>> On Thursday, October 22, 2015 at 12:15:59 PM UTC+1, Žan Anderle wrote:
>>>
>>> Adam: I don't think they should be optional, or if they are, they should 
>>> be opt-out. The checks are a brilliant guard against error, but not running 
>>> them as part of test invites them not being run at all in a TDD 
>>> workflow, as often code can be developed with nothing but running the 
>>> tests. It is also surprising that *only* test doesn't run them, since 
>>> every other manage command does.
>>>
>>>
>>> I agree with this. I don't see why an opt-out option is not valid? 
>>>
>>> Dne torek, 20. oktober 2015 02.48.29 UTC+2 je oseba Tim Graham napisala:

 A ticket [1] and pull request [2] note that `manage.py test` ran the 
 system checks in Django 1.7 (as a side effect of call_command('migrate')), 
 but this is no longer the case in Django 1.8 since call_command() doesn't 
 trigger the system checks anymore.

 me: I'm of the opinion that running the system checks as part of the 
 manage.py 
 test command should be opt-in (for example, by writing a test that 
 asserts the call_command('check') output is empty. For example, when 
 debugging a single test, it doesn't seem necessary to have the overhead of 
 running check. As more options are added to check (e.g. #25500 
 ), a default 
 implementation as currently proposed could become increasing inflexible. 
 I'd like to document the change in the 1.8 release notes and suggest the 
 alternative.


 Adam: I don't think they should be optional, or if they are, they 
 should be opt-out. The checks are a brilliant guard against error, but not 
 running them as part of test invites them not being run at all in a 
 TDD workflow, as often code can be developed with nothing but running the 
 tests. It is also surprising that *only* test doesn't run them, since 
 every other manage command does.
 At YPlan we couldn't do without them as part of tests. Our 
 aforementioned 'installed packages' check saves a lot of time that would 
 otherwise be wasted understanding confusing error messages about imports 
 not working, and our other custom checks do verification similar to 
 Django's, for issues that without resolution it does not make sense to 
 even 
 attempt do any tests. Also we don't notice any real overhead, we can still 
 get a single test to run in 1 second (with --keepdb :) ) despite all 
 our extra messing around with pip freeze etc.

 Other opinions?

 [1] https://code.djangoproject.com/ticket/25415
 [2] https://github.com/django/django/pull/5293

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/bd24735e-b8d6-443d-a564-3b4a81b4df03%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3 level nested prefetch is not working as expected on Django 1.8.x

[Dheerendra Rathor]

Posting this from 
django-users https://groups.google.com/forum/#!topic/django-users/v3KelvexqiI.

Three level nested prefetch with custom queryset throws an Attribute Error. 
> It works perfectly fine on Django 1.9 but I was unable to find release 
> notes or bug fix notes of the same. 
>
> To reproduce: 
> 1. Install Django 1.8.6+
> 2. Add models from 
> https://github.com/DheerendraRathor/Django-bugs/blob/master/prefetch_one/models.py
> 3. Run the script from 
> https://github.com/DheerendraRathor/Django-bugs/blob/master/prefetch_one/script.py
>
> It throws: AttributeError: Cannot find 'videos' on QuizMarker object, 
> 'videos__markers__quiz__quiz_histories' is an invalid parameter to 
> prefetch_related()
> however, running the same with Django 1.9 works fine as expected. 
>

 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/837377ce-28c1-41ba-a08d-f6a719c0db04%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.