Re: Improving django.setup() and app loading performance

Hello Rich, > On 01 Mar 2016, at 01:44, Rich Jones wrote: > > I think the interesting things to explore would be: > - Loading apps in parallel > - "Pre-loading" apps before app deployment, then loading that cached state > at runtime. I guess I'd need to know more about what it means to "load

Re: Improving django.setup() and app loading performance

In my opinion, those latencies are still a bit high, how much is really used on python/lambda code? On a project of mine without hitting the database and django-rest-framework my times were around 1-4ms excluding any network latency. Debug and loggers might have high impact on your times if it

Re: Improving django.setup() and app loading performance

Ah, interesting! Thanks for tracking that down. In chat we basically discovered that the intercontinental latency and shoddy wifi connections were responsible for a lot of the confusion. Testing from a US-based fiber connection, we got results of ~40ms in both scenarios. --- https://arb9clq9

Re: Improving django.setup() and app loading performance

I think I have found your issue, if I'm not wrong, django won't initialize twice if you call it twice (which was your previous behaviour) at least not completely, since apps registry has a "ready" flag. Check this line: https://github.com/django/django/blob/master/django/apps/registry.py#L66 So

Re: Improving django.setup() and app loading performance

Hm. This is the downside of cloud services - we cannot look under the hood. Since I think that since this is something we _want_ cached, and because it will make the function being executed shorter in length - it is a good code decision to make. Thank you for the idea! However, it looks like the

Re: Improving django.setup() and app loading performance

Rich, I have just performed a real test, with a simple lambda and a datetime.now() defined at the top of the module as I said, and out of 100 requests, this was the result: {u'2016-03-01T00:37:30.476828': [43], u'2016-03-01T00:36:51.536025': [58]} Where the date is the datetime.now() defined at t

Re: Improving django.setup() and app loading performance

That certainly could have something to do with it - there isn't very much transparency about how API Gateway works. It's super new and pretty janky, TBQH. However, I think that the behavior describing is not what's expected - the caching seems to be for the assets of the whole environment, not o

Re: Improving django.setup() and app loading performance

I have repeated the test this time through API Gateway, and out of many calls I only got two different dates that were instantiated at module level, meaning my module was only imported twice. I fail to see why it doesn't behave the same with your code. El lunes, 29 de febrero de 2016, 21:33:05

Re: Improving django.setup() and app loading performance

That's quite odd, I recall testing this once, where I created a lambda which had a datetime.now() at the top, and just returned that value. Out of a few calls, it returned two different results, meaning the module was re used "most" of the time. This was tested calling the lambda from the AWS T

Re: Improving django.setup() and app loading performance

As I suspected, moving setup() outside of the handler had a negligible effect - in fact the test showed a slight drop in performance. :( Testing from httping. From Berlin to US-East-1: Before: --- https://arb9clq9k9.execute-api.us-east-1.amazonaws.com/unicode/json_example/ ping statistics ---

Re: Improving django.setup() and app loading performance

I'm almost sure that right now you are calling setup() with django already initialized in some cases where the enviorment is reused, I'm amazed django doesn't complain when setup() is called twice. El lunes, 29 de febrero de 2016, 21:07:32 (UTC-3), Rich Jones escribió: > > Haven't tried that! I

Re: Improving django.setup() and app loading performance

Haven't tried that! I don't _think_ that'll work.. but worth a shot (I don't think they only cache the handler.. I think they cache the whole environment). Will report back! And as you're mentioning, we really shouldn't be doing it every request, so if there were even a way to cache that manual

Re: Improving django.setup() and app loading performance

Sorry if this looks like a retarded question, but have you tried the setup calls to the top of the lambda_handler module? I'm not sure why you need the settings data as an argument to the lambda handler, but if you find a way to move those 4 lines near setup(), you will only load the whole djang

Re: Improving django.setup() and app loading performance

For those who are still following along, these are the lines in question: https://github.com/Miserlou/django-zappa/blob/master/django_zappa/handler.py#L31 https://github.com/Miserlou/django-zappa/blob/master/django_zappa/handler.py#L68 It's very possible there are ways of significantly improving

Re: Improving django.setup() and app loading performance

Hey all! Let me clarify a few of the terms here and describe a bit how Django operates in this context. "Serverless" in this contexts means "without any permanent infrastructure". The server is created _after_ the request comes in, and it dies once the response is returned. The means that we n

Re: Improving django.setup() and app loading performance

Hi Collin, On Friday, February 26, 2016 at 10:56:33 PM UTC+1, Collin Green wrote: > > Ugh. As a strong advocate of both django and zappa, I'd love it if we > could keep the conversation on target without degenerating into stack > attacks. If you don't want to weigh in, please feel no obligation

Re: Improving django.setup() and app loading performance

Ugh. As a strong advocate of both django and zappa, I'd love it if we could keep the conversation on target without degenerating into stack attacks. If you don't want to weigh in, please feel no obligation to do so. I do agree that we could pare down the profile into more actionable sections. I

Re: Improving django.setup() and app loading performance

Rich, I believe you know a lot way more than me about AWS Lambda since you have made such a great project, and I'm really interested to know how it really works since theyr documentation is a bit superficial. On their FAQ this is what they state: *Q: Will AWS Lambda reuse function instances?*

Re: Improving django.setup() and app loading performance

On Friday, February 26, 2016 at 9:53:00 PM UTC+1, Rich Jones wrote: > > @Aymeric > > In my opinion, the first concrete step would be to measure how much time > is spent executing Django code rather than importing Python modules. > > Over 70% of the total request time is spent in django.setup() -

Re: Improving django.setup() and app loading performance

@Aymeric > In my opinion, the first concrete step would be to measure how much time is spent executing Django code rather than importing Python modules. You can find a complete profile of a Django request as it goes through the complete request/response loop here: https://github.com/Miserlou/dj

Re: Improving django.setup() and app loading performance

If with "serverless" you are talking deployments such as Amazon Lambda or similar, I don't think setup is called on every request, at least for AWS Lambda, the enviorment is cached so it will only happen once each time it needs to scale up. Are there any other issues? El viernes, 26 de febrero

Re: Improving django.setup() and app loading performance

Hi Rich, On 26 févr. 2016, at 16:36, Rich Jones wrote: > I'd love to discuss ideas for performance optimizations to Django's setup > method. In my opinion, the first concrete step would be to measure how much time is spent executing Django code rather than importing Python modules. That’s t

Re: Improving django.setup() and app loading performance

That might be true in theory.. but in practice? Do any of the core/contrib/top100 apps actually depend on loading orders? I've never encountered that problem before, but I don't know. It seems like we could easily get a 10x performance improvement by doing this in parallel by default, and have

Re: Improving django.setup() and app loading performance

I'm not sure we can load apps in parallel -- they are loaded in the order in which they appear in INSTALLED_APPS so the process is consistent. Maybe it would work for some setups, but I guess it would introduce subtle bugs in others. https://docs.djangoproject.com/es/stable/ref/applications/#ho

Improving django.setup() and app loading performance

(Originally posted this as a ticket, but we can have discussion here and leave the ticket for just more specific discussion.) I imagine that this is an area that hasn't really been given much consideration with regards to optimization, because it is