That isn't really a solution, it is more of a hack that doesn't really solve the real problem.
If you had watched the the video: https://www.youtube.com/watch?v=SGleKfigMsk&t=593s <https://www.youtube.com/watch?v=SGleKfigMsk&t=593s> I was hoping you would start to understand that the real problem is that you are using the default processes/threads configuration, which is one process with fifteen threads. If your requests all take 2 seconds, then, that means you can only handle 7 1/2 requests per second. What you should be doing is adding the processes option to specify many more processes, and then for those, provided you are only I/O bound and not CPU bound, increasing the number of threads. Thus: WSGIDaemonProcess pe-ta-dev.knowdl.com python-path=/opt/data/knauthor/knauthor processes=10 threads=30 A large number of threads in Python is not great, but if is really I/O bound and not CPU bound, it may be okay. With that configuration, you can know handle 150 requests per second at 2 seconds per request. Do note that increasing the capacity in the daemon process, may mean you have to adjust the MPM settings in Apache for its workers. Also make sure you are not using prefork MPM, use event MPM (or worker MPM if no choice). If the MPM settings have less capacity that processes*threads for daemon process, then the MPM settings would need to be increased to allow more capacity. Overall though, Apache is not going to handle 5000 concurrent connections at 2+ seconds per request. So I question that figure. What is probably really happening is that clients are stuck waiting in the Apache socket listener queue, which can cause big backlog issues when things get overloaded. Important thing is that simply increasing timeouts generally means your configuration is still wrong and not really designed to handle the capacity properly. BTW, your inclusion of '/usr/lib64/python3.4/site-packages' in python-path is also wrong. > On 9 Jan 2019, at 5:56 pm, Mahesh Vernekar <[email protected]> wrote: > > Hi Graham, > > Thanks for your time. > > We resolved the issue by adding the parameter connect-timeout to the > deamon-process configuration as below > > WSGIDaemonProcess pe-ta-dev.knowdl.com <http://pe-ta-dev.knowdl.com/> > python-path=/opt/data/knauthor/knauthor:/usr/lib64/python3.4/site-packages > connect-timeout=300 > > Since our application need to support 5000 concurrent users we kept the time > as 300 secs. > > > Regards > > Mahesh > > > > On Fri, Jan 4, 2019 at 11:58 AM Graham Dumpleton <[email protected] > <mailto:[email protected]>> wrote: > I have been busy today so far so haven't had time to reply. > > Did you watch the video and read the blog post I linked? > > A couple of other videos worth watching to give you better background on > issues and how Apache/mod_wsgi works are: > > * https://www.youtube.com/watch?v=SGleKfigMsk&t=593s > <https://www.youtube.com/watch?v=SGleKfigMsk&t=593s> > > * https://www.youtube.com/watch?v=k6Erh7oHvns > <https://www.youtube.com/watch?v=k6Erh7oHvns> > > Watch the first of those especially and some of the problems with your > configuration should become more obvious. > > Graham > >> On 4 Jan 2019, at 5:24 pm, Mahesh Vernekar <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Graham, >> >> Did you get a chance to go through the apache configuration file ? >> >> Do you need more details ? >> >> >> >> Regards >> >> Mahesh Vernekar >> >> On Thu, Jan 3, 2019 at 3:56 PM Mahesh Vernekar <[email protected] >> <mailto:[email protected]>> wrote: >> Hi Graham, >> >> I have attached the httpd.conf file contents. Will that help ? >> >> >> Regards >> >> Mahesh >> >> On Thu, Jan 3, 2019 at 12:04 PM Graham Dumpleton <[email protected] >> <mailto:[email protected]>> wrote: >> >> >>> On 3 Jan 2019, at 5:26 pm, Mahesh Vernekar <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hi Graham, >>> >>> Thanks for your response. Find my answers as below: >>> >>> What is your current mod_wsgi configuration? We are using deamon process >> >> I need to see the actual configuration, just saying you are using daemon >> mode is not enough as means I have to guess everything, which makes it >> impossible to recommend anything. >> >> That is, I need to see the directives you put in the Apache configuration >> file related to mod_wsgi. This is so I can see if embedded mode is disabled, >> that daemon mode is configured appropriate for processes/threads. Whether >> you set all the recommended timeouts etc etc. >> >> For a bit of background also suggest you watch: >> >> https://www.youtube.com/watch?v=CPz0s1CQsTE >> <https://www.youtube.com/watch?v=CPz0s1CQsTE> >> >> This blog post may also be relevant in upcoming discussion: >> >> http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html >> <http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html> >> >>> How long is typical request response time? Min : 2 secs / Max : 15 secs >> >> That is a very large value for a web server that is dependent on >> processes/threads and not async. >> >>> Are the requests high CPU activities, or I/O bound waiting on a backend >>> service like a database? Though the application is connected to a backend >>> mariadb database there are no high cpu activities. The CPU does reaches 50% >>> during the tests. >> >> Also, have you added any instrumentation your web application to monitoring >> response times and/or mod_wsgi performance? >> >>> >>> >>> Regards >>> >>> Mahesh >>> >>> >>> On Thu, Jan 3, 2019 at 3:06 AM Graham Dumpleton <[email protected] >>> <mailto:[email protected]>> wrote: >>> What is your current mod_wsgi configuration? >>> >>> How long is typical request response time? >>> >>> Are the requests high CPU activities, or I/O bound waiting on a backend >>> service like a database? >>> >>>> On 3 Jan 2019, at 2:12 am, Mahesh Vernekar <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> We are hosting a web application with configuration as below: >>>> Application: Apache 2.4 / Python 3.4 / Django 2.0 >>>> >>>> Database servers x 6 : 32 GB / 8 cores behind load balancer >>>> Web servers x 6 : 4 GB / 2 cores behind load balancer >>>> >>>> Setup is in AWS so we are using AWS-Elastic load balancer. >>>> >>>> The application supports 2000 concurrent connections but failing for 2500 >>>> and beyond. >>>> In load testing around 1800 requests are failing out of 3.5 lakh total >>>> requests. >>>> >>>> In the apache error log we are seeing the error message as below ? >>>> >>>> [Wed Jan 02 14:05:53.349209 2019] [wsgi:error] [pid 21559:tid >>>> 139722408036096] (11)Resource temporarily unavailable: [client >>>> 172.31.12.61:51748 <http://172.31.12.61:51748/>] mod_wsgi (pid=21559): >>>> Unable to connect to WSGI daemon process 'pe-ta-dev.knowdl.com >>>> <http://pe-ta-dev.knowdl.com/>' on '/run/httpd/wsgi.21472.0.1.sock' after >>>> multiple attempts as listener backlog limit was exceeded or the socket >>>> does not exist. >>>> >>>> >>>> Any idea how we can resolve the issue ? >>>> >>>> >>>> >>>> Regards >>>> >>>> Mahesh >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "modwsgi" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected] >>>> <mailto:[email protected]>. >>>> To post to this group, send email to [email protected] >>>> <mailto:[email protected]>. >>>> Visit this group at https://groups.google.com/group/modwsgi >>>> <https://groups.google.com/group/modwsgi>. >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "modwsgi" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/modwsgi/aIcET2VqJbA/unsubscribe >>> <https://groups.google.com/d/topic/modwsgi/aIcET2VqJbA/unsubscribe>. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected] >>> <mailto:[email protected]>. >>> To post to this group, send email to [email protected] >>> <mailto:[email protected]>. >>> Visit this group at https://groups.google.com/group/modwsgi >>> <https://groups.google.com/group/modwsgi>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] >>> <mailto:[email protected]>. >>> To post to this group, send email to [email protected] >>> <mailto:[email protected]>. >>> Visit this group at https://groups.google.com/group/modwsgi >>> <https://groups.google.com/group/modwsgi>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "modwsgi" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/modwsgi/aIcET2VqJbA/unsubscribe >> <https://groups.google.com/d/topic/modwsgi/aIcET2VqJbA/unsubscribe>. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] >> <mailto:[email protected]>. >> To post to this group, send email to [email protected] >> <mailto:[email protected]>. >> Visit this group at https://groups.google.com/group/modwsgi >> <https://groups.google.com/group/modwsgi>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To post to this group, send email to [email protected] >> <mailto:[email protected]>. >> Visit this group at https://groups.google.com/group/modwsgi >> <https://groups.google.com/group/modwsgi>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to a topic in the Google > Groups "modwsgi" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/modwsgi/aIcET2VqJbA/unsubscribe > <https://groups.google.com/d/topic/modwsgi/aIcET2VqJbA/unsubscribe>. > To unsubscribe from this group and all its topics, send an email to > [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/modwsgi > <https://groups.google.com/group/modwsgi>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/modwsgi > <https://groups.google.com/group/modwsgi>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
