As logs show, you have a problem with thread locking related to logging subsystem of Python.
What do you have in your wsgi.py file? The messages suggest you are calling Django's get_wsgi_application() on every request, which is a bad idea. It should only be called once at top level scope in wsgi.py, not in a request handler function. Graham > On 10 Sep 2022, at 2:41 am, Juan Khawly <[email protected]> wrote: > > Graham, > > After adding the timeout, as you said, the server auto recovers from the > problem. > > After mod_wsgi is logging info level. My error log now gives traces of where > the problem is. I'm attaching my error.log from today (sep 09), any ideas? > > Thanks for the support. > Juan Khawly > > On Wednesday, September 7, 2022 at 8:59:22 PM UTC-4 Juan Khawly wrote: > Makes total sense. > > Just added the option to the DaemonProcess and LogLevel info to the virtual > host. I will be monitoring the the logs and report back in a couple days for > reference. > > Appreciate your help. > Juan Khawly > > On Wednesday, September 7, 2022 at 6:30:17 PM UTC-4 Graham Dumpleton wrote: > >> On 7 Sep 2022, at 11:43 pm, Juan Khawly <[email protected] <>> wrote: >> >> Graham, >> >> Going to make that change, monitor and keep this chat updated with the >> result. >> >> 2 Questions: >> >> 1) The option request-timeout = 60 is included inside the virtual host along >> with the rest of the Daemon code right ? > > It is an option to be added to the existing WSGIDaemonProcess directive. > >> 2) Under no traffic, do you have any idea of why this problem could happen? >> As I explained, it is usually, but not always, preceded by couple of GET >> Request from a random IP (bot requests) to random urls. My assumption was >> Slow DDOS and this is why I enabled modreqtimeout, mod security and mod qos. >> But at this point I'm clueless of how to diagnose. > > No idea. If it was truly a slow DDOS attack the request wouldn't actually > show in the access logs because Apache only logs requests on completion. So > am not sure one could say is related to those other requests. I would say it > is more likely that over time a trickle of requests come in to your > application as normal which block and slowly use up capacity. Hopefully the > stack trace created when get a forced restart due to request timeout will > show where. Just keep in mind that since the request timeout will cause auto > recovery you may not notice it occurred, so you will need to periodically > check Apache error logs yourself. Make sure that have info LogLevel for the > virtual host so get more useful information out of mod_wsgi. > > >> Thanks >> Juan Khawly >> >> On Tuesday, September 6, 2022 at 6:04:39 PM UTC-4 Graham Dumpleton wrote: >> Sorry, seems I didn't see your update. >> >> Add an option: >> >> request-timeout=60 >> >> to the WSGIDaemonProcess. >> >> Set the value (in seconds) greater than you would expect your HTTP requests >> to normally run. >> >> What will happen is that when the average running time for all possible >> concurrent requests exceeds that timeout value, the daemon process will be >> forcibly restarted. This will have the effect of unblocking the process and >> a new one will be started in its place. So acts as a fail safe to ensure >> your application keeps running. >> >> What this will also do is attempt to dump out Python stack traces for what >> all the request handler threads were doing when the process is restarted. >> This will hopefully allow you to work out why your request handlers are >> getting stuck, be it they are getting stuck on a lock, or waiting on a >> backend service. >> >> In short, your request handlers are getting stuck and not completing. Over >> time these are building up and the thread pool for handling requests is >> exhausted and so the process stops handling requests. >> >> Graham >> >> >>> On 7 Sep 2022, at 5:44 am, Juan Khawly <[email protected] <>> wrote: >>> >> >>> Any ideas? >>> >>> Thanks >>> >>> On Friday, September 2, 2022 at 8:47:47 AM UTC-4 Juan Khawly wrote: >>> Hello Graham, >>> >>> I'm going to try to address your questions: >>> >>> Inside my Virtual Host >>> >>> Alias /static /data/home/user/project/frontend/build/static >>> <Directory /data/home/user/project/frontend/build/static> >>> Require all granted >>> </Directory> >>> >>> <Directory /data/home/user/project/my_project> >>> <Files wsgi.py> >>> Require all granted >>> </Files> >>> </Directory> >>> >>> WSGIScriptAlias / /data/home/user/project/my_project/wsgi.py >>> WSGIDaemonProcess my_project python-path=/data/home/user/project >>> python-home=/data/home/user/environment/venv >>> WSGIProcessGroup my_project >>> >>> >>> Inside apache2.conf >>> >>> WSGIApplicationGroup %{GLOBAL} >>> >>> On the apache/error.log >>> When I get the 503 on the access.log, these are the types of errors seen on >>> the error.log >>> >>> One type of error >>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >>> 140518453380864] [client 118.126.82.157:37722 >>> <http://118.126.82.157:37722/>] Timeout when reading response headers from >>> daemon process 'my_project': >>> /data/home/project/my_project/my_project/wsgi.py >>> >>> Another type of error >>> [Thu Sep 01 04:27:00.053558 2022] [wsgi:error] [pid 3267:tid >>> 140518595991296] (11)Resource temporarily unavailable: [client >>> 172.31.17.102:31880 <http://172.31.17.102:31880/>] mod_wsgi (pid=3267): >>> Unable to connect to WSGI daemon process ' my_project ' on >>> '/var/run/apache2/wsgi.2385.1.1.sock' after multiple attempts as listener >>> backlog limit was exceeded or the socket does not exist. >>> >>> >>> >>> Thanks, >>> Juan Khawly >>> >>> On Thursday, September 1, 2022 at 5:54:43 PM UTC-4 Graham Dumpleton wrote: >>> Would need to see the mod_wsgi configuration you are using to configure the >>> WSGI application, including how WSGIDaemonProcess is configured and whether >>> you are using WSGIApplicationGroup. Also, what errors are in the Apache >>> error log when the 503 errors occur. >>> >>> >>>> On 2 Sep 2022, at 4:57 am, Juan Khawly <juan...@ <>gmail.com >>>> <http://gmail.com/>> wrote: >>>> >>> >>>> Hello, >>>> >>>> I've been running into this problem for a while. >>>> >>>> CONTEXT >>>> >>>> I have an application developed in python (3.10), django 4.0.3, using >>>> mod_wsgi and apache. The application is in a DEV environment and hosted in >>>> AWS EC2. Currently, it does not receive traffic at all. >>>> >>>> Installation of Mod WSGI >>>> apt-get install -y apache2-dev >>>> >>>> Setup out of the VENV >>>> mod_wsgi-express install-module >>>> >>>> editing: /etc/apache2/mods-available/wsgi.load >>>> >>>> LoadModule wsgi_module >>>> "/usr/lib/apache2/modules/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so >>>> <http://mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so/>" >>>> WSGIPythonHome "/data/home/user/environment/venv" >>>> >>>> Module Enabled >>>> a2enmod wsgi >>>> >>>> PROBLEM >>>> >>>> The application works perfect most of the time. Couple of times a week, >>>> without traffic the apache server goes down into 503. Usually it is >>>> preceded by a random request but it does not always happen that way. I am >>>> assuming that is Slow DDOS but I want to make sure it is not miss >>>> configuration of the WSGI. >>>> >>>> access.log example >>> >>>> <access.PNG> >>>> >>>> error.log example >>>> I masked the internal routes >>>> >>>> This is one of the errors: >>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >>>> 140518453380864] [client 118.126.82.157:37722 >>>> <http://118.126.82.157:37722/>] Timeout when reading response headers from >>>> daemon process 'XXXXX': /XXX/XXXX/XXXXX/XXXXX/XXXXXX/wsgi.py >>>> >>>> Another type of error: >>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >>>> 140518453380864] [client 118.126.82.157:37722 >>>> <http://118.126.82.157:37722/>] Timeout when reading response headers from >>>> daemon process 'XXXXXXX': /XXX/XXXX/XXXXX/XXXXXX/XXXXXXX/wsgi.py >>>> >>>> SOLUTION >>>> >>>> If I restart the server, all works again until next failure. >>>> >>>> I've enabled the following modules, in case it is SlowDDOS >>>> modreqtimeout >>>> libapache2-mod-qos >>>> libapache2-mod-security2. >>>> >>>> Any recommendation? >>>> >>>> Thanks, >>>> Juan Khawly >>>> >>>> >>>> >>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "modwsgi" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to modwsgi+u...@ <>googlegroups.com <http://googlegroups.com/>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>> <access.PNG> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] <>. >> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com?utm_medium=email&utm_source=footer>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <>. > >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com?utm_medium=email&utm_source=footer>. > <error.log> -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/EB906D21-8C50-4AAD-AEA0-0DC3252109FE%40gmail.com.
