> On 7 Sep 2022, at 11:43 pm, Juan Khawly <[email protected]> wrote: > > Graham, > > Going to make that change, monitor and keep this chat updated with the result. > > 2 Questions: > > 1) The option request-timeout = 60 is included inside the virtual host along > with the rest of the Daemon code right ?
It is an option to be added to the existing WSGIDaemonProcess directive. > 2) Under no traffic, do you have any idea of why this problem could happen? > As I explained, it is usually, but not always, preceded by couple of GET > Request from a random IP (bot requests) to random urls. My assumption was > Slow DDOS and this is why I enabled modreqtimeout, mod security and mod qos. > But at this point I'm clueless of how to diagnose. No idea. If it was truly a slow DDOS attack the request wouldn't actually show in the access logs because Apache only logs requests on completion. So am not sure one could say is related to those other requests. I would say it is more likely that over time a trickle of requests come in to your application as normal which block and slowly use up capacity. Hopefully the stack trace created when get a forced restart due to request timeout will show where. Just keep in mind that since the request timeout will cause auto recovery you may not notice it occurred, so you will need to periodically check Apache error logs yourself. Make sure that have info LogLevel for the virtual host so get more useful information out of mod_wsgi. > Thanks > Juan Khawly > > On Tuesday, September 6, 2022 at 6:04:39 PM UTC-4 Graham Dumpleton wrote: > Sorry, seems I didn't see your update. > > Add an option: > > request-timeout=60 > > to the WSGIDaemonProcess. > > Set the value (in seconds) greater than you would expect your HTTP requests > to normally run. > > What will happen is that when the average running time for all possible > concurrent requests exceeds that timeout value, the daemon process will be > forcibly restarted. This will have the effect of unblocking the process and a > new one will be started in its place. So acts as a fail safe to ensure your > application keeps running. > > What this will also do is attempt to dump out Python stack traces for what > all the request handler threads were doing when the process is restarted. > This will hopefully allow you to work out why your request handlers are > getting stuck, be it they are getting stuck on a lock, or waiting on a > backend service. > > In short, your request handlers are getting stuck and not completing. Over > time these are building up and the thread pool for handling requests is > exhausted and so the process stops handling requests. > > Graham > > >> On 7 Sep 2022, at 5:44 am, Juan Khawly <[email protected] >> <applewebdata://83390249-FC68-4765-BC72-810C8A1703EB>> wrote: >> > >> Any ideas? >> >> Thanks >> >> On Friday, September 2, 2022 at 8:47:47 AM UTC-4 Juan Khawly wrote: >> Hello Graham, >> >> I'm going to try to address your questions: >> >> Inside my Virtual Host >> >> Alias /static /data/home/user/project/frontend/build/static >> <Directory /data/home/user/project/frontend/build/static> >> Require all granted >> </Directory> >> >> <Directory /data/home/user/project/my_project> >> <Files wsgi.py> >> Require all granted >> </Files> >> </Directory> >> >> WSGIScriptAlias / /data/home/user/project/my_project/wsgi.py >> WSGIDaemonProcess my_project python-path=/data/home/user/project >> python-home=/data/home/user/environment/venv >> WSGIProcessGroup my_project >> >> >> Inside apache2.conf >> >> WSGIApplicationGroup %{GLOBAL} >> >> On the apache/error.log >> When I get the 503 on the access.log, these are the types of errors seen on >> the error.log >> >> One type of error >> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >> 140518453380864] [client 118.126.82.157:37722 >> <http://118.126.82.157:37722/>] Timeout when reading response headers from >> daemon process 'my_project': /data/home/project/my_project/my_project/wsgi.py >> >> Another type of error >> [Thu Sep 01 04:27:00.053558 2022] [wsgi:error] [pid 3267:tid >> 140518595991296] (11)Resource temporarily unavailable: [client >> 172.31.17.102:31880 <http://172.31.17.102:31880/>] mod_wsgi (pid=3267): >> Unable to connect to WSGI daemon process ' my_project ' on >> '/var/run/apache2/wsgi.2385.1.1.sock' after multiple attempts as listener >> backlog limit was exceeded or the socket does not exist. >> >> >> >> Thanks, >> Juan Khawly >> >> On Thursday, September 1, 2022 at 5:54:43 PM UTC-4 Graham Dumpleton wrote: >> Would need to see the mod_wsgi configuration you are using to configure the >> WSGI application, including how WSGIDaemonProcess is configured and whether >> you are using WSGIApplicationGroup. Also, what errors are in the Apache >> error log when the 503 errors occur. >> >> >>> On 2 Sep 2022, at 4:57 am, Juan Khawly <juan...@ <>gmail.com >>> <http://gmail.com/>> wrote: >>> >> >>> Hello, >>> >>> I've been running into this problem for a while. >>> >>> CONTEXT >>> >>> I have an application developed in python (3.10), django 4.0.3, using >>> mod_wsgi and apache. The application is in a DEV environment and hosted in >>> AWS EC2. Currently, it does not receive traffic at all. >>> >>> Installation of Mod WSGI >>> apt-get install -y apache2-dev >>> >>> Setup out of the VENV >>> mod_wsgi-express install-module >>> >>> editing: /etc/apache2/mods-available/wsgi.load >>> >>> LoadModule wsgi_module >>> "/usr/lib/apache2/modules/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so >>> <http://mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so/>" >>> WSGIPythonHome "/data/home/user/environment/venv" >>> >>> Module Enabled >>> a2enmod wsgi >>> >>> PROBLEM >>> >>> The application works perfect most of the time. Couple of times a week, >>> without traffic the apache server goes down into 503. Usually it is >>> preceded by a random request but it does not always happen that way. I am >>> assuming that is Slow DDOS but I want to make sure it is not miss >>> configuration of the WSGI. >>> >>> access.log example >> >>> <access.PNG> >>> >>> error.log example >>> I masked the internal routes >>> >>> This is one of the errors: >>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >>> 140518453380864] [client 118.126.82.157:37722 >>> <http://118.126.82.157:37722/>] Timeout when reading response headers from >>> daemon process 'XXXXX': /XXX/XXXX/XXXXX/XXXXX/XXXXXX/wsgi.py >>> >>> Another type of error: >>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >>> 140518453380864] [client 118.126.82.157:37722 >>> <http://118.126.82.157:37722/>] Timeout when reading response headers from >>> daemon process 'XXXXXXX': /XXX/XXXX/XXXXX/XXXXXX/XXXXXXX/wsgi.py >>> >>> SOLUTION >>> >>> If I restart the server, all works again until next failure. >>> >>> I've enabled the following modules, in case it is SlowDDOS >>> modreqtimeout >>> libapache2-mod-qos >>> libapache2-mod-security2. >>> >>> Any recommendation? >>> >>> Thanks, >>> Juan Khawly >>> >>> >>> >> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to modwsgi+u...@ <>googlegroups.com <http://googlegroups.com/>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com?utm_medium=email&utm_source=footer>. >>> <access.PNG> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <applewebdata://83390249-FC68-4765-BC72-810C8A1703EB>. > >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/F5D8C58B-4265-45AB-A602-DBD973A0E1D7%40gmail.com.
