Graham, Going to make that change, monitor and keep this chat updated with the result.
2 Questions: 1) The option request-timeout = 60 is included inside the virtual host along with the rest of the Daemon code right ? 2) *Under no traffic*, do you have any idea of why this problem could happen? As I explained, it is usually, but not always, preceded by couple of GET Request from a random IP (bot requests) to random urls. My assumption was Slow DDOS and this is why I enabled modreqtimeout, mod security and mod qos. But at this point I'm clueless of how to diagnose. Thanks Juan Khawly On Tuesday, September 6, 2022 at 6:04:39 PM UTC-4 Graham Dumpleton wrote: > Sorry, seems I didn't see your update. > > Add an option: > > request-timeout=60 > > to the WSGIDaemonProcess. > > Set the value (in seconds) greater than you would expect your HTTP > requests to normally run. > > What will happen is that when the average running time for all possible > concurrent requests exceeds that timeout value, the daemon process will be > forcibly restarted. This will have the effect of unblocking the process and > a new one will be started in its place. So acts as a fail safe to ensure > your application keeps running. > > What this will also do is attempt to dump out Python stack traces for what > all the request handler threads were doing when the process is restarted. > This will hopefully allow you to work out why your request handlers are > getting stuck, be it they are getting stuck on a lock, or waiting on a > backend service. > > In short, your request handlers are getting stuck and not completing. Over > time these are building up and the thread pool for handling requests is > exhausted and so the process stops handling requests. > > Graham > > On 7 Sep 2022, at 5:44 am, Juan Khawly <[email protected]> wrote: > > Any ideas? > > Thanks > > On Friday, September 2, 2022 at 8:47:47 AM UTC-4 Juan Khawly wrote: > >> Hello Graham, >> >> I'm going to try to address your questions: >> >> *Inside my Virtual Host* >> >> Alias /static /data/home/user/project/frontend/build/static >> <Directory /data/home/user/project/frontend/build/static> >> Require all granted >> </Directory> >> >> <Directory /data/home/user/project/my_project> >> <Files wsgi.py> >> Require all granted >> </Files> >> </Directory> >> >> WSGIScriptAlias / /data/home/user/project/my_project/wsgi.py >> WSGIDaemonProcess my_project python-path=/data/home/user/project >> python-home=/data/home/user/environment/venv >> WSGIProcessGroup my_project >> >> >> *Inside apache2.conf* >> >> WSGIApplicationGroup %{GLOBAL} >> >> *On the apache/error.log* >> When I get the 503 on the access.log, these are the types of errors seen >> on the error.log >> >> *One type of error* >> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >> 140518453380864] [client 118.126.82.157:37722] Timeout when reading >> response headers from daemon process 'my_project': >> /data/home/project/my_project/my_project/wsgi.py >> >> *Another type of error* >> [Thu Sep 01 04:27:00.053558 2022] [wsgi:error] [pid 3267:tid >> 140518595991296] (11)Resource temporarily unavailable: [client >> 172.31.17.102:31880] mod_wsgi (pid=3267): Unable to connect to WSGI >> daemon process ' my_project ' on '/var/run/apache2/wsgi.2385.1.1.sock' >> after multiple attempts as listener backlog limit was exceeded or the >> socket does not exist. >> >> >> >> Thanks, >> Juan Khawly >> >> On Thursday, September 1, 2022 at 5:54:43 PM UTC-4 Graham Dumpleton wrote: >> >>> Would need to see the mod_wsgi configuration you are using to configure >>> the WSGI application, including how WSGIDaemonProcess is configured and >>> whether you are using WSGIApplicationGroup. Also, what errors are in the >>> Apache error log when the 503 errors occur. >>> >>> On 2 Sep 2022, at 4:57 am, Juan Khawly <[email protected]> wrote: >>> >>> Hello, >>> >>> I've been running into this problem for a while. >>> >>> *CONTEXT * >>> >>> I have an application developed in python (3.10), django 4.0.3, using >>> mod_wsgi and apache. The application is in a DEV environment and hosted in >>> AWS EC2. Currently, it does not receive traffic at all. >>> >>> *Installation of Mod WSGI* >>> apt-get install -y apache2-dev >>> >>> *Setup out of the VENV* >>> mod_wsgi-express install-module >>> >>> editing: /etc/apache2/mods-available/wsgi.load >>> >>> LoadModule wsgi_module "/usr/lib/apache2/modules/ >>> mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so" >>> WSGIPythonHome "/data/home/user/environment/venv" >>> >>> *Module Enabled* >>> a2enmod wsgi >>> >>> *PROBLEM* >>> >>> The application works perfect most of the time. Couple of times a week, >>> without traffic the apache server goes down into 503. Usually it is >>> preceded by a random request but it does not always happen that way. I am >>> assuming that is Slow DDOS but I want to make sure it is not miss >>> configuration of the WSGI. >>> >>> access.log example >>> >>> <access.PNG> >>> >>> error.log example >>> I masked the internal routes >>> >>> *This is one of the errors:* >>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading >>> response headers from daemon process 'XXXXX': >>> /XXX/XXXX/XXXXX/XXXXX/XXXXXX/wsgi.py >>> >>> *Another type of error:* >>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading >>> response headers from daemon process 'XXXXXXX': >>> /XXX/XXXX/XXXXX/XXXXXX/XXXXXXX/wsgi.py >>> >>> *SOLUTION* >>> >>> If I restart the server, all works again until next failure. >>> >>> I've enabled the following modules, in case it is SlowDDOS >>> modreqtimeout >>> libapache2-mod-qos >>> libapache2-mod-security2. >>> >>> Any recommendation? >>> >>> Thanks, >>> Juan Khawly >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> <access.PNG> >>> >>> >>> > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com.
