Graham,
It does click now. Couple of months ago, I had to modify the original
wsgi.py file to include Environmental variables read from my Virtual Host.
__________________________________________________________________________
*I changed from: *
*On wsgi.py*
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
application = get_wsgi_application()
*to*
*On Virtual Host*
SetEnv VAR1 xxxx
SetEnv VAR2 yyyy
*On wsgi.py*
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' project .settings')
def application(environ, start_response):
os.environ['VAR1'] = environ.get('VAR1', '')
os.environ['VAR2'] = environ.get('VAR2', '')
_application = get_wsgi_application()
return _application(environ, start_response)
__________________________________________________________________________
I think I found this solution on a forum and it worked and never expected
that it would yield on such consequences.
Do you have any suggestion on the right way to do this ? I remember testing
multiple options and this was the only one that worked.
Thanks,
Juan Khawly
On Friday, September 9, 2022 at 4:46:42 PM UTC-4 Graham Dumpleton wrote:
> As logs show, you have a problem with thread locking related to logging
> subsystem of Python.
>
> What do you have in your wsgi.py file?
>
> The messages suggest you are calling Django's get_wsgi_application() on
> every request, which is a bad idea. It should only be called once at top
> level scope in wsgi.py, not in a request handler function.
>
> Graham
>
> On 10 Sep 2022, at 2:41 am, Juan Khawly <[email protected]> wrote:
>
> Graham,
>
> After adding the timeout, as you said, the server auto recovers from the
> problem.
>
> After mod_wsgi is logging info level. My error log now gives traces of
> where the problem is. I'm attaching my error.log from today (sep 09), any
> ideas?
>
> Thanks for the support.
> Juan Khawly
>
> On Wednesday, September 7, 2022 at 8:59:22 PM UTC-4 Juan Khawly wrote:
>
>> Makes total sense.
>>
>> Just added the option to the DaemonProcess and LogLevel info to the
>> virtual host. I will be monitoring the the logs and report back in a couple
>> days for reference.
>>
>> Appreciate your help.
>> Juan Khawly
>>
>> On Wednesday, September 7, 2022 at 6:30:17 PM UTC-4 Graham Dumpleton
>> wrote:
>>
>>>
>>> On 7 Sep 2022, at 11:43 pm, Juan Khawly <[email protected]> wrote:
>>>
>>> Graham,
>>>
>>> Going to make that change, monitor and keep this chat updated with the
>>> result.
>>>
>>> 2 Questions:
>>>
>>> 1) The option request-timeout = 60 is included inside the virtual host
>>> along with the rest of the Daemon code right ?
>>>
>>>
>>> It is an option to be added to the existing WSGIDaemonProcess directive.
>>>
>>> 2) *Under no traffic*, do you have any idea of why this problem could
>>> happen? As I explained, it is usually, but not always, preceded by couple
>>> of GET Request from a random IP (bot requests) to random urls. My
>>> assumption was Slow DDOS and this is why I enabled modreqtimeout, mod
>>> security and mod qos. But at this point I'm clueless of how to diagnose.
>>>
>>>
>>> No idea. If it was truly a slow DDOS attack the request wouldn't
>>> actually show in the access logs because Apache only logs requests on
>>> completion. So am not sure one could say is related to those other
>>> requests. I would say it is more likely that over time a trickle of
>>> requests come in to your application as normal which block and slowly use
>>> up capacity. Hopefully the stack trace created when get a forced restart
>>> due to request timeout will show where. Just keep in mind that since the
>>> request timeout will cause auto recovery you may not notice it occurred, so
>>> you will need to periodically check Apache error logs yourself. Make sure
>>> that have info LogLevel for the virtual host so get more useful information
>>> out of mod_wsgi.
>>>
>>> Thanks
>>> Juan Khawly
>>>
>>> On Tuesday, September 6, 2022 at 6:04:39 PM UTC-4 Graham Dumpleton wrote:
>>>
>>>> Sorry, seems I didn't see your update.
>>>>
>>>> Add an option:
>>>>
>>>> request-timeout=60
>>>>
>>>> to the WSGIDaemonProcess.
>>>>
>>>> Set the value (in seconds) greater than you would expect your HTTP
>>>> requests to normally run.
>>>>
>>>> What will happen is that when the average running time for all possible
>>>> concurrent requests exceeds that timeout value, the daemon process will be
>>>> forcibly restarted. This will have the effect of unblocking the process
>>>> and
>>>> a new one will be started in its place. So acts as a fail safe to ensure
>>>> your application keeps running.
>>>>
>>>> What this will also do is attempt to dump out Python stack traces for
>>>> what all the request handler threads were doing when the process is
>>>> restarted. This will hopefully allow you to work out why your request
>>>> handlers are getting stuck, be it they are getting stuck on a lock, or
>>>> waiting on a backend service.
>>>>
>>>> In short, your request handlers are getting stuck and not completing.
>>>> Over time these are building up and the thread pool for handling requests
>>>> is exhausted and so the process stops handling requests.
>>>>
>>>> Graham
>>>>
>>>> On 7 Sep 2022, at 5:44 am, Juan Khawly <[email protected]> wrote:
>>>>
>>>> Any ideas?
>>>>
>>>> Thanks
>>>>
>>>> On Friday, September 2, 2022 at 8:47:47 AM UTC-4 Juan Khawly wrote:
>>>>
>>>>> Hello Graham,
>>>>>
>>>>> I'm going to try to address your questions:
>>>>>
>>>>> *Inside my Virtual Host*
>>>>>
>>>>> Alias /static /data/home/user/project/frontend/build/static
>>>>> <Directory /data/home/user/project/frontend/build/static>
>>>>> Require all granted
>>>>> </Directory>
>>>>>
>>>>> <Directory /data/home/user/project/my_project>
>>>>> <Files wsgi.py>
>>>>> Require all granted
>>>>> </Files>
>>>>> </Directory>
>>>>>
>>>>> WSGIScriptAlias / /data/home/user/project/my_project/wsgi.py
>>>>> WSGIDaemonProcess my_project
>>>>> python-path=/data/home/user/project
>>>>> python-home=/data/home/user/environment/venv
>>>>> WSGIProcessGroup my_project
>>>>>
>>>>>
>>>>> *Inside apache2.conf*
>>>>>
>>>>> WSGIApplicationGroup %{GLOBAL}
>>>>>
>>>>> *On the apache/error.log*
>>>>> When I get the 503 on the access.log, these are the types of errors
>>>>> seen on the error.log
>>>>>
>>>>> *One type of error*
>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading
>>>>> response headers from daemon process 'my_project':
>>>>> /data/home/project/my_project/my_project/wsgi.py
>>>>>
>>>>> *Another type of error*
>>>>> [Thu Sep 01 04:27:00.053558 2022] [wsgi:error] [pid 3267:tid
>>>>> 140518595991296] (11)Resource temporarily unavailable: [client
>>>>> 172.31.17.102:31880] mod_wsgi (pid=3267): Unable to connect to WSGI
>>>>> daemon process ' my_project ' on '/var/run/apache2/wsgi.2385.1.1.sock'
>>>>> after multiple attempts as listener backlog limit was exceeded or the
>>>>> socket does not exist.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Juan Khawly
>>>>>
>>>>> On Thursday, September 1, 2022 at 5:54:43 PM UTC-4 Graham Dumpleton
>>>>> wrote:
>>>>>
>>>>>> Would need to see the mod_wsgi configuration you are using to
>>>>>> configure the WSGI application, including how WSGIDaemonProcess is
>>>>>> configured and whether you are using WSGIApplicationGroup. Also, what
>>>>>> errors are in the Apache error log when the 503 errors occur.
>>>>>>
>>>>>> On 2 Sep 2022, at 4:57 am, Juan Khawly <[email protected]> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I've been running into this problem for a while.
>>>>>>
>>>>>> *CONTEXT *
>>>>>>
>>>>>> I have an application developed in python (3.10), django 4.0.3, using
>>>>>> mod_wsgi and apache. The application is in a DEV environment and hosted
>>>>>> in
>>>>>> AWS EC2. Currently, it does not receive traffic at all.
>>>>>>
>>>>>> *Installation of Mod WSGI*
>>>>>> apt-get install -y apache2-dev
>>>>>>
>>>>>> *Setup out of the VENV*
>>>>>> mod_wsgi-express install-module
>>>>>>
>>>>>> editing: /etc/apache2/mods-available/wsgi.load
>>>>>>
>>>>>> LoadModule wsgi_module "/usr/lib/apache2/modules/
>>>>>> mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so"
>>>>>> WSGIPythonHome "/data/home/user/environment/venv"
>>>>>>
>>>>>> *Module Enabled*
>>>>>> a2enmod wsgi
>>>>>>
>>>>>> *PROBLEM*
>>>>>>
>>>>>> The application works perfect most of the time. Couple of times a
>>>>>> week, without traffic the apache server goes down into 503. Usually it
>>>>>> is
>>>>>> preceded by a random request but it does not always happen that way. I
>>>>>> am
>>>>>> assuming that is Slow DDOS but I want to make sure it is not miss
>>>>>> configuration of the WSGI.
>>>>>>
>>>>>> access.log example
>>>>>>
>>>>>> <access.PNG>
>>>>>>
>>>>>> error.log example
>>>>>> I masked the internal routes
>>>>>>
>>>>>> *This is one of the errors:*
>>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>>>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading
>>>>>> response headers from daemon process 'XXXXX':
>>>>>> /XXX/XXXX/XXXXX/XXXXX/XXXXXX/wsgi.py
>>>>>>
>>>>>> *Another type of error:*
>>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>>>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading
>>>>>> response headers from daemon process 'XXXXXXX':
>>>>>> /XXX/XXXX/XXXXX/XXXXXX/XXXXXXX/wsgi.py
>>>>>>
>>>>>> *SOLUTION*
>>>>>>
>>>>>> If I restart the server, all works again until next failure.
>>>>>>
>>>>>> I've enabled the following modules, in case it is SlowDDOS
>>>>>> modreqtimeout
>>>>>> libapache2-mod-qos
>>>>>> libapache2-mod-security2.
>>>>>>
>>>>>> Any recommendation?
>>>>>>
>>>>>> Thanks,
>>>>>> Juan Khawly
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "modwsgi" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to [email protected].
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>> <access.PNG>
>>>>>>
>>>>>>
>>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "modwsgi" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>>
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>>
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>>
>>>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
>
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com
>
> <https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> <error.log>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/modwsgi/a69d70f4-faac-4515-b991-90590efd7b95n%40googlegroups.com.
