Be aware that what you are doing is not technically safe because environment
variables are global and so if different URLs ended up in different values for
the variables, then a different request could change it before you got to use
it.
That said, use:
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' project .settings')
_application = get_wsgi_application()
def application(environ, start_response):
os.environ['VAR1'] = environ.get('VAR1', '')
os.environ['VAR2'] = environ.get('VAR2', '')
return _application(environ, start_response)
In other words, just call get_wsgi_application() once outside of the function.
Graham
> On 10 Sep 2022, at 11:36 am, Juan Khawly <[email protected]> wrote:
>
> Graham,
>
> It does click now. Couple of months ago, I had to modify the original wsgi.py
> file to include Environmental variables read from my Virtual Host.
>
> __________________________________________________________________________
> I changed from:
>
> On wsgi.py
>
> import os
> from django.core.wsgi import get_wsgi_application
> os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
> application = get_wsgi_application()
>
> to
>
> On Virtual Host
>
> SetEnv VAR1 xxxx
> SetEnv VAR2 yyyy
>
> On wsgi.py
>
> import os
> from django.core.wsgi import get_wsgi_application
> os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' project .settings')
>
> def application(environ, start_response):
> os.environ['VAR1'] = environ.get('VAR1', '')
> os.environ['VAR2'] = environ.get('VAR2', '')
> _application = get_wsgi_application()
> return _application(environ, start_response)
> __________________________________________________________________________
>
> I think I found this solution on a forum and it worked and never expected
> that it would yield on such consequences.
>
> Do you have any suggestion on the right way to do this ? I remember testing
> multiple options and this was the only one that worked.
>
> Thanks,
> Juan Khawly
>
>
>
>
>
>
>
>
>
> On Friday, September 9, 2022 at 4:46:42 PM UTC-4 Graham Dumpleton wrote:
> As logs show, you have a problem with thread locking related to logging
> subsystem of Python.
>
> What do you have in your wsgi.py file?
>
> The messages suggest you are calling Django's get_wsgi_application() on every
> request, which is a bad idea. It should only be called once at top level
> scope in wsgi.py, not in a request handler function.
>
> Graham
>
>
>> On 10 Sep 2022, at 2:41 am, Juan Khawly <[email protected]
>> <applewebdata://6550DADD-C814-4030-B377-69A9432A4E76>> wrote:
>>
>
>> Graham,
>>
>> After adding the timeout, as you said, the server auto recovers from the
>> problem.
>>
>> After mod_wsgi is logging info level. My error log now gives traces of where
>> the problem is. I'm attaching my error.log from today (sep 09), any ideas?
>>
>> Thanks for the support.
>> Juan Khawly
>>
>> On Wednesday, September 7, 2022 at 8:59:22 PM UTC-4 Juan Khawly wrote:
>> Makes total sense.
>>
>> Just added the option to the DaemonProcess and LogLevel info to the virtual
>> host. I will be monitoring the the logs and report back in a couple days for
>> reference.
>>
>> Appreciate your help.
>> Juan Khawly
>>
>> On Wednesday, September 7, 2022 at 6:30:17 PM UTC-4 Graham Dumpleton wrote:
>>
>>> On 7 Sep 2022, at 11:43 pm, Juan Khawly <[email protected] <>> wrote:
>>>
>>> Graham,
>>>
>>> Going to make that change, monitor and keep this chat updated with the
>>> result.
>>>
>>> 2 Questions:
>>>
>>> 1) The option request-timeout = 60 is included inside the virtual host
>>> along with the rest of the Daemon code right ?
>>
>> It is an option to be added to the existing WSGIDaemonProcess directive.
>>
>>> 2) Under no traffic, do you have any idea of why this problem could happen?
>>> As I explained, it is usually, but not always, preceded by couple of GET
>>> Request from a random IP (bot requests) to random urls. My assumption was
>>> Slow DDOS and this is why I enabled modreqtimeout, mod security and mod
>>> qos. But at this point I'm clueless of how to diagnose.
>>
>> No idea. If it was truly a slow DDOS attack the request wouldn't actually
>> show in the access logs because Apache only logs requests on completion. So
>> am not sure one could say is related to those other requests. I would say it
>> is more likely that over time a trickle of requests come in to your
>> application as normal which block and slowly use up capacity. Hopefully the
>> stack trace created when get a forced restart due to request timeout will
>> show where. Just keep in mind that since the request timeout will cause auto
>> recovery you may not notice it occurred, so you will need to periodically
>> check Apache error logs yourself. Make sure that have info LogLevel for the
>> virtual host so get more useful information out of mod_wsgi.
>>
>>
>>> Thanks
>>> Juan Khawly
>>>
>>> On Tuesday, September 6, 2022 at 6:04:39 PM UTC-4 Graham Dumpleton wrote:
>>> Sorry, seems I didn't see your update.
>>>
>>> Add an option:
>>>
>>> request-timeout=60
>>>
>>> to the WSGIDaemonProcess.
>>>
>>> Set the value (in seconds) greater than you would expect your HTTP requests
>>> to normally run.
>>>
>>> What will happen is that when the average running time for all possible
>>> concurrent requests exceeds that timeout value, the daemon process will be
>>> forcibly restarted. This will have the effect of unblocking the process and
>>> a new one will be started in its place. So acts as a fail safe to ensure
>>> your application keeps running.
>>>
>>> What this will also do is attempt to dump out Python stack traces for what
>>> all the request handler threads were doing when the process is restarted.
>>> This will hopefully allow you to work out why your request handlers are
>>> getting stuck, be it they are getting stuck on a lock, or waiting on a
>>> backend service.
>>>
>>> In short, your request handlers are getting stuck and not completing. Over
>>> time these are building up and the thread pool for handling requests is
>>> exhausted and so the process stops handling requests.
>>>
>>> Graham
>>>
>>>
>>>> On 7 Sep 2022, at 5:44 am, Juan Khawly <[email protected] <>> wrote:
>>>>
>>>
>>>> Any ideas?
>>>>
>>>> Thanks
>>>>
>>>> On Friday, September 2, 2022 at 8:47:47 AM UTC-4 Juan Khawly wrote:
>>>> Hello Graham,
>>>>
>>>> I'm going to try to address your questions:
>>>>
>>>> Inside my Virtual Host
>>>>
>>>> Alias /static /data/home/user/project/frontend/build/static
>>>> <Directory /data/home/user/project/frontend/build/static>
>>>> Require all granted
>>>> </Directory>
>>>>
>>>> <Directory /data/home/user/project/my_project>
>>>> <Files wsgi.py>
>>>> Require all granted
>>>> </Files>
>>>> </Directory>
>>>>
>>>> WSGIScriptAlias / /data/home/user/project/my_project/wsgi.py
>>>> WSGIDaemonProcess my_project python-path=/data/home/user/project
>>>> python-home=/data/home/user/environment/venv
>>>> WSGIProcessGroup my_project
>>>>
>>>>
>>>> Inside apache2.conf
>>>>
>>>> WSGIApplicationGroup %{GLOBAL}
>>>>
>>>> On the apache/error.log
>>>> When I get the 503 on the access.log, these are the types of errors seen
>>>> on the error.log
>>>>
>>>> One type of error
>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>> 140518453380864] [client 118.126.82.157:37722
>>>> <http://118.126.82.157:37722/>] Timeout when reading response headers from
>>>> daemon process 'my_project':
>>>> /data/home/project/my_project/my_project/wsgi.py
>>>>
>>>> Another type of error
>>>> [Thu Sep 01 04:27:00.053558 2022] [wsgi:error] [pid 3267:tid
>>>> 140518595991296] (11)Resource temporarily unavailable: [client
>>>> 172.31.17.102:31880 <http://172.31.17.102:31880/>] mod_wsgi (pid=3267):
>>>> Unable to connect to WSGI daemon process ' my_project ' on
>>>> '/var/run/apache2/wsgi.2385.1.1.sock' after multiple attempts as listener
>>>> backlog limit was exceeded or the socket does not exist.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Juan Khawly
>>>>
>>>> On Thursday, September 1, 2022 at 5:54:43 PM UTC-4 Graham Dumpleton wrote:
>>>> Would need to see the mod_wsgi configuration you are using to configure
>>>> the WSGI application, including how WSGIDaemonProcess is configured and
>>>> whether you are using WSGIApplicationGroup. Also, what errors are in the
>>>> Apache error log when the 503 errors occur.
>>>>
>>>>
>>>>> On 2 Sep 2022, at 4:57 am, Juan Khawly <juan...@ <>gmail.com
>>>>> <http://gmail.com/>> wrote:
>>>>>
>>>>
>>>>> Hello,
>>>>>
>>>>> I've been running into this problem for a while.
>>>>>
>>>>> CONTEXT
>>>>>
>>>>> I have an application developed in python (3.10), django 4.0.3, using
>>>>> mod_wsgi and apache. The application is in a DEV environment and hosted
>>>>> in AWS EC2. Currently, it does not receive traffic at all.
>>>>>
>>>>> Installation of Mod WSGI
>>>>> apt-get install -y apache2-dev
>>>>>
>>>>> Setup out of the VENV
>>>>> mod_wsgi-express install-module
>>>>>
>>>>> editing: /etc/apache2/mods-available/wsgi.load
>>>>>
>>>>> LoadModule wsgi_module
>>>>> "/usr/lib/apache2/modules/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so
>>>>> <http://mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so/>"
>>>>> WSGIPythonHome "/data/home/user/environment/venv"
>>>>>
>>>>> Module Enabled
>>>>> a2enmod wsgi
>>>>>
>>>>> PROBLEM
>>>>>
>>>>> The application works perfect most of the time. Couple of times a week,
>>>>> without traffic the apache server goes down into 503. Usually it is
>>>>> preceded by a random request but it does not always happen that way. I am
>>>>> assuming that is Slow DDOS but I want to make sure it is not miss
>>>>> configuration of the WSGI.
>>>>>
>>>>> access.log example
>>>>
>>>>> <access.PNG>
>>>>>
>>>>> error.log example
>>>>> I masked the internal routes
>>>>>
>>>>> This is one of the errors:
>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>>> 140518453380864] [client 118.126.82.157:37722
>>>>> <http://118.126.82.157:37722/>] Timeout when reading response headers
>>>>> from daemon process 'XXXXX': /XXX/XXXX/XXXXX/XXXXX/XXXXXX/wsgi.py
>>>>>
>>>>> Another type of error:
>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>>> 140518453380864] [client 118.126.82.157:37722
>>>>> <http://118.126.82.157:37722/>] Timeout when reading response headers
>>>>> from daemon process 'XXXXXXX': /XXX/XXXX/XXXXX/XXXXXX/XXXXXXX/wsgi.py
>>>>>
>>>>> SOLUTION
>>>>>
>>>>> If I restart the server, all works again until next failure.
>>>>>
>>>>> I've enabled the following modules, in case it is SlowDDOS
>>>>> modreqtimeout
>>>>> libapache2-mod-qos
>>>>> libapache2-mod-security2.
>>>>>
>>>>> Any recommendation?
>>>>>
>>>>> Thanks,
>>>>> Juan Khawly
>>>>>
>>>>>
>>>>>
>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "modwsgi" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>>> email to modwsgi+u...@ <>googlegroups.com <http://googlegroups.com/>.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>>>> <access.PNG>
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "modwsgi" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to [email protected] <>.
>>>
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to [email protected] <>.
>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected]
>> <applewebdata://6550DADD-C814-4030-B377-69A9432A4E76>.
>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com?utm_medium=email&utm_source=footer>.
>> <error.log>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected]
> <mailto:[email protected]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/modwsgi/a69d70f4-faac-4515-b991-90590efd7b95n%40googlegroups.com
>
> <https://groups.google.com/d/msgid/modwsgi/a69d70f4-faac-4515-b991-90590efd7b95n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/modwsgi/ED910F9F-EC68-451F-AE34-1C9F7763D81D%40gmail.com.
