> On 11 Sep 2022, at 2:25 am, Juan Khawly <[email protected]> wrote:
>
> Graham,
>
> If I take the the get_wsgi_application() out of the function, my site throws
> error an internal error. Looking inside the logs, it all comes down that my
> app is not receiving the value of VAR1 or VAR2 that are further used in my
> settings.py to setup the environment. VAR1 and VAR2 are environment variables
> that help me recognize if I am in LOCAL, DEV or PROD.
>
> Also, I honestly didn't understand this comment if you can help me see more
> clearly. "Be aware that what you are doing is not technically safe because
> environment variables are global and so if different URLs ended up in
> different values for the variables, then a different request could change it
> before you got to use it."
SetEnv is not technically for setting environment variables. How they are used
is dependent on the specific Apache request handler. For CGI scripts they do
translate to process environment variables since each CGI request is handled in
a separate process invocation. For PHP, mod_wsgi, mod_perl etc, anything set
using SetEnv is specific to the request variables and using them to set
environment variables is not recommended since different contexts in the Apache
configuration could use SetEnv to pass different values for the same variable
name. The consequence of this if a request comes in which has one value for the
variable and sets a process wide environment variable but before it gets to use
that environment variable another parallel request came in which also overrides
the environment variable, but with a different value based on what SetEnv
dictates, then the first request may end up seeing a different value to what
you would expect to see. In other words, in a multi threaded system you will
have race conditions because there is no mutex locking on the usage of process
wide environment variables.
The recommended way of doing things is for different environment types have a
different WSGI script file as entry point. That WSGI script can then set once
any process environment variables before importing the real wsgi.py module for
the project. Do note this still only works where different instances of your
application are delegated to run in different mod_wsgi daemon process groups.
You cannot do this if using embedded mode, or where you have only a single
mod_wsgi daemon process group with multiple instances of your web application.
WSGIDaemonProcess my_project_production
python-path=/data/home/user/project python-home=/data/home/user/environment/venv
WSGIScriptAlias /prod
/data/home/user/project/my_project/wsgi_production.py
process-group=my_project_production application-group=%{GLOBAL}
WSGIDaemonProcess my_project_development
python-path=/data/home/user/project python-home=/data/home/user/environment/venv
WSGIScriptAlias /dev
/data/home/user/project/my_project/wsgi_development.py
process-group=my_project_development application-group=%{GLOBAL}
Since multiple mod_wsgi daemon process groups are going to be required, an
alternative way is to base decisions off the name of the mod_wsgi daemon
process group. A wsgi.py file could thus do:
import mod_wsgi
if mod_wsgi.process_group() == "my_project_production":
...
elif mod_wsgi.process_group() == "my_project_development":
...
The important thing is use different daemon process groups for different
instances of the application using different global configuration, use of
Python sub interpreters for separation is not enough.
> My goal is having environmental variables inside my virtual host and pass
> them to the Django app. I tried this using the environment variables files
> that Linux uses but I was never successful to pass them to my Django app.
If you mean environment variables set in your user login shell environment then
no it will not as Apache runs as a completely different user. If you want to
override environment variables for Apache, they would need to be set in the
systemd/Apache startup scripts.
>
> Thanks,
> Juan Khawly
>
>
>
>
> On Friday, September 9, 2022 at 9:57:06 PM UTC-4 Graham Dumpleton wrote:
> Be aware that what you are doing is not technically safe because environment
> variables are global and so if different URLs ended up in different values
> for the variables, then a different request could change it before you got to
> use it.
>
> That said, use:
>
> import os
> from django.core.wsgi import get_wsgi_application
> os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' project .settings')
>
> _application = get_wsgi_application()
>
> def application(environ, start_response):
> os.environ['VAR1'] = environ.get('VAR1', '')
> os.environ['VAR2'] = environ.get('VAR2', '')
> return _application(environ, start_response)
>
> In other words, just call get_wsgi_application() once outside of the function.
>
> Graham
>
>
>> On 10 Sep 2022, at 11:36 am, Juan Khawly <[email protected]
>> <applewebdata://09BFE633-0485-42F0-85F9-24687F36450C>> wrote:
>>
>
>> Graham,
>>
>> It does click now. Couple of months ago, I had to modify the original
>> wsgi.py file to include Environmental variables read from my Virtual Host.
>>
>> __________________________________________________________________________
>> I changed from:
>>
>> On wsgi.py
>>
>> import os
>> from django.core.wsgi import get_wsgi_application
>> os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
>> application = get_wsgi_application()
>>
>> to
>>
>> On Virtual Host
>>
>> SetEnv VAR1 xxxx
>> SetEnv VAR2 yyyy
>>
>> On wsgi.py
>>
>> import os
>> from django.core.wsgi import get_wsgi_application
>> os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' project .settings')
>>
>> def application(environ, start_response):
>> os.environ['VAR1'] = environ.get('VAR1', '')
>> os.environ['VAR2'] = environ.get('VAR2', '')
>> _application = get_wsgi_application()
>> return _application(environ, start_response)
>> __________________________________________________________________________
>>
>> I think I found this solution on a forum and it worked and never expected
>> that it would yield on such consequences.
>>
>> Do you have any suggestion on the right way to do this ? I remember testing
>> multiple options and this was the only one that worked.
>>
>> Thanks,
>> Juan Khawly
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Friday, September 9, 2022 at 4:46:42 PM UTC-4 Graham Dumpleton wrote:
>> As logs show, you have a problem with thread locking related to logging
>> subsystem of Python.
>>
>> What do you have in your wsgi.py file?
>>
>> The messages suggest you are calling Django's get_wsgi_application() on
>> every request, which is a bad idea. It should only be called once at top
>> level scope in wsgi.py, not in a request handler function.
>>
>> Graham
>>
>>
>>> On 10 Sep 2022, at 2:41 am, Juan Khawly <juan...@ <>gmail.com
>>> <http://gmail.com/>> wrote:
>>>
>>
>>> Graham,
>>>
>>> After adding the timeout, as you said, the server auto recovers from the
>>> problem.
>>>
>>> After mod_wsgi is logging info level. My error log now gives traces of
>>> where the problem is. I'm attaching my error.log from today (sep 09), any
>>> ideas?
>>>
>>> Thanks for the support.
>>> Juan Khawly
>>>
>>> On Wednesday, September 7, 2022 at 8:59:22 PM UTC-4 Juan Khawly wrote:
>>> Makes total sense.
>>>
>>> Just added the option to the DaemonProcess and LogLevel info to the virtual
>>> host. I will be monitoring the the logs and report back in a couple days
>>> for reference.
>>>
>>> Appreciate your help.
>>> Juan Khawly
>>>
>>> On Wednesday, September 7, 2022 at 6:30:17 PM UTC-4 Graham Dumpleton wrote:
>>>
>>>> On 7 Sep 2022, at 11:43 pm, Juan Khawly <juan...@ <>gmail.com
>>>> <http://gmail.com/>> wrote:
>>>>
>>>> Graham,
>>>>
>>>> Going to make that change, monitor and keep this chat updated with the
>>>> result.
>>>>
>>>> 2 Questions:
>>>>
>>>> 1) The option request-timeout = 60 is included inside the virtual host
>>>> along with the rest of the Daemon code right ?
>>>
>>> It is an option to be added to the existing WSGIDaemonProcess directive.
>>>
>>>> 2) Under no traffic, do you have any idea of why this problem could
>>>> happen? As I explained, it is usually, but not always, preceded by couple
>>>> of GET Request from a random IP (bot requests) to random urls. My
>>>> assumption was Slow DDOS and this is why I enabled modreqtimeout, mod
>>>> security and mod qos. But at this point I'm clueless of how to diagnose.
>>>
>>> No idea. If it was truly a slow DDOS attack the request wouldn't actually
>>> show in the access logs because Apache only logs requests on completion. So
>>> am not sure one could say is related to those other requests. I would say
>>> it is more likely that over time a trickle of requests come in to your
>>> application as normal which block and slowly use up capacity. Hopefully the
>>> stack trace created when get a forced restart due to request timeout will
>>> show where. Just keep in mind that since the request timeout will cause
>>> auto recovery you may not notice it occurred, so you will need to
>>> periodically check Apache error logs yourself. Make sure that have info
>>> LogLevel for the virtual host so get more useful information out of
>>> mod_wsgi.
>>>
>>>
>>>> Thanks
>>>> Juan Khawly
>>>>
>>>> On Tuesday, September 6, 2022 at 6:04:39 PM UTC-4 Graham Dumpleton wrote:
>>>> Sorry, seems I didn't see your update.
>>>>
>>>> Add an option:
>>>>
>>>> request-timeout=60
>>>>
>>>> to the WSGIDaemonProcess.
>>>>
>>>> Set the value (in seconds) greater than you would expect your HTTP
>>>> requests to normally run.
>>>>
>>>> What will happen is that when the average running time for all possible
>>>> concurrent requests exceeds that timeout value, the daemon process will be
>>>> forcibly restarted. This will have the effect of unblocking the process
>>>> and a new one will be started in its place. So acts as a fail safe to
>>>> ensure your application keeps running.
>>>>
>>>> What this will also do is attempt to dump out Python stack traces for what
>>>> all the request handler threads were doing when the process is restarted.
>>>> This will hopefully allow you to work out why your request handlers are
>>>> getting stuck, be it they are getting stuck on a lock, or waiting on a
>>>> backend service.
>>>>
>>>> In short, your request handlers are getting stuck and not completing. Over
>>>> time these are building up and the thread pool for handling requests is
>>>> exhausted and so the process stops handling requests.
>>>>
>>>> Graham
>>>>
>>>>
>>>>> On 7 Sep 2022, at 5:44 am, Juan Khawly <juan...@ <>gmail.com
>>>>> <http://gmail.com/>> wrote:
>>>>>
>>>>
>>>>> Any ideas?
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Friday, September 2, 2022 at 8:47:47 AM UTC-4 Juan Khawly wrote:
>>>>> Hello Graham,
>>>>>
>>>>> I'm going to try to address your questions:
>>>>>
>>>>> Inside my Virtual Host
>>>>>
>>>>> Alias /static /data/home/user/project/frontend/build/static
>>>>> <Directory /data/home/user/project/frontend/build/static>
>>>>> Require all granted
>>>>> </Directory>
>>>>>
>>>>> <Directory /data/home/user/project/my_project>
>>>>> <Files wsgi.py>
>>>>> Require all granted
>>>>> </Files>
>>>>> </Directory>
>>>>>
>>>>> WSGIScriptAlias / /data/home/user/project/my_project/wsgi.py
>>>>> WSGIDaemonProcess my_project python-path=/data/home/user/project
>>>>> python-home=/data/home/user/environment/venv
>>>>> WSGIProcessGroup my_project
>>>>>
>>>>>
>>>>> Inside apache2.conf
>>>>>
>>>>> WSGIApplicationGroup %{GLOBAL}
>>>>>
>>>>> On the apache/error.log
>>>>> When I get the 503 on the access.log, these are the types of errors seen
>>>>> on the error.log
>>>>>
>>>>> One type of error
>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>>> 140518453380864] [client 118.126.82.157:37722
>>>>> <http://118.126.82.157:37722/>] Timeout when reading response headers
>>>>> from daemon process 'my_project':
>>>>> /data/home/project/my_project/my_project/wsgi.py
>>>>>
>>>>> Another type of error
>>>>> [Thu Sep 01 04:27:00.053558 2022] [wsgi:error] [pid 3267:tid
>>>>> 140518595991296] (11)Resource temporarily unavailable: [client
>>>>> 172.31.17.102:31880 <http://172.31.17.102:31880/>] mod_wsgi (pid=3267):
>>>>> Unable to connect to WSGI daemon process ' my_project ' on
>>>>> '/var/run/apache2/wsgi.2385.1.1.sock' after multiple attempts as listener
>>>>> backlog limit was exceeded or the socket does not exist.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Juan Khawly
>>>>>
>>>>> On Thursday, September 1, 2022 at 5:54:43 PM UTC-4 Graham Dumpleton wrote:
>>>>> Would need to see the mod_wsgi configuration you are using to configure
>>>>> the WSGI application, including how WSGIDaemonProcess is configured and
>>>>> whether you are using WSGIApplicationGroup. Also, what errors are in the
>>>>> Apache error log when the 503 errors occur.
>>>>>
>>>>>
>>>>>> On 2 Sep 2022, at 4:57 am, Juan Khawly <juan...@ <>gmail.com
>>>>>> <http://gmail.com/>> wrote:
>>>>>>
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I've been running into this problem for a while.
>>>>>>
>>>>>> CONTEXT
>>>>>>
>>>>>> I have an application developed in python (3.10), django 4.0.3, using
>>>>>> mod_wsgi and apache. The application is in a DEV environment and hosted
>>>>>> in AWS EC2. Currently, it does not receive traffic at all.
>>>>>>
>>>>>> Installation of Mod WSGI
>>>>>> apt-get install -y apache2-dev
>>>>>>
>>>>>> Setup out of the VENV
>>>>>> mod_wsgi-express install-module
>>>>>>
>>>>>> editing: /etc/apache2/mods-available/wsgi.load
>>>>>>
>>>>>> LoadModule wsgi_module
>>>>>> "/usr/lib/apache2/modules/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so
>>>>>> <http://mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so/>"
>>>>>> WSGIPythonHome "/data/home/user/environment/venv"
>>>>>>
>>>>>> Module Enabled
>>>>>> a2enmod wsgi
>>>>>>
>>>>>> PROBLEM
>>>>>>
>>>>>> The application works perfect most of the time. Couple of times a week,
>>>>>> without traffic the apache server goes down into 503. Usually it is
>>>>>> preceded by a random request but it does not always happen that way. I
>>>>>> am assuming that is Slow DDOS but I want to make sure it is not miss
>>>>>> configuration of the WSGI.
>>>>>>
>>>>>> access.log example
>>>>>
>>>>>> <access.PNG>
>>>>>>
>>>>>> error.log example
>>>>>> I masked the internal routes
>>>>>>
>>>>>> This is one of the errors:
>>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>>>> 140518453380864] [client 118.126.82.157:37722
>>>>>> <http://118.126.82.157:37722/>] Timeout when reading response headers
>>>>>> from daemon process 'XXXXX': /XXX/XXXX/XXXXX/XXXXX/XXXXXX/wsgi.py
>>>>>>
>>>>>> Another type of error:
>>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>>>> 140518453380864] [client 118.126.82.157:37722
>>>>>> <http://118.126.82.157:37722/>] Timeout when reading response headers
>>>>>> from daemon process 'XXXXXXX': /XXX/XXXX/XXXXX/XXXXXX/XXXXXXX/wsgi.py
>>>>>>
>>>>>> SOLUTION
>>>>>>
>>>>>> If I restart the server, all works again until next failure.
>>>>>>
>>>>>> I've enabled the following modules, in case it is SlowDDOS
>>>>>> modreqtimeout
>>>>>> libapache2-mod-qos
>>>>>> libapache2-mod-security2.
>>>>>>
>>>>>> Any recommendation?
>>>>>>
>>>>>> Thanks,
>>>>>> Juan Khawly
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "modwsgi" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>> an email to modwsgi+u...@ <>googlegroups.com <http://googlegroups.com/>.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>>>>> <access.PNG>
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "modwsgi" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>>> email to modwsgi+u...@ <>googlegroups.com <http://googlegroups.com/>.
>>>>
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "modwsgi" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to modwsgi+u...@ <>googlegroups.com <http://googlegroups.com/>.
>>>
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to modwsgi+u...@ <>googlegroups.com <http://googlegroups.com/>.
>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>> <error.log>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected]
>> <applewebdata://09BFE633-0485-42F0-85F9-24687F36450C>.
>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/modwsgi/a69d70f4-faac-4515-b991-90590efd7b95n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/modwsgi/a69d70f4-faac-4515-b991-90590efd7b95n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected]
> <mailto:[email protected]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/modwsgi/33b4cf84-410f-48d8-a062-4d1ef5e92a50n%40googlegroups.com
>
> <https://groups.google.com/d/msgid/modwsgi/33b4cf84-410f-48d8-a062-4d1ef5e92a50n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/modwsgi/DAF4ED41-48E2-4205-8EDD-6F4A3D92B46D%40gmail.com.
