Graham,
Thanks for the deep explanation. This was really helpful to understand
better the architecture.
In my case, my PROD and DEV applications are in separate machines. I don't
think I have to create multiple daemon processes.
This is what I settled down:
1) Use /etc/environment file to setup my environment variables. There are
other processes living in the server that will need to use this file, not
only the DJANGO APP. I dont like the idea to create a separate wsgi_dev,
wsgi_prod and duplicate the vars.
2) Modify the wsgi.py to:
a) Call application = get_wsgi_application() only one time.
b) Created a small routine that opens the previous file and reads the
VARS to store them ONE TIME at os.environ['VAR1'], os.environ['VAR2']
c) Remove VARS from the virtual host since I wont be able to pass them to
the DJANGO App.
I will report back if something goes wrong, but thanks for your time and
help
Juan Khawly
On Saturday, September 10, 2022 at 12:25:15 PM UTC-4 Juan Khawly wrote:
> Graham,
>
> If I take the the get_wsgi_application() out of the function, my site
> throws error an internal error. Looking inside the logs, it all comes down
> that my app is not receiving the value of VAR1 or VAR2 that are further
> used in my settings.py to setup the environment. VAR1 and VAR2 are
> environment variables that help me recognize if I am in LOCAL, DEV or PROD.
>
> Also, I honestly didn't understand this comment if you can help me see
> more clearly. *"Be aware that what you are doing is not technically safe
> because environment variables are global and so if different URLs ended up
> in different values for the variables, then a different request could
> change it before you got to use it." *
>
> My goal is having environmental variables inside my virtual host and pass
> them to the Django app. I tried this using the environment variables files
> that Linux uses but I was never successful to pass them to my Django app.
>
> Thanks,
> Juan Khawly
>
>
>
>
> On Friday, September 9, 2022 at 9:57:06 PM UTC-4 Graham Dumpleton wrote:
>
>> Be aware that what you are doing is not technically safe because
>> environment variables are global and so if different URLs ended up in
>> different values for the variables, then a different request could change
>> it before you got to use it.
>>
>> That said, use:
>>
>> import os
>> from django.core.wsgi import get_wsgi_application
>> os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' project .settings')
>>
>> _application = get_wsgi_application()
>>
>> def application(environ, start_response):
>> os.environ['VAR1'] = environ.get('VAR1', '')
>> os.environ['VAR2'] = environ.get('VAR2', '')
>> return _application(environ, start_response)
>>
>> In other words, just call get_wsgi_application() once outside of the
>> function.
>>
>> Graham
>>
>> On 10 Sep 2022, at 11:36 am, Juan Khawly <[email protected]> wrote:
>>
>> Graham,
>>
>> It does click now. Couple of months ago, I had to modify the original
>> wsgi.py file to include Environmental variables read from my Virtual Host.
>>
>> __________________________________________________________________________
>> *I changed from: *
>>
>> *On wsgi.py*
>>
>> import os
>> from django.core.wsgi import get_wsgi_application
>> os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
>> application = get_wsgi_application()
>>
>> *to*
>>
>> *On Virtual Host*
>>
>> SetEnv VAR1 xxxx
>> SetEnv VAR2 yyyy
>>
>> *On wsgi.py*
>>
>> import os
>> from django.core.wsgi import get_wsgi_application
>> os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' project .settings')
>>
>> def application(environ, start_response):
>> os.environ['VAR1'] = environ.get('VAR1', '')
>> os.environ['VAR2'] = environ.get('VAR2', '')
>> _application = get_wsgi_application()
>> return _application(environ, start_response)
>> __________________________________________________________________________
>>
>> I think I found this solution on a forum and it worked and never expected
>> that it would yield on such consequences.
>>
>> Do you have any suggestion on the right way to do this ? I remember
>> testing multiple options and this was the only one that worked.
>>
>> Thanks,
>> Juan Khawly
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Friday, September 9, 2022 at 4:46:42 PM UTC-4 Graham Dumpleton wrote:
>>
>>> As logs show, you have a problem with thread locking related to logging
>>> subsystem of Python.
>>>
>>> What do you have in your wsgi.py file?
>>>
>>> The messages suggest you are calling Django's get_wsgi_application() on
>>> every request, which is a bad idea. It should only be called once at top
>>> level scope in wsgi.py, not in a request handler function.
>>>
>>> Graham
>>>
>>> On 10 Sep 2022, at 2:41 am, Juan Khawly <[email protected]> wrote:
>>>
>>> Graham,
>>>
>>> After adding the timeout, as you said, the server auto recovers from the
>>> problem.
>>>
>>> After mod_wsgi is logging info level. My error log now gives traces of
>>> where the problem is. I'm attaching my error.log from today (sep 09), any
>>> ideas?
>>>
>>> Thanks for the support.
>>> Juan Khawly
>>>
>>> On Wednesday, September 7, 2022 at 8:59:22 PM UTC-4 Juan Khawly wrote:
>>>
>>>> Makes total sense.
>>>>
>>>> Just added the option to the DaemonProcess and LogLevel info to the
>>>> virtual host. I will be monitoring the the logs and report back in a
>>>> couple
>>>> days for reference.
>>>>
>>>> Appreciate your help.
>>>> Juan Khawly
>>>>
>>>> On Wednesday, September 7, 2022 at 6:30:17 PM UTC-4 Graham Dumpleton
>>>> wrote:
>>>>
>>>>>
>>>>> On 7 Sep 2022, at 11:43 pm, Juan Khawly <[email protected]> wrote:
>>>>>
>>>>> Graham,
>>>>>
>>>>> Going to make that change, monitor and keep this chat updated with the
>>>>> result.
>>>>>
>>>>> 2 Questions:
>>>>>
>>>>> 1) The option request-timeout = 60 is included inside the virtual host
>>>>> along with the rest of the Daemon code right ?
>>>>>
>>>>>
>>>>> It is an option to be added to the existing WSGIDaemonProcess
>>>>> directive.
>>>>>
>>>>> 2) *Under no traffic*, do you have any idea of why this problem could
>>>>> happen? As I explained, it is usually, but not always, preceded by couple
>>>>> of GET Request from a random IP (bot requests) to random urls. My
>>>>> assumption was Slow DDOS and this is why I enabled modreqtimeout, mod
>>>>> security and mod qos. But at this point I'm clueless of how to diagnose.
>>>>>
>>>>>
>>>>> No idea. If it was truly a slow DDOS attack the request wouldn't
>>>>> actually show in the access logs because Apache only logs requests on
>>>>> completion. So am not sure one could say is related to those other
>>>>> requests. I would say it is more likely that over time a trickle of
>>>>> requests come in to your application as normal which block and slowly use
>>>>> up capacity. Hopefully the stack trace created when get a forced restart
>>>>> due to request timeout will show where. Just keep in mind that since the
>>>>> request timeout will cause auto recovery you may not notice it occurred,
>>>>> so
>>>>> you will need to periodically check Apache error logs yourself. Make sure
>>>>> that have info LogLevel for the virtual host so get more useful
>>>>> information
>>>>> out of mod_wsgi.
>>>>>
>>>>> Thanks
>>>>> Juan Khawly
>>>>>
>>>>> On Tuesday, September 6, 2022 at 6:04:39 PM UTC-4 Graham Dumpleton
>>>>> wrote:
>>>>>
>>>>>> Sorry, seems I didn't see your update.
>>>>>>
>>>>>> Add an option:
>>>>>>
>>>>>> request-timeout=60
>>>>>>
>>>>>> to the WSGIDaemonProcess.
>>>>>>
>>>>>> Set the value (in seconds) greater than you would expect your HTTP
>>>>>> requests to normally run.
>>>>>>
>>>>>> What will happen is that when the average running time for all
>>>>>> possible concurrent requests exceeds that timeout value, the daemon
>>>>>> process
>>>>>> will be forcibly restarted. This will have the effect of unblocking the
>>>>>> process and a new one will be started in its place. So acts as a fail
>>>>>> safe
>>>>>> to ensure your application keeps running.
>>>>>>
>>>>>> What this will also do is attempt to dump out Python stack traces for
>>>>>> what all the request handler threads were doing when the process is
>>>>>> restarted. This will hopefully allow you to work out why your request
>>>>>> handlers are getting stuck, be it they are getting stuck on a lock, or
>>>>>> waiting on a backend service.
>>>>>>
>>>>>> In short, your request handlers are getting stuck and not completing.
>>>>>> Over time these are building up and the thread pool for handling
>>>>>> requests
>>>>>> is exhausted and so the process stops handling requests.
>>>>>>
>>>>>> Graham
>>>>>>
>>>>>> On 7 Sep 2022, at 5:44 am, Juan Khawly <[email protected]> wrote:
>>>>>>
>>>>>> Any ideas?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Friday, September 2, 2022 at 8:47:47 AM UTC-4 Juan Khawly wrote:
>>>>>>
>>>>>>> Hello Graham,
>>>>>>>
>>>>>>> I'm going to try to address your questions:
>>>>>>>
>>>>>>> *Inside my Virtual Host*
>>>>>>>
>>>>>>> Alias /static /data/home/user/project/frontend/build/static
>>>>>>> <Directory /data/home/user/project/frontend/build/static>
>>>>>>> Require all granted
>>>>>>> </Directory>
>>>>>>>
>>>>>>> <Directory /data/home/user/project/my_project>
>>>>>>> <Files wsgi.py>
>>>>>>> Require all granted
>>>>>>> </Files>
>>>>>>> </Directory>
>>>>>>>
>>>>>>> WSGIScriptAlias / /data/home/user/project/my_project/wsgi.py
>>>>>>> WSGIDaemonProcess my_project
>>>>>>> python-path=/data/home/user/project
>>>>>>> python-home=/data/home/user/environment/venv
>>>>>>> WSGIProcessGroup my_project
>>>>>>>
>>>>>>>
>>>>>>> *Inside apache2.conf*
>>>>>>>
>>>>>>> WSGIApplicationGroup %{GLOBAL}
>>>>>>>
>>>>>>> *On the apache/error.log*
>>>>>>> When I get the 503 on the access.log, these are the types of errors
>>>>>>> seen on the error.log
>>>>>>>
>>>>>>> *One type of error*
>>>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>>>>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading
>>>>>>> response headers from daemon process 'my_project':
>>>>>>> /data/home/project/my_project/my_project/wsgi.py
>>>>>>>
>>>>>>> *Another type of error*
>>>>>>> [Thu Sep 01 04:27:00.053558 2022] [wsgi:error] [pid 3267:tid
>>>>>>> 140518595991296] (11)Resource temporarily unavailable: [client
>>>>>>> 172.31.17.102:31880] mod_wsgi (pid=3267): Unable to connect to WSGI
>>>>>>> daemon process ' my_project ' on '/var/run/apache2/wsgi.2385.1.1.sock'
>>>>>>> after multiple attempts as listener backlog limit was exceeded or the
>>>>>>> socket does not exist.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Juan Khawly
>>>>>>>
>>>>>>> On Thursday, September 1, 2022 at 5:54:43 PM UTC-4 Graham Dumpleton
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Would need to see the mod_wsgi configuration you are using to
>>>>>>>> configure the WSGI application, including how WSGIDaemonProcess is
>>>>>>>> configured and whether you are using WSGIApplicationGroup. Also, what
>>>>>>>> errors are in the Apache error log when the 503 errors occur.
>>>>>>>>
>>>>>>>> On 2 Sep 2022, at 4:57 am, Juan Khawly <[email protected]> wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I've been running into this problem for a while.
>>>>>>>>
>>>>>>>> *CONTEXT *
>>>>>>>>
>>>>>>>> I have an application developed in python (3.10), django 4.0.3,
>>>>>>>> using mod_wsgi and apache. The application is in a DEV environment and
>>>>>>>> hosted in AWS EC2. Currently, it does not receive traffic at all.
>>>>>>>>
>>>>>>>> *Installation of Mod WSGI*
>>>>>>>> apt-get install -y apache2-dev
>>>>>>>>
>>>>>>>> *Setup out of the VENV*
>>>>>>>> mod_wsgi-express install-module
>>>>>>>>
>>>>>>>> editing: /etc/apache2/mods-available/wsgi.load
>>>>>>>>
>>>>>>>> LoadModule wsgi_module "/usr/lib/apache2/modules/
>>>>>>>> mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so"
>>>>>>>> WSGIPythonHome "/data/home/user/environment/venv"
>>>>>>>>
>>>>>>>> *Module Enabled*
>>>>>>>> a2enmod wsgi
>>>>>>>>
>>>>>>>> *PROBLEM*
>>>>>>>>
>>>>>>>> The application works perfect most of the time. Couple of times a
>>>>>>>> week, without traffic the apache server goes down into 503. Usually it
>>>>>>>> is
>>>>>>>> preceded by a random request but it does not always happen that way. I
>>>>>>>> am
>>>>>>>> assuming that is Slow DDOS but I want to make sure it is not miss
>>>>>>>> configuration of the WSGI.
>>>>>>>>
>>>>>>>> access.log example
>>>>>>>>
>>>>>>>> <access.PNG>
>>>>>>>>
>>>>>>>> error.log example
>>>>>>>> I masked the internal routes
>>>>>>>>
>>>>>>>> *This is one of the errors:*
>>>>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>>>>>> 140518453380864] [client 118.126.82.157:37722] Timeout when
>>>>>>>> reading response headers from daemon process 'XXXXX':
>>>>>>>> /XXX/XXXX/XXXXX/XXXXX/XXXXXX/wsgi.py
>>>>>>>>
>>>>>>>> *Another type of error:*
>>>>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>>>>>> 140518453380864] [client 118.126.82.157:37722] Timeout when
>>>>>>>> reading response headers from daemon process 'XXXXXXX':
>>>>>>>> /XXX/XXXX/XXXXX/XXXXXX/XXXXXXX/wsgi.py
>>>>>>>>
>>>>>>>> *SOLUTION*
>>>>>>>>
>>>>>>>> If I restart the server, all works again until next failure.
>>>>>>>>
>>>>>>>> I've enabled the following modules, in case it is SlowDDOS
>>>>>>>> modreqtimeout
>>>>>>>> libapache2-mod-qos
>>>>>>>> libapache2-mod-security2.
>>>>>>>>
>>>>>>>> Any recommendation?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Juan Khawly
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "modwsgi" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to [email protected].
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com
>>>>>>>>
>>>>>>>> <https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>> <access.PNG>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "modwsgi" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to [email protected].
>>>>>>
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "modwsgi" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>>
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>>
>>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> <error.log>
>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/modwsgi/a69d70f4-faac-4515-b991-90590efd7b95n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/modwsgi/a69d70f4-faac-4515-b991-90590efd7b95n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/modwsgi/e77b2844-6948-457b-8a13-65ef7698c377n%40googlegroups.com.
