Yeah, that is what people often do. I didn't want to dictate where you stored the config as people do things different.
> On 13 Sep 2022, at 4:56 am, Juan Khawly <[email protected]> wrote: > > Graham, > > Thanks for the deep explanation. This was really helpful to understand better > the architecture. > > In my case, my PROD and DEV applications are in separate machines. I don't > think I have to create multiple daemon processes. > > This is what I settled down: > > 1) Use /etc/environment file to setup my environment variables. There are > other processes living in the server that will need to use this file, not > only the DJANGO APP. I dont like the idea to create a separate wsgi_dev, > wsgi_prod and duplicate the vars. > 2) Modify the wsgi.py to: > a) Call application = get_wsgi_application() only one time. > b) Created a small routine that opens the previous file and reads the VARS > to store them ONE TIME at os.environ['VAR1'], os.environ['VAR2'] > c) Remove VARS from the virtual host since I wont be able to pass them to > the DJANGO App. > > I will report back if something goes wrong, but thanks for your time and help > > Juan Khawly > > > > > > > > > On Saturday, September 10, 2022 at 12:25:15 PM UTC-4 Juan Khawly wrote: > Graham, > > If I take the the get_wsgi_application() out of the function, my site throws > error an internal error. Looking inside the logs, it all comes down that my > app is not receiving the value of VAR1 or VAR2 that are further used in my > settings.py to setup the environment. VAR1 and VAR2 are environment variables > that help me recognize if I am in LOCAL, DEV or PROD. > > Also, I honestly didn't understand this comment if you can help me see more > clearly. "Be aware that what you are doing is not technically safe because > environment variables are global and so if different URLs ended up in > different values for the variables, then a different request could change it > before you got to use it." > > My goal is having environmental variables inside my virtual host and pass > them to the Django app. I tried this using the environment variables files > that Linux uses but I was never successful to pass them to my Django app. > > Thanks, > Juan Khawly > > > > > On Friday, September 9, 2022 at 9:57:06 PM UTC-4 Graham Dumpleton wrote: > Be aware that what you are doing is not technically safe because environment > variables are global and so if different URLs ended up in different values > for the variables, then a different request could change it before you got to > use it. > > That said, use: > > import os > from django.core.wsgi import get_wsgi_application > os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' project .settings') > > _application = get_wsgi_application() > > def application(environ, start_response): > os.environ['VAR1'] = environ.get('VAR1', '') > os.environ['VAR2'] = environ.get('VAR2', '') > return _application(environ, start_response) > > In other words, just call get_wsgi_application() once outside of the function. > > Graham > > >> On 10 Sep 2022, at 11:36 am, Juan Khawly <[email protected] <>> wrote: >> > >> Graham, >> >> It does click now. Couple of months ago, I had to modify the original >> wsgi.py file to include Environmental variables read from my Virtual Host. >> >> __________________________________________________________________________ >> I changed from: >> >> On wsgi.py >> >> import os >> from django.core.wsgi import get_wsgi_application >> os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings') >> application = get_wsgi_application() >> >> to >> >> On Virtual Host >> >> SetEnv VAR1 xxxx >> SetEnv VAR2 yyyy >> >> On wsgi.py >> >> import os >> from django.core.wsgi import get_wsgi_application >> os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' project .settings') >> >> def application(environ, start_response): >> os.environ['VAR1'] = environ.get('VAR1', '') >> os.environ['VAR2'] = environ.get('VAR2', '') >> _application = get_wsgi_application() >> return _application(environ, start_response) >> __________________________________________________________________________ >> >> I think I found this solution on a forum and it worked and never expected >> that it would yield on such consequences. >> >> Do you have any suggestion on the right way to do this ? I remember testing >> multiple options and this was the only one that worked. >> >> Thanks, >> Juan Khawly >> >> >> >> >> >> >> >> >> >> On Friday, September 9, 2022 at 4:46:42 PM UTC-4 Graham Dumpleton wrote: >> As logs show, you have a problem with thread locking related to logging >> subsystem of Python. >> >> What do you have in your wsgi.py file? >> >> The messages suggest you are calling Django's get_wsgi_application() on >> every request, which is a bad idea. It should only be called once at top >> level scope in wsgi.py, not in a request handler function. >> >> Graham >> >> >>> On 10 Sep 2022, at 2:41 am, Juan Khawly <[email protected] <>> wrote: >>> >> >>> Graham, >>> >>> After adding the timeout, as you said, the server auto recovers from the >>> problem. >>> >>> After mod_wsgi is logging info level. My error log now gives traces of >>> where the problem is. I'm attaching my error.log from today (sep 09), any >>> ideas? >>> >>> Thanks for the support. >>> Juan Khawly >>> >>> On Wednesday, September 7, 2022 at 8:59:22 PM UTC-4 Juan Khawly wrote: >>> Makes total sense. >>> >>> Just added the option to the DaemonProcess and LogLevel info to the virtual >>> host. I will be monitoring the the logs and report back in a couple days >>> for reference. >>> >>> Appreciate your help. >>> Juan Khawly >>> >>> On Wednesday, September 7, 2022 at 6:30:17 PM UTC-4 Graham Dumpleton wrote: >>> >>>> On 7 Sep 2022, at 11:43 pm, Juan Khawly <[email protected] <>> wrote: >>>> >>>> Graham, >>>> >>>> Going to make that change, monitor and keep this chat updated with the >>>> result. >>>> >>>> 2 Questions: >>>> >>>> 1) The option request-timeout = 60 is included inside the virtual host >>>> along with the rest of the Daemon code right ? >>> >>> It is an option to be added to the existing WSGIDaemonProcess directive. >>> >>>> 2) Under no traffic, do you have any idea of why this problem could >>>> happen? As I explained, it is usually, but not always, preceded by couple >>>> of GET Request from a random IP (bot requests) to random urls. My >>>> assumption was Slow DDOS and this is why I enabled modreqtimeout, mod >>>> security and mod qos. But at this point I'm clueless of how to diagnose. >>> >>> No idea. If it was truly a slow DDOS attack the request wouldn't actually >>> show in the access logs because Apache only logs requests on completion. So >>> am not sure one could say is related to those other requests. I would say >>> it is more likely that over time a trickle of requests come in to your >>> application as normal which block and slowly use up capacity. Hopefully the >>> stack trace created when get a forced restart due to request timeout will >>> show where. Just keep in mind that since the request timeout will cause >>> auto recovery you may not notice it occurred, so you will need to >>> periodically check Apache error logs yourself. Make sure that have info >>> LogLevel for the virtual host so get more useful information out of >>> mod_wsgi. >>> >>> >>>> Thanks >>>> Juan Khawly >>>> >>>> On Tuesday, September 6, 2022 at 6:04:39 PM UTC-4 Graham Dumpleton wrote: >>>> Sorry, seems I didn't see your update. >>>> >>>> Add an option: >>>> >>>> request-timeout=60 >>>> >>>> to the WSGIDaemonProcess. >>>> >>>> Set the value (in seconds) greater than you would expect your HTTP >>>> requests to normally run. >>>> >>>> What will happen is that when the average running time for all possible >>>> concurrent requests exceeds that timeout value, the daemon process will be >>>> forcibly restarted. This will have the effect of unblocking the process >>>> and a new one will be started in its place. So acts as a fail safe to >>>> ensure your application keeps running. >>>> >>>> What this will also do is attempt to dump out Python stack traces for what >>>> all the request handler threads were doing when the process is restarted. >>>> This will hopefully allow you to work out why your request handlers are >>>> getting stuck, be it they are getting stuck on a lock, or waiting on a >>>> backend service. >>>> >>>> In short, your request handlers are getting stuck and not completing. Over >>>> time these are building up and the thread pool for handling requests is >>>> exhausted and so the process stops handling requests. >>>> >>>> Graham >>>> >>>> >>>>> On 7 Sep 2022, at 5:44 am, Juan Khawly <[email protected] <>> wrote: >>>>> >>>> >>>>> Any ideas? >>>>> >>>>> Thanks >>>>> >>>>> On Friday, September 2, 2022 at 8:47:47 AM UTC-4 Juan Khawly wrote: >>>>> Hello Graham, >>>>> >>>>> I'm going to try to address your questions: >>>>> >>>>> Inside my Virtual Host >>>>> >>>>> Alias /static /data/home/user/project/frontend/build/static >>>>> <Directory /data/home/user/project/frontend/build/static> >>>>> Require all granted >>>>> </Directory> >>>>> >>>>> <Directory /data/home/user/project/my_project> >>>>> <Files wsgi.py> >>>>> Require all granted >>>>> </Files> >>>>> </Directory> >>>>> >>>>> WSGIScriptAlias / /data/home/user/project/my_project/wsgi.py >>>>> WSGIDaemonProcess my_project python-path=/data/home/user/project >>>>> python-home=/data/home/user/environment/venv >>>>> WSGIProcessGroup my_project >>>>> >>>>> >>>>> Inside apache2.conf >>>>> >>>>> WSGIApplicationGroup %{GLOBAL} >>>>> >>>>> On the apache/error.log >>>>> When I get the 503 on the access.log, these are the types of errors seen >>>>> on the error.log >>>>> >>>>> One type of error >>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >>>>> 140518453380864] [client 118.126.82.157:37722 >>>>> <http://118.126.82.157:37722/>] Timeout when reading response headers >>>>> from daemon process 'my_project': >>>>> /data/home/project/my_project/my_project/wsgi.py >>>>> >>>>> Another type of error >>>>> [Thu Sep 01 04:27:00.053558 2022] [wsgi:error] [pid 3267:tid >>>>> 140518595991296] (11)Resource temporarily unavailable: [client >>>>> 172.31.17.102:31880 <http://172.31.17.102:31880/>] mod_wsgi (pid=3267): >>>>> Unable to connect to WSGI daemon process ' my_project ' on >>>>> '/var/run/apache2/wsgi.2385.1.1.sock' after multiple attempts as listener >>>>> backlog limit was exceeded or the socket does not exist. >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> Juan Khawly >>>>> >>>>> On Thursday, September 1, 2022 at 5:54:43 PM UTC-4 Graham Dumpleton wrote: >>>>> Would need to see the mod_wsgi configuration you are using to configure >>>>> the WSGI application, including how WSGIDaemonProcess is configured and >>>>> whether you are using WSGIApplicationGroup. Also, what errors are in the >>>>> Apache error log when the 503 errors occur. >>>>> >>>>> >>>>>> On 2 Sep 2022, at 4:57 am, Juan Khawly <juan...@ <>gmail.com >>>>>> <http://gmail.com/>> wrote: >>>>>> >>>>> >>>>>> Hello, >>>>>> >>>>>> I've been running into this problem for a while. >>>>>> >>>>>> CONTEXT >>>>>> >>>>>> I have an application developed in python (3.10), django 4.0.3, using >>>>>> mod_wsgi and apache. The application is in a DEV environment and hosted >>>>>> in AWS EC2. Currently, it does not receive traffic at all. >>>>>> >>>>>> Installation of Mod WSGI >>>>>> apt-get install -y apache2-dev >>>>>> >>>>>> Setup out of the VENV >>>>>> mod_wsgi-express install-module >>>>>> >>>>>> editing: /etc/apache2/mods-available/wsgi.load >>>>>> >>>>>> LoadModule wsgi_module >>>>>> "/usr/lib/apache2/modules/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so >>>>>> <http://mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so/>" >>>>>> WSGIPythonHome "/data/home/user/environment/venv" >>>>>> >>>>>> Module Enabled >>>>>> a2enmod wsgi >>>>>> >>>>>> PROBLEM >>>>>> >>>>>> The application works perfect most of the time. Couple of times a week, >>>>>> without traffic the apache server goes down into 503. Usually it is >>>>>> preceded by a random request but it does not always happen that way. I >>>>>> am assuming that is Slow DDOS but I want to make sure it is not miss >>>>>> configuration of the WSGI. >>>>>> >>>>>> access.log example >>>>> >>>>>> <access.PNG> >>>>>> >>>>>> error.log example >>>>>> I masked the internal routes >>>>>> >>>>>> This is one of the errors: >>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >>>>>> 140518453380864] [client 118.126.82.157:37722 >>>>>> <http://118.126.82.157:37722/>] Timeout when reading response headers >>>>>> from daemon process 'XXXXX': /XXX/XXXX/XXXXX/XXXXX/XXXXXX/wsgi.py >>>>>> >>>>>> Another type of error: >>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >>>>>> 140518453380864] [client 118.126.82.157:37722 >>>>>> <http://118.126.82.157:37722/>] Timeout when reading response headers >>>>>> from daemon process 'XXXXXXX': /XXX/XXXX/XXXXX/XXXXXX/XXXXXXX/wsgi.py >>>>>> >>>>>> SOLUTION >>>>>> >>>>>> If I restart the server, all works again until next failure. >>>>>> >>>>>> I've enabled the following modules, in case it is SlowDDOS >>>>>> modreqtimeout >>>>>> libapache2-mod-qos >>>>>> libapache2-mod-security2. >>>>>> >>>>>> Any recommendation? >>>>>> >>>>>> Thanks, >>>>>> Juan Khawly >>>>>> >>>>>> >>>>>> >>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "modwsgi" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to modwsgi+u...@ <>googlegroups.com <http://googlegroups.com/>. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>>>> <access.PNG> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "modwsgi" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected] <>. >>>> >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "modwsgi" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected] <>. >>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com?utm_medium=email&utm_source=footer>. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] <>. >> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com?utm_medium=email&utm_source=footer>. >>> <error.log> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <>. > >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/a69d70f4-faac-4515-b991-90590efd7b95n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/a69d70f4-faac-4515-b991-90590efd7b95n%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/e77b2844-6948-457b-8a13-65ef7698c377n%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/e77b2844-6948-457b-8a13-65ef7698c377n%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/4E2EABA9-4B12-4C89-8CE4-BC6BA5B5E1B4%40gmail.com.
