The env in upstart is, indeed, minimalistic. For the fun of it, I've changed the upstart script to export the same environment so that it looked like the shell. I didn't expect it to work, but... it did :(
I've nailed it down to "env LANG='en_US.UTF-8' ". Seems this makes the difference between locking up and not locking up. Now I'm testing to see if I can reproduce the problem using the minimalistic environment from upstart (env -i) and bottle's internal server. But no luck, it works :( I've even tried some fancy start-stop-daemon to make it as close as the upstart scenario as possible, but I can't reproduce. So I'm guessing that the problem is because of uwsgi + kazoo without LANG set. I've further isolated the problem to the DataWatch component of kazoo, which locks up when trying to invoke the callback if logging on DEBUG (INFO and above work). Guess I need to ask on kazoo's mailing list about this, too. PS: For reference (written before testing the env thing): I've re-done the backtraces and logs using the latest uWSGI version (2.0.6). Upstart, application hangs: http://pastebin.com/LkWBVAbZ Command line, application runs: http://pastebin.com/gdRJiCsA As far as the current directory goes, I've not mentioned but when running from the command line I'm in the same directory as the `chdir` in upstart (/home/ubuntu/uwsgi). On 1/7/14 11:00, Roberto De Ioris wrote: >> Hi, >> >> I've got a pretty weird situation here and I can't figure out what's >> going on. >> >> My application is a simple WSGI app which uses Kazoo >> (https://kazoo.readthedocs.org/en/latest/) to stay connected to a >> Zookeeper server and dynamically update the app's configuration when the >> data changes. The problem I'm facing is the >> following: if I run the application from the command line (as root), >> uWSGI and the app start and behave properly, but whenever >> I start the application using an Upstart script (I'm on Ubuntu 12.04) it >> hangs at some point after starting up and it's not >> answering to requests. >> >> From what I know, Kazoo uses a background thread to talk to the >> zookeeper server. This one produces periodic debug >> messages "sent ping", "received ping" in my kazoo version so I know that >> when I see these messages, the application is >> working. When it's hang though, the log *looks* like this thread has not >> even started (that's not true, see below). >> >> So, what's the difference between running the application under upstart >> and from the command line? >> Why is it working in one case and not in the other? To me it looks like >> uWSGI deadlocks for a reason unknown to me. >> >> >> Technical details >> ========== >> >> * Versions >> >> - Ubuntu: 12.04 >> - Upstart: 1.5-0ubuntu7.2 >> - uwsgi: 1.0.3+dfsg-1ubuntu0.1 (ubuntu package) , 1.9.21 (in-house built >> .deb) or 2.0.5.1 (pip install uwsgi) - problem >> persists with all of them >> - python: 2.7.3-0ubuntu2.2 >> - python-kazoo: 1.3.1-1 (in-house built .deb) or 2.0 (pip install kazoo) >> - python-bottle: 0.12.0 (in-house built .deb) >> >> * Platform: EC2 Ubuntu 12.04 instance >> * Application (needs kazoo and bottle): >> >> test.py, save as /home/ubuntu/uwsgi/test.py: http://pastebin.com/w0B6WMat >> wsgi.xml, save as /home/ubuntu/uwsgi/wsgi.xml: >> http://pastebin.com/fZmiUZCY >> >> * How to test: >> >> $ curl http://localhost:9090/ping , if it hangs, it's broken. >> >> * uWSGI configuration: >> >> I'm running a setup with one master process and a single worker, threads >> enabled (see wsgi.xml above). >> >> Running from the command line: /usr/bin/uwsgi_python --xmlconfig >> /home/ubuntu/uwsgi/wsgi.xml --http-socket 0.0.0.0:9090 >> Running from upstart (config file): http://pastebin.com/9HrAaZxP >> >> * Debugging >> >> If i remove the <lazy /> option from the config file or if I replace it >> with <lazy-app /> the application only works when started from the >> command line. If started from Upstart, it deadlocks and doesn't even >> create the worker process, nor does it change the process title as >> specified in the config file. So removing <lazy> makes things worse. >> >> >> ----[ application hang (running in Upstart) >> >> 1) attached gdb to the worker process >> >> (gdb) info thread >> Id Target Id Frame >> 4 Thread 0x7f87c1ae0700 (LWP 4990) "uwsgi_python" >> 0x00007f87c8db6fd0 in sem_wait () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> 3 Thread 0x7f87c12df700 (LWP 4991) "uwsgi_python" >> 0x00007f87c8db6fd0 in sem_wait () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> 2 Thread 0x7f87c0ade700 (LWP 4992) "uwsgi_python" >> 0x00007f87c8db6fd0 in sem_wait () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> * 1 Thread 0x7f87c91d0780 (LWP 4985) "uwsgi_python" >> 0x00007f87c8db6fd0 in sem_wait () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> >> 2) gdb 'thread apply all where' : http://pastebin.com/z5USTQGW >> 3) log file: http://pastebin.com/eyRujTBh >> >> >> ----[ application working (running from commandline) >> >> 1) attached gdb to the worker process >> >> (gdb) info thread >> Id Target Id Frame >> 4 Thread 0x7f94fc391700 (LWP 5630) "uwsgi_python" >> 0x00007f950362afd0 in sem_wait () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> 3 Thread 0x7f94fbb90700 (LWP 5631) "uwsgi_python" >> 0x00007f950362afd0 in sem_wait () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> 2 Thread 0x7f94fb38f700 (LWP 5632) "uwsgi_python" >> 0x00007f9501f95763 in select () >> from /lib/x86_64-linux-gnu/libc.so.6 >> * 1 Thread 0x7f9503a44780 (LWP 5625) "uwsgi_python" >> 0x00007f9501f9ca93 in epoll_wait () >> from /lib/x86_64-linux-gnu/libc.so.6 >> >> 2) gdb 'thread apply all where': http://pastebin.com/r4iMB9Za >> 3) log file: http://pastebin.com/ybrMjVU6 >> >> GDB backtraces are against uwsgi 1.0.3 , log files are from 1.9.21. >> However, there seems to be no difference between the versions with >> respect to this issue. >> >> Any ideas? >> >> Regards, >> Cosmin >> >> > > The only differences could be in the current working directory (you force > it with chdir option) and some environment variable not set by upstart > (add <hook-asap>exec:export</hook-asap> to get them printed when uWSGI > runs). > > Btw try to always run stable releases when debugging (currently 1.4.10 and > 2.0.6) > > _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
