> Hi, > > I've got a pretty weird situation here and I can't figure out what's > going on. > > My application is a simple WSGI app which uses Kazoo > (https://kazoo.readthedocs.org/en/latest/) to stay connected to a > Zookeeper server and dynamically update the app's configuration when the > data changes. The problem I'm facing is the > following: if I run the application from the command line (as root), > uWSGI and the app start and behave properly, but whenever > I start the application using an Upstart script (I'm on Ubuntu 12.04) it > hangs at some point after starting up and it's not > answering to requests. > > From what I know, Kazoo uses a background thread to talk to the > zookeeper server. This one produces periodic debug > messages "sent ping", "received ping" in my kazoo version so I know that > when I see these messages, the application is > working. When it's hang though, the log *looks* like this thread has not > even started (that's not true, see below). > > So, what's the difference between running the application under upstart > and from the command line? > Why is it working in one case and not in the other? To me it looks like > uWSGI deadlocks for a reason unknown to me. > > > Technical details > ========== > > * Versions > > - Ubuntu: 12.04 > - Upstart: 1.5-0ubuntu7.2 > - uwsgi: 1.0.3+dfsg-1ubuntu0.1 (ubuntu package) , 1.9.21 (in-house built > .deb) or 2.0.5.1 (pip install uwsgi) - problem > persists with all of them > - python: 2.7.3-0ubuntu2.2 > - python-kazoo: 1.3.1-1 (in-house built .deb) or 2.0 (pip install kazoo) > - python-bottle: 0.12.0 (in-house built .deb) > > * Platform: EC2 Ubuntu 12.04 instance > * Application (needs kazoo and bottle): > > test.py, save as /home/ubuntu/uwsgi/test.py: http://pastebin.com/w0B6WMat > wsgi.xml, save as /home/ubuntu/uwsgi/wsgi.xml: > http://pastebin.com/fZmiUZCY > > * How to test: > > $ curl http://localhost:9090/ping , if it hangs, it's broken. > > * uWSGI configuration: > > I'm running a setup with one master process and a single worker, threads > enabled (see wsgi.xml above). > > Running from the command line: /usr/bin/uwsgi_python --xmlconfig > /home/ubuntu/uwsgi/wsgi.xml --http-socket 0.0.0.0:9090 > Running from upstart (config file): http://pastebin.com/9HrAaZxP > > * Debugging > > If i remove the <lazy /> option from the config file or if I replace it > with <lazy-app /> the application only works when started from the > command line. If started from Upstart, it deadlocks and doesn't even > create the worker process, nor does it change the process title as > specified in the config file. So removing <lazy> makes things worse. > > > ----[ application hang (running in Upstart) > > 1) attached gdb to the worker process > > (gdb) info thread > Id Target Id Frame > 4 Thread 0x7f87c1ae0700 (LWP 4990) "uwsgi_python" > 0x00007f87c8db6fd0 in sem_wait () > from /lib/x86_64-linux-gnu/libpthread.so.0 > 3 Thread 0x7f87c12df700 (LWP 4991) "uwsgi_python" > 0x00007f87c8db6fd0 in sem_wait () > from /lib/x86_64-linux-gnu/libpthread.so.0 > 2 Thread 0x7f87c0ade700 (LWP 4992) "uwsgi_python" > 0x00007f87c8db6fd0 in sem_wait () > from /lib/x86_64-linux-gnu/libpthread.so.0 > * 1 Thread 0x7f87c91d0780 (LWP 4985) "uwsgi_python" > 0x00007f87c8db6fd0 in sem_wait () > from /lib/x86_64-linux-gnu/libpthread.so.0 > > 2) gdb 'thread apply all where' : http://pastebin.com/z5USTQGW > 3) log file: http://pastebin.com/eyRujTBh > > > ----[ application working (running from commandline) > > 1) attached gdb to the worker process > > (gdb) info thread > Id Target Id Frame > 4 Thread 0x7f94fc391700 (LWP 5630) "uwsgi_python" > 0x00007f950362afd0 in sem_wait () > from /lib/x86_64-linux-gnu/libpthread.so.0 > 3 Thread 0x7f94fbb90700 (LWP 5631) "uwsgi_python" > 0x00007f950362afd0 in sem_wait () > from /lib/x86_64-linux-gnu/libpthread.so.0 > 2 Thread 0x7f94fb38f700 (LWP 5632) "uwsgi_python" > 0x00007f9501f95763 in select () > from /lib/x86_64-linux-gnu/libc.so.6 > * 1 Thread 0x7f9503a44780 (LWP 5625) "uwsgi_python" > 0x00007f9501f9ca93 in epoll_wait () > from /lib/x86_64-linux-gnu/libc.so.6 > > 2) gdb 'thread apply all where': http://pastebin.com/r4iMB9Za > 3) log file: http://pastebin.com/ybrMjVU6 > > GDB backtraces are against uwsgi 1.0.3 , log files are from 1.9.21. > However, there seems to be no difference between the versions with > respect to this issue. > > Any ideas? > > Regards, > Cosmin > >
The only differences could be in the current working directory (you force it with chdir option) and some environment variable not set by upstart (add <hook-asap>exec:export</hook-asap> to get them printed when uWSGI runs). Btw try to always run stable releases when debugging (currently 1.4.10 and 2.0.6) -- Roberto De Ioris http://unbit.it _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
