On 1/7/14 12:53, Young King wrote:
> I had met the same problem about kazoo before.
>
> Python's import mechanism will add the import lock. Your code
> had instantiate the kazoo client during the import, which will add
> another import lock during serialize, if the locale is not utf8, so
> the deadlock happened.
>
I'm not sure this is the case, I've narrowed the problem down to Kazoo's
logging being on DEBUG and triggering the callback of DataWatch, while
running under uWSGI with LANG unset. If I switch kazoo to INFO,
it magically starts working (even without LANG!).
> Specific process is shown:
>
> import test.py --> add import lock
> |
>
> instantiate kazoo
>
> |
>
> serialize network data
>
> | (if locale is not utf8)
>
> mod = __import__('encodings.' + modnam ...) --> add
> another import lock
>
>
> On Tue, Jul 1, 2014 at 4:00 PM, Roberto De Ioris <[email protected]
> <mailto:[email protected]>> wrote:
>
>
> > Hi,
> >
> > I've got a pretty weird situation here and I can't figure out what's
> > going on.
> >
> > My application is a simple WSGI app which uses Kazoo
> > (https://kazoo.readthedocs.org/en/latest/) to stay connected to a
> > Zookeeper server and dynamically update the app's configuration
> when the
> > data changes. The problem I'm facing is the
> > following: if I run the application from the command line (as root),
> > uWSGI and the app start and behave properly, but whenever
> > I start the application using an Upstart script (I'm on Ubuntu
> 12.04) it
> > hangs at some point after starting up and it's not
> > answering to requests.
> >
> > From what I know, Kazoo uses a background thread to talk to the
> > zookeeper server. This one produces periodic debug
> > messages "sent ping", "received ping" in my kazoo version so I
> know that
> > when I see these messages, the application is
> > working. When it's hang though, the log *looks* like this thread
> has not
> > even started (that's not true, see below).
> >
> > So, what's the difference between running the application under
> upstart
> > and from the command line?
> > Why is it working in one case and not in the other? To me it
> looks like
> > uWSGI deadlocks for a reason unknown to me.
> >
> >
> > Technical details
> > ==========
> >
> > * Versions
> >
> > - Ubuntu: 12.04
> > - Upstart: 1.5-0ubuntu7.2
> > - uwsgi: 1.0.3+dfsg-1ubuntu0.1 (ubuntu package) , 1.9.21
> (in-house built
> > .deb) or 2.0.5.1 (pip install uwsgi) - problem
> > persists with all of them
> > - python: 2.7.3-0ubuntu2.2
> > - python-kazoo: 1.3.1-1 (in-house built .deb) or 2.0 (pip
> install kazoo)
> > - python-bottle: 0.12.0 (in-house built .deb)
> >
> > * Platform: EC2 Ubuntu 12.04 instance
> > * Application (needs kazoo and bottle):
> >
> > test.py, save as /home/ubuntu/uwsgi/test.py:
> http://pastebin.com/w0B6WMat
> > wsgi.xml, save as /home/ubuntu/uwsgi/wsgi.xml:
> > http://pastebin.com/fZmiUZCY
> >
> > * How to test:
> >
> > $ curl http://localhost:9090/ping , if it hangs, it's broken.
> >
> > * uWSGI configuration:
> >
> > I'm running a setup with one master process and a single worker,
> threads
> > enabled (see wsgi.xml above).
> >
> > Running from the command line: /usr/bin/uwsgi_python --xmlconfig
> > /home/ubuntu/uwsgi/wsgi.xml --http-socket 0.0.0.0:9090
> <http://0.0.0.0:9090>
> > Running from upstart (config file): http://pastebin.com/9HrAaZxP
> >
> > * Debugging
> >
> > If i remove the <lazy /> option from the config file or if I
> replace it
> > with <lazy-app /> the application only works when started from the
> > command line. If started from Upstart, it deadlocks and doesn't even
> > create the worker process, nor does it change the process title as
> > specified in the config file. So removing <lazy> makes things worse.
> >
> >
> > ----[ application hang (running in Upstart)
> >
> > 1) attached gdb to the worker process
> >
> > (gdb) info thread
> > Id Target Id Frame
> > 4 Thread 0x7f87c1ae0700 (LWP 4990) "uwsgi_python"
> > 0x00007f87c8db6fd0 in sem_wait ()
> > from /lib/x86_64-linux-gnu/libpthread.so.0
> > 3 Thread 0x7f87c12df700 (LWP 4991) "uwsgi_python"
> > 0x00007f87c8db6fd0 in sem_wait ()
> > from /lib/x86_64-linux-gnu/libpthread.so.0
> > 2 Thread 0x7f87c0ade700 (LWP 4992) "uwsgi_python"
> > 0x00007f87c8db6fd0 in sem_wait ()
> > from /lib/x86_64-linux-gnu/libpthread.so.0
> > * 1 Thread 0x7f87c91d0780 (LWP 4985) "uwsgi_python"
> > 0x00007f87c8db6fd0 in sem_wait ()
> > from /lib/x86_64-linux-gnu/libpthread.so.0
> >
> > 2) gdb 'thread apply all where' : http://pastebin.com/z5USTQGW
> > 3) log file: http://pastebin.com/eyRujTBh
> >
> >
> > ----[ application working (running from commandline)
> >
> > 1) attached gdb to the worker process
> >
> > (gdb) info thread
> > Id Target Id Frame
> > 4 Thread 0x7f94fc391700 (LWP 5630) "uwsgi_python"
> > 0x00007f950362afd0 in sem_wait ()
> > from /lib/x86_64-linux-gnu/libpthread.so.0
> > 3 Thread 0x7f94fbb90700 (LWP 5631) "uwsgi_python"
> > 0x00007f950362afd0 in sem_wait ()
> > from /lib/x86_64-linux-gnu/libpthread.so.0
> > 2 Thread 0x7f94fb38f700 (LWP 5632) "uwsgi_python"
> > 0x00007f9501f95763 in select ()
> > from /lib/x86_64-linux-gnu/libc.so.6
> > * 1 Thread 0x7f9503a44780 (LWP 5625) "uwsgi_python"
> > 0x00007f9501f9ca93 in epoll_wait ()
> > from /lib/x86_64-linux-gnu/libc.so.6
> >
> > 2) gdb 'thread apply all where': http://pastebin.com/r4iMB9Za
> > 3) log file: http://pastebin.com/ybrMjVU6
> >
> > GDB backtraces are against uwsgi 1.0.3 , log files are from 1.9.21.
> > However, there seems to be no difference between the versions with
> > respect to this issue.
> >
> > Any ideas?
> >
> > Regards,
> > Cosmin
> >
> >
>
>
> The only differences could be in the current working directory
> (you force
> it with chdir option) and some environment variable not set by upstart
> (add <hook-asap>exec:export</hook-asap> to get them printed when uWSGI
> runs).
>
> Btw try to always run stable releases when debugging (currently
> 1.4.10 and
> 2.0.6)
>
>
> --
> Roberto De Ioris
> http://unbit.it
> _______________________________________________
> uWSGI mailing list
> [email protected] <mailto:[email protected]>
> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
>
>
>
>
> --
> Name: Young King
> Tel: 18610773920
> Add:
>
> Zhihu.com
> A-West 3-010
> 768 Creative Park
> 5 Xueyuan Road
> Haidian District,Beijing 100083
> China
>
>
> _______________________________________________
> uWSGI mailing list
> [email protected]
> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
