Hi,
My production servers are deployed as 4 uwsgi instances:
front1 -> admin3+admin4
front2 -> admin3+admin4
front1/2 are load balanced at the DNS level and perform ssl termination
with uwsgi and forward requests to admin3/4 by using an http subscription
server. admin3/4 run my python application within a simple uwsgi master+5
slaves.
Before you ask: on all 4 servers,
$ uwsgi --version
2.0.15-debian
Recently, I became aware of a low-probability bug triggered by multiple
customers across a wide range of access networks that happens when reading
the body of my incoming POST HTTP requests at the application level.
Here is a typical backtrace:
Mar 09 08:32:10 admin3 uwsgi[8246]: admin-5 ERROR root timeout
during read(2) on wsgi.input
Traceback (most recent call last):
File
"/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request,
response, e)
File
"/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request,
response)
File
"/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1278, in
default_dispatcher
return
route.handler_adapter(request, response)
File
"/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "admin2/BaseHandler.py", line
99, in dispatch
self._do_dispatch(user)
File "admin2/BaseHandler.py", line
327, in _do_dispatch
self._begin_transaction(company_id,
user)
File "admin2/BaseRequestHandler.py",
line 210, in _begin_transaction
self._data = self.request.body
File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 702, in body
self.make_body_seekable() # we need
this to have content_length
File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 930, in
make_body_seekable
self.copy_body()
File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 954, in copy_body
data = input.read(min(todo, 65535))
File
"/usr/lib/python2.7/dist-packages/webob/request.py", line 1540, in readinto
data = self.file.read(sz0)
IOError: timeout during read(2) on
wsgi.input
Mar 09 08:32:10 admin3 admin[3875]: [uwsgi-body-read] Timeout reading 2
bytes. Content-Length: 2 consumed: 0 left: 2
Ok, I get this: my application is trying to read these damn 2 bytes but
they are not in the wsgi input file so, uwsgi triggers the 4s (default
value) socket timeout.
Now, the issue here is that I'd like to figure out if these damn bytes were
actually received on the application server and if not, where they got lost
so I can blame someone else.
Has anyone any idea on what I could do to debug/trace this further on my
production infrastructure ? (clearly, logging all incoming traffic with
tcpdump is not an option)
Mathieu
--
Mathieu Lacage <[email protected]>
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
