We’re managing a fairly high-traffic site and we’re seeing a lot of HTTP 503s with termination state SQ. I’m trying to figure out what this means, since Q indicates that we’re still in the queue (in the server-specific queue in our case) whereas S indicates a TCP connection issue, which I would have thought meant that we’d made it into a connection slot.
Here’s a sanitized version of a sample log line: Nov 24 03:58:54 lb03 haproxy[20966]: 71.72.73.74:60738 [24/Nov/2014:03:58:54.245] foo3 foo/app53 0/-1/-1/-1/0 503 212 - - SQVN 159/131/132/25/0 0/0 "PUT /foo/bar/baz HTTP/1.1" The server maxconn is set to 25, and this shows that the server has 25 connections already, so it makes sense that we’re in the queue. We do have retries set (to 3) and redispatch on, but neither appears to be happening here. We’re running HAproxy 1.5.4 on CentOS 6.5. (Note that we know that the app servers are having issues. Our main goals here are to learn what we can about the app server behavior, and to figure out what we can do in HAproxy to mitigate the issues until the developers manage to fix them.) Thanks, -Daniel Daniel Lieberman BitPusher, LLC
