Hello,
We have a critical issue where our app running in Tomcat (8.x) attempts to
make a request to our other app running in Jetty (9.2.7) with Java 8
occasionally gets one or two ConnectExceptions which results in that app
being marked unavailable - because our previous understanding was that we'd
only receive that in cases where the app/server was physically down (or
some other permanent issue like firewall, etc).
We can't determine why this happens (in one of the two cases) or when it
might happen (no pattern yet) and are at a loss on how to troubleshoot this
further. Is there logging for the OS or Jetty that might help here? Or
any other ideas why we might have this?
It's happened a few times at one customer site running Windows 2012 - but
there it's always happened under extremely heavy load. So my guess is that
it's a thread or OS or port limit that's being encountered. (But we still
can't say for sure yet.)
The other case is on CentOS release 6.9 (Final) on a popular cloud server
provider, and it's happened two separate times, and in neither was the
server or applications under any kind of substantial load. There's nothing
in the (minimal) system logs or the app logs to indicate an issue. Just a
request from Tomcat that ends up getting a ConnectException and so we mark
the server as unavailable.
We use something along these lines for our Embedded Jetty server...
WebAppContext context = new WebAppContext();
QueuedThreadPool threadPool = new QueuedThreadPool(maxThreads); // usually
256
Server server = new Server(threadPool);
HttpConfiguration http_config = new HttpConfiguration();
ServerConnector http = new ServerConnector(server, new
HttpConnectionFactory(http_config));
http.setPort(port);
server.setConnectors(new Connector[]
{
http
});
server.setHandler(context);
server.start();
server.join();
The app works quite well most of the time but we're baffled under what
circumstances Jetty would briefly refuse connections (when not under heavy
load). Nothing on the server indicates any issues at the time.
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1031937
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 8192
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 8192
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Memory of the server or app is not low at the time this occurs. And I've
tried to reproduce in-house by forcing long gc cycles (or overloading the
app) but we just end up getting timeouts rather than ConnectExceptions.
Any help appreciated.
_______________________________________________
jetty-users mailing list
[email protected]
To change your delivery options, retrieve your password, or unsubscribe from
this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
