On 1/3/2013 8:11 AM, Michael Ryan wrote:
We see these EofExceptions in our system occasionally. I believe they occur when our SolrJ client times out and closes the connection, before Jetty returns the response.
This would make sense - the load balancer probably drops the healthcheck connection after its five second timeout. I also have a two second SoTimeout on the server objects in my app that checks /admin/mbeans (16 times (cores) per server). That app also checks the ping handler on one core for each server.
I suppose the question is why is it taking so long to make the ping request? The ping request just makes a query for *:* ... it is a distributed request, but even so, for that to take more than two (or even five) seconds seems to indicate there may be a larger problem.
Long stop-the-world GC pauses would explain it, if that's happening. I guess it's time to turn on GC logging. Jconsole's overall stats don't suggest that it's a problem, but that is only summary information.
On my 4.1 dev server using Oracle Java7, the only memory tuning options I have are a max heap of 8GB and -XX:+UseG1GC. On my production 3.5 servers using Oracle Java6, I have the following in addition to a max heap of 8GB:
-Xmx8192M -XX:NewRatio=1 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
Any pointers would be appreciated! Thanks, Shawn
