I'd like to warn people about the default configuration of Jetty in the Solr trunk release (not present in Solr 1.4 and prior). There is a difference in the jetty configuration which is for the latest Solr to use the QueuedThreadPool (as seen in jetty.xml). Previously, it had used a BoundedThreadPool implementation that I've heard is considered deprecated presently. I have a multi-core setup where Jetty is serving up lots of Solr cores 9+ and when our client does a distributed search (3 of them at a time actually), it triggers a condition in which the query takes 50 plus seconds to respond. During this time, the machine is effectively idle, seemingly waiting for something. To fix this, go back to the former BoundedThreadPool implementation or don't use Jetty. FWIW this has triggered us to swtich to Tomcat.
Sorry but I have sunk so much resources into tracking down this nasty problem that I can't spend much more on further figuring out why QueuedThreadPool is failing us. ~ David Smiley Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/