Thanks for the response. Yes we have 16 shards/partitions each on 16 different nodes and a separate master Solr receiving continuous parallel requests from 10 client threads running on a single separate machine. Our observation was that the perf degraded non linearly as the load ( no of concurrent clients ) increased.
Have some followup questions : 1. What is the default maxnumber of threads configured when a Solr instance make calls to other 16 partitions ? 2. How do I increase the max no of connections for solr<-->solr interactions as u mentioned in your mail ? On 2/22/12, Yonik Seeley <yo...@lucidimagination.com> wrote: > On Wed, Feb 22, 2012 at 9:27 AM, prasenjit mukherjee > <prasen....@gmail.com> wrote: >> Is anybody aware of any effort regarding porting solr to a netty ( or >> any other async-io based framework ) based framework. >> >> Even on medium load ( 10 parallel clients ) with 16 shards >> performance seems to deteriorate quite sharply compared another >> alternative ( async-io based ) solution as load increases. > > By "16 shards" do you mean you have 16 nodes and each single client > request causes a distributed search across all of them them? How many > concurrent requests are your 10 clients making to each node? > > NIO works well when there are many clients, but when servicing those > client requests only needs intermittent CPU. That's not the pattern > we see for search. > You *can* easily configure Solr's Jetty to use NIO when accepting > client connections, but it won't do you any good, just as switching to > Netty wouldn't do anything here. > > Where NIO could help a little is with the requests that Solr makes to > other Solr instances. Solr is already architected for async > request-response to other nodes, but the current underlying > implementation uses HttpClient 3 (which doesn't have NIO). > > Anyway, it's unlikely that NIO vs BIO will make much of a difference > with the numbers you're talking about (16 shards). > > Someone else reported that we have the number of connections per host > set too low, and they saw big gains by increasing this. There's an > issue open to make this configurable in 3x: > https://issues.apache.org/jira/browse/SOLR-3079 > We should probably up the max connections per host by default. > > -Yonik > lucidimagination.com > -- Sent from my mobile device
