Am Mittwoch, 28. September 2011 um 16:40 schrieb Toke Eskildsen:
> On Wed, 2011-09-28 at 12:58 +0200, Frederik Kraus wrote: > > - 10 shards per server (needed for response times) running in a single > > tomcat instance > > Have you tested that sharding actually decreases response times in your > case? I see the idea in decreasing response times with sharding at the > cost of decreasing throughput, but the added overhead of merging is > non-trivial. Yep unfortunately, the queries have huge boolean filterqueries for ACLs etc. which just take too long to compute in a single thread. > > > - each query queries all 20 shards (distributed search) > > > > - each shard holds about 1.5 mio documents (small shards are needed due to > > rather complex queries) > > - all caches are warmed / high cache hit rates (99%) etc. > > > Now for some reason we cannot seem to fully utilize all CPU power (no disk > > IO), ie. increasing concurrent users doesn't increase CPU-Load at a point, > > decreases throughput and increases the response times of the individual > > queries. > > It sounds as if there's a hard limit on the number of concurrent users > somewhere. I am no expert in httpclient, but the blocked threads in your > thread dump seems to indicate that they wait for connections to be > established rather than for results to be produced. > > I seem to remember that tomcat has a default limit on 200 concurrent > connections and with 10 shards/search, that is just 200 / (10 > shard_connections + 1 incoming_connection) = 18 concurrent searches. > I have gradually bumped all of this up to (almost) infinity with no effect ;) > > Also 1-2% of the queries take significantly longer: avg somewhere at 100ms > > while 1-2% take 1.5s or longer. > > Could be garbage collection, especially since it shows under high load > which might result in more old objects and thereby trigger full gc. GC is only spending something like 50-100ms total for a 10min load test
