On 7/2/2011 12:34 PM, Yonik Seeley wrote:
OK, I tried a quick test of 1.4.1 vs 3x on optimized indexes (unoptimized had different numbers of segments so I didn't try that). 3x (as of today) was 28% faster at a large filter query (300 terms in one big disjunction, with each term matching ~1000 docs).
A lot of the terms used in my filter queries may match hundreds of thousands or even millions of documents. The largest search group (sg:stdp) matches about 1.4 million out of 9.5 million docs on each shard, and is probably present in most filter queries.
Right now I have the default termIndexInterval of 128, and a setTermIndexDivisor of 8. I think this probably has the same memory footprint as a termIndexInterval of 1024, but because it can do seeks in the tii file (taking good advantage of disk cache) before it ultimately seeks in the tis file, there are probably fewer seeks. My warm time is slightly better than it was with the interval at 1024, and my average query speed hasn't changed much. I am going to try an interval of 64 and a divisor of 16.
I'm interested in other performance enhancing ideas that don't involve tweaking tons of options all at the same time. I think my best bet for performance is adding more memory, of course.
Shawn
