I am assuming that you are running on linux here, I have found atop to be very
useful to see what is going on.
http://freshmeat.net/projects/atop/
dstat is also very useful too but needs a little more work to 'decode'.
Obviously there is contention going on, you just need to figure out where it
is, most likely it is disk I/O but it could also be the number of cores you
have. Also I would not say that performance is decreasing rapidly, probably
more of a gentle slope down if you plot it (your double the number of cores
every time).
I would be very interested in hearing about what you find.
Cheers
François
On Jun 16, 2011, at 10:00 AM, Andrzej Bialecki wrote:
> On 6/16/11 3:22 PM, Mark Schoy wrote:
>> Hi,
>>
>> I set up a Solr instance with 512 cores. Each core has 100k documents and 15
>> fields. Solr is running on a CPU with 4 cores (2.7Ghz) and 16GB RAM.
>>
>> Now I've done some benchmarks with JMeter. On each thread iteration JMeter
>> queriing another Core by random. Here are the results (Duration: each with
>> 180 second):
>>
>> Randomly queried cores | queries per second
>> 1| 2016
>> 2 | 2001
>> 4 | 1978
>> 8 | 1958
>> 16 | 2047
>> 32 | 1959
>> 64 | 1879
>> 128 | 1446
>> 256 | 1009
>> 512 | 428
>>
>> Why are the queries per second until 64 constant and then the performance is
>> degreasing rapidly?
>>
>> Solr only uses 10GB of the 16GB memory so I think it is not a memory issue.
>>
>
> This may be an OS-level disk buffer issue. With a limited disk buffer space
> the more random IO occurs from different files, the higher is the churn rate,
> and if the buffers are full then the churn rate may increase dramatically
> (and the performance will drop then). Modern OS-es try to keep as much data
> in memory as possible, so the memory usage itself is not that informative -
> but check what are the pagein/pageout rates when you start hitting the 32 vs
> 64 cores.
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
