Thanks for your answers. Andrzej was right with his assumption. Solr only needs about 9GB memory but the system needs the rest of it for disc IO:
64 Cores: 64*100MB index size = 6,4GB + 9 GB Solr Cache + about 600 MB OS = 16GB Conclusion: My system can exactly buffer the data of 64 Cores. Every additional core cant be buffered and the performance is decreasing. 2011/6/16 François Schiettecatte <fschietteca...@gmail.com> > I am assuming that you are running on linux here, I have found atop to be > very useful to see what is going on. > > http://freshmeat.net/projects/atop/ > > dstat is also very useful too but needs a little more work to 'decode'. > > Obviously there is contention going on, you just need to figure out where > it is, most likely it is disk I/O but it could also be the number of cores > you have. Also I would not say that performance is decreasing rapidly, > probably more of a gentle slope down if you plot it (your double the number > of cores every time). > > I would be very interested in hearing about what you find. > > Cheers > > François > > On Jun 16, 2011, at 10:00 AM, Andrzej Bialecki wrote: > > > On 6/16/11 3:22 PM, Mark Schoy wrote: > >> Hi, > >> > >> I set up a Solr instance with 512 cores. Each core has 100k documents > and 15 > >> fields. Solr is running on a CPU with 4 cores (2.7Ghz) and 16GB RAM. > >> > >> Now I've done some benchmarks with JMeter. On each thread iteration > JMeter > >> queriing another Core by random. Here are the results (Duration: each > with > >> 180 second): > >> > >> Randomly queried cores | queries per second > >> 1| 2016 > >> 2 | 2001 > >> 4 | 1978 > >> 8 | 1958 > >> 16 | 2047 > >> 32 | 1959 > >> 64 | 1879 > >> 128 | 1446 > >> 256 | 1009 > >> 512 | 428 > >> > >> Why are the queries per second until 64 constant and then the > performance is > >> degreasing rapidly? > >> > >> Solr only uses 10GB of the 16GB memory so I think it is not a memory > issue. > >> > > > > This may be an OS-level disk buffer issue. With a limited disk buffer > space the more random IO occurs from different files, the higher is the > churn rate, and if the buffers are full then the churn rate may increase > dramatically (and the performance will drop then). Modern OS-es try to keep > as much data in memory as possible, so the memory usage itself is not that > informative - but check what are the pagein/pageout rates when you start > hitting the 32 vs 64 cores. > > > > -- > > Best regards, > > Andrzej Bialecki <>< > > ___. ___ ___ ___ _ _ __________________________________ > > [__ || __|__/|__||\/| Information Retrieval, Semantic Web > > ___|||__|| \| || | Embedded Unix, System Integration > > http://www.sigram.com Contact: info at sigram dot com > > > >
