On Mon, 2011-01-10 at 21:43 +0100, Paul wrote: > > I see from your other messages that these indexes all live on the same > > machine. > > You're almost certainly I/O bound, because you don't have enough memory for > > the > > OS to cache your index files. With 100GB of total index size, you'll get > > best > > results with between 64GB and 128GB of total RAM. > > Is that a general rule of thumb? That it is best to have about the > same amount of RAM as the size of your index?
I does not seems like there is a clear current consensus on hardware to handle IO problems. I am firmly in the SSD camp, but as you can see from the current thread, other people recommend RAM and/or extra machines. I can say that our tests with RAM and spinning disks showed us that a lot of RAM certainly helps a lot, but also that it takes a considerable amount of time to warm the index before the performance is satisfactory. It might be helped with disk cache tricks, such as copying the whole index to /dev/null before opening it in Solr. > So, with a 5GB index, I should have between 4GB and 8GB of RAM > dedicated to solr? Not as -Xmx, but free for disk cache, yes. If you follow the RAM ~= index size recommendation.
