There are certainly a number of widely varying opinions on the use of RAM directory. Basically, though, if you need the index to be persistent at some point (i.e. saved across reboots, crashes etc.), you'll need to write to a disk, so RAM directory becomes somewhat superfluous in this case.
Generally, good hardware and fast disks are a better bet, since you'll probably want to have them anyway :-) >From my own experiences with varying types/sizes of indexes, and the general wisdom gleamed from the experts, the amount of memory required for a given environment is very much a 'how long is a piece of string' type of scenario. It depends on so many factors that it's impractical to come up with a easy 'standardized' formula. What I've found useful as a rough guidance (in additon to the very useful URL I mentioned earlier), is if your server is doing lots of indexing and not much searching, you want your os fs cache to have access to a healthy amount of memory. If you're doing lots of searching/reading (and particularly faceting), you'll want a good amount of ram for Solr/Lucene caching (which caches need what depends on the type of data you're searching). If you have a server that is doing a lot of both indexing and searching, you should consider breaking them out using replication and possibly using load balancers (if you have lots of concurrent querying going on). It stands to reason that the bigger the index gets, the more memory will generally be required for working on various aspects of it. When you get into very large indexes, it becomes more efficient to distribute the indexing across servers (and replicating those servers), so that no single machine has huge cache lists to traverse. Again, the 'Scaling Lucene and Solr' page goes into these scenarios and is well worth studying. On Wed, Mar 17, 2010 at 12:29 AM, KaktuChakarabati <jimmoe...@gmail.com>wrote: > > Hey Peter, > Thanks for your reply. > My question was mainly about the fact there seems to be two different > aspects to the solr RAM usage: in-process and out-process. > By that I mean, yes i know the many different parameters/caches to do with > solr in-process memory usage and related culprits, however I also > understand > that as for actual index access (posting list, positional index etc), solr > mostly delegates the access/caching of this to the OS/disk cache. > So I guess my question is more about that: namely, what would be a good way > to calculate an overall ram requirement profile for a server running solr? > Also, I was under the impression benefits from RAMDirectory would be > minimal > when disk caches are effective no? > And does RAMDirectory work with replication? if so, doesnt it slow it down? > ( on each replication, load up entire index to RAM at once? ) > > > > Peter Sturge wrote: > > > > On Tue, Mar 16, 2010 at 9:08 PM, KaktuChakarabati > > <jimmoe...@gmail.com>wrote: > > > >> > >> Hey, > >> I am trying to understand what kind of calculation I should do in order > >> to > >> come up with reasonable RAM size for a given solr machine. > >> > >> Suppose the index size is at 16GB. > >> The Max heap allocated to JVM is about 12GB. > >> > >> The machine I'm trying now has 24GB. > >> When the machine is running for a while serving production, I can see in > >> top > >> that the resident memory taken by the jvm is indeed at 12gb. > >> Now, on top of this i should assume that if i want the whole index to > fit > >> in > >> disk cache i need about 12gb+16gb = 28GB of RAM just for that. Is this > >> kind > >> of calculation correct or am i off here? > >> > >> > > Hmmm..not quite. The idea of the ram usage isn't to simply hold the index > > in > > memory - if you want this use a RAMDirectory. > > The memory being used will be a combination of various caches (Lucene and > > Solr), index buffers et al., and of course the server itself. The > > specifics > > depend very > > much on what your server is doing at any given time - e.g. lots of > > concurrent searches, lots of indexing, both etc., and how things are > setup > > in your solrconfig.xml. > > > > A really excellent resource that's worth looking at regarding all this > can > > be found here: > > > > > http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Scaling-Lucene-and-Solr > > > > > > > >> Any other recommendations Anyone could make w.r.t these numbers ? > >> > >> Thanks, > >> -Chak > >> -- > >> View this message in context: > >> http://old.nabble.com/Solr-RAM-Requirements-tp27924551p27924551.html > >> Sent from the Solr - User mailing list archive at Nabble.com. > >> > >> > > > > > > -- > View this message in context: > http://old.nabble.com/Solr-RAM-Requirements-tp27924551p27926536.html > Sent from the Solr - User mailing list archive at Nabble.com. > >
