How can one calculate an ideal max shard size for a solr core instance if I am running a cloud with multiple systems of 4GB?
Thanks On 11 June 2013 11:18, Walter Underwood <wun...@wunderwood.org> wrote: > An index does not need to fit into the heap. But a 4GB machine is almost > certainly too small to run Solr with 40 million documents. > > wunder > > On Jun 10, 2013, at 10:36 PM, gururaj kosuru wrote: > > > Hi Walter, > > thanks for replying. Do you mean that it is necessary for > > the index to fit into the heap? if so, will a heap size that is greater > > than the actual RAM size slow down the queries? > > > > Thanks, > > Gururaj > > > > > > On 11 June 2013 10:36, Walter Underwood <wun...@wunderwood.org> wrote: > > > >> 2GB is a rather small heap. Our production systems run with 8GB and > >> smaller indexes than that. Our dev and test systems run with 6GB heaps. > >> > >> wunder > >> > >> On Jun 10, 2013, at 9:52 PM, gururaj kosuru wrote: > >> > >>> Hello, > >>> I have recently started using solr 3.4 and have a standalone > >>> system deployed that has 40,000,000 data rows with 3 indexed fields > >>> totalling around 9 GB. I have given a Heap size of 2GB and I run the > >>> instance on Tomcat on an i7 system with 4 GB RAM. My queries involve > >>> searching among the indexed fields using the keyword 'OR' to search for > >>> multiple words in a single field and the keyword 'AND' to get the > >>> intersection of multiple matches from different fields. The problem I > >> face > >>> is an out of memory exception that happens when the solr core has been > >>> queried for a long time and I am forced to restart the solr instance. > My > >>> primary questions are : > >>> > >>> 1. Can I make any changes in my query as I have noticed that if I > divide > >> my > >>> query into parts eg- if my query has "A and B and C", executing only A > >>> gives out 75,000 results. However, If I run the whole query, I get an > out > >>> of memory error in SegmentNorms.java at line 156 which allocates a new > >> byte > >>> array of size count. > >>> > >>> 2. Does my index need to be able to fit into the RAM in one go? > >>> > >>> 3. Will moving to Solr Cloud solve the out of memory exception issue > and > >> if > >>> so, what will be the ratio of my RAM size to the shard size that should > >> be > >>> used? > >>> > >>> Thanks a lot, > >>> Gururaj > >> > >> -- > >> Walter Underwood > >> wun...@wunderwood.org > >> > >> > >> > >> > > -- > Walter Underwood > wun...@wunderwood.org > > > >
