reg: efficient querying using solr
Hello, I have recently started using solr 3.4 and have a standalone system deployed that has 40,000,000 data rows with 3 indexed fields totalling around 9 GB. I have given a Heap size of 2GB and I run the instance on Tomcat on an i7 system with 4 GB RAM. My queries involve searching among the indexed fields using the keyword 'OR' to search for multiple words in a single field and the keyword 'AND' to get the intersection of multiple matches from different fields. The problem I face is an out of memory exception that happens when the solr core has been queried for a long time and I am forced to restart the solr instance. My primary questions are : 1. Can I make any changes in my query as I have noticed that if I divide my query into parts eg- if my query has "A and B and C", executing only A gives out 75,000 results. However, If I run the whole query, I get an out of memory error in SegmentNorms.java at line 156 which allocates a new byte array of size count. 2. Does my index need to be able to fit into the RAM in one go? 3. Will moving to Solr Cloud solve the out of memory exception issue and if so, what will be the ratio of my RAM size to the shard size that should be used? Thanks a lot, Gururaj
Re: reg: efficient querying using solr
Hi Walter, thanks for replying. Do you mean that it is necessary for the index to fit into the heap? if so, will a heap size that is greater than the actual RAM size slow down the queries? Thanks, Gururaj On 11 June 2013 10:36, Walter Underwood wrote: > 2GB is a rather small heap. Our production systems run with 8GB and > smaller indexes than that. Our dev and test systems run with 6GB heaps. > > wunder > > On Jun 10, 2013, at 9:52 PM, gururaj kosuru wrote: > > > Hello, > > I have recently started using solr 3.4 and have a standalone > > system deployed that has 40,000,000 data rows with 3 indexed fields > > totalling around 9 GB. I have given a Heap size of 2GB and I run the > > instance on Tomcat on an i7 system with 4 GB RAM. My queries involve > > searching among the indexed fields using the keyword 'OR' to search for > > multiple words in a single field and the keyword 'AND' to get the > > intersection of multiple matches from different fields. The problem I > face > > is an out of memory exception that happens when the solr core has been > > queried for a long time and I am forced to restart the solr instance. My > > primary questions are : > > > > 1. Can I make any changes in my query as I have noticed that if I divide > my > > query into parts eg- if my query has "A and B and C", executing only A > > gives out 75,000 results. However, If I run the whole query, I get an out > > of memory error in SegmentNorms.java at line 156 which allocates a new > byte > > array of size count. > > > > 2. Does my index need to be able to fit into the RAM in one go? > > > > 3. Will moving to Solr Cloud solve the out of memory exception issue and > if > > so, what will be the ratio of my RAM size to the shard size that should > be > > used? > > > > Thanks a lot, > > Gururaj > > -- > Walter Underwood > wun...@wunderwood.org > > > >
Re: reg: efficient querying using solr
How can one calculate an ideal max shard size for a solr core instance if I am running a cloud with multiple systems of 4GB? Thanks On 11 June 2013 11:18, Walter Underwood wrote: > An index does not need to fit into the heap. But a 4GB machine is almost > certainly too small to run Solr with 40 million documents. > > wunder > > On Jun 10, 2013, at 10:36 PM, gururaj kosuru wrote: > > > Hi Walter, > > thanks for replying. Do you mean that it is necessary for > > the index to fit into the heap? if so, will a heap size that is greater > > than the actual RAM size slow down the queries? > > > > Thanks, > > Gururaj > > > > > > On 11 June 2013 10:36, Walter Underwood wrote: > > > >> 2GB is a rather small heap. Our production systems run with 8GB and > >> smaller indexes than that. Our dev and test systems run with 6GB heaps. > >> > >> wunder > >> > >> On Jun 10, 2013, at 9:52 PM, gururaj kosuru wrote: > >> > >>> Hello, > >>>I have recently started using solr 3.4 and have a standalone > >>> system deployed that has 40,000,000 data rows with 3 indexed fields > >>> totalling around 9 GB. I have given a Heap size of 2GB and I run the > >>> instance on Tomcat on an i7 system with 4 GB RAM. My queries involve > >>> searching among the indexed fields using the keyword 'OR' to search for > >>> multiple words in a single field and the keyword 'AND' to get the > >>> intersection of multiple matches from different fields. The problem I > >> face > >>> is an out of memory exception that happens when the solr core has been > >>> queried for a long time and I am forced to restart the solr instance. > My > >>> primary questions are : > >>> > >>> 1. Can I make any changes in my query as I have noticed that if I > divide > >> my > >>> query into parts eg- if my query has "A and B and C", executing only A > >>> gives out 75,000 results. However, If I run the whole query, I get an > out > >>> of memory error in SegmentNorms.java at line 156 which allocates a new > >> byte > >>> array of size count. > >>> > >>> 2. Does my index need to be able to fit into the RAM in one go? > >>> > >>> 3. Will moving to Solr Cloud solve the out of memory exception issue > and > >> if > >>> so, what will be the ratio of my RAM size to the shard size that should > >> be > >>> used? > >>> > >>> Thanks a lot, > >>> Gururaj > >> > >> -- > >> Walter Underwood > >> wun...@wunderwood.org > >> > >> > >> > >> > > -- > Walter Underwood > wun...@wunderwood.org > > > >
Re: reg: efficient querying using solr
Thanks Walter, Shawn and Otis for the assistance, I will look into tuning the parameters by experimenting as seems to be the only way to go. On 11 June 2013 19:17, Shawn Heisey wrote: > On 6/11/2013 12:15 AM, gururaj kosuru wrote: > > How can one calculate an ideal max shard size for a solr core instance > if I > > am running a cloud with multiple systems of 4GB? > > That question is impossible to answer without experimentation, but > here's a good starting point. That's all it is, a starting point: > > http://wiki.apache.org/solr/SolrPerformanceProblems > > Thanks, > Shawn > >
Re: reg: efficient querying using solr
Hi Chris, thanks for sharing the document. It is very helpful to have an estimate of what is consuming the memory. On 12 June 2013 10:47, Chris Morley wrote: > This might help (indirectly): > > http://svn.apache.org/repos/asf/lucene/dev/trunk/dev-tools/size-estimator-lu > cene-solr.xls > > ---- > From: "gururaj kosuru" > Sent: Wednesday, June 12, 2013 12:28 AM > To: "solr-user" > Subject: Re: reg: efficient querying using solr > > Thanks Walter, Shawn and Otis for the assistance, I will look into tuning > the parameters by experimenting as seems to be the only way to go. > > On 11 June 2013 19:17, Shawn Heisey wrote: > > > On 6/11/2013 12:15 AM, gururaj kosuru wrote: > > > How can one calculate an ideal max shard size for a solr core instance > > if I > > > am running a cloud with multiple systems of 4GB? > > > > That question is impossible to answer without experimentation, but > > here's a good starting point. That's all it is, a starting point: > > > > http://wiki.apache.org/solr/SolrPerformanceProblems > > > > Thanks, > > Shawn > > > > > >
503 - server is shutting down error
Hi, I am trying to run solr 4.3 on a standalone system using tomcat 6 and I am facing a ExceptionInInitializerError. I am attaching the trace of the log for details. I also get a log4j error(no such file or directory), but I think it is not related to this. The error logs says: [main] ERROR org.apache.solr.core.SolrCore ? null:java.lang.ExceptionInInitializerError at org.apache.solr.core.SolrResourceLoader.reloadLuceneSPI(SolrResourceLoader.java:201) at org.apache.solr.core.SolrResourceLoader.(SolrResourceLoader.java:115) at org.apache.solr.core.SolrResourceLoader.(SolrResourceLoader.java:250) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:380) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:358) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:326) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:124) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:422) at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:115) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4072) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4726) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065) at org.apache.catalina.core.StandardHost.start(StandardHost.java:840) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) at org.apache.catalina.core.StandardService.start(StandardService.java:525) at org.apache.catalina.core.StandardServer.start(StandardServer.java:754) at org.apache.catalina.startup.Catalina.start(Catalina.java:595) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: java.lang.ClassCastException: class org.apache.lucene.facet.codecs.facet42.Facet42DocValuesFormat at java.lang.Class.asSubclass(Class.java:3027) at org.apache.lucene.util.SPIClassIterator.next(SPIClassIterator.java:137) at org.apache.lucene.util.NamedSPILoader.reload(NamedSPILoader.java:65) at org.apache.lucene.util.NamedSPILoader.(NamedSPILoader.java:47) at org.apache.lucene.util.NamedSPILoader.(NamedSPILoader.java:37) at org.apache.lucene.codecs.DocValuesFormat.(DocValuesFormat.java:43) ... 34 more Any help is appreciated. Thanks, Gururaj
