Multi-core memory problem
Hi, We are using Solr 1.4 running 2 cores each containing ~90M documents. Each core index size on the disk is ~ 120 G. The machine is a 64-bit quad-core 64G RAM running Windows Server 2008. Max heap size is set to 9G for the Tomcat process. Default caches are used. Our queries are complex and involve 8 facet fields (3 of them are boolean) and sorting on up 2 fields in addition to Solr score. I noticed a new behavior that didn't happened before: the first core being queried after startup answers all the queries, even the ones bringing ten millions of the documents, while the other core (that was queried the second) causes OutOfMemory exceptions for any query, even the "smallest" one. The heap is shown as not fully used 6-5 G out of 9. This is very strange because till recently both cores were working well, handling the heaviest queries, while the heap usage was on 8 G. Any ideas? Thanks
Re: Multi-core memory problem
The queries do require sorting (on int) and faceting. They should fetch first 200 docs. The current problematic core has 10 entries in fieldCache and 5 entries in filterCache. The other caches are empty. Is there any way to know how much memory specific cache takes? The problem is that one core behaves well, while the other one throws OutOfMemory exceptions right from the restart. This behavior is consistent if I switch the order of the cores initialization. It feels like there the core initialized second has no memory resources assigned. On Thu, Apr 8, 2010 at 4:26 AM, Lance Norskog wrote: > Sorting takes memory. What data types are the fields sorted on? If > they're strings, that could be a space-eater. If they are ints or > dates, not a problem. > > Do the queries pull all of the documents found? Or do they just fetch > the, for example, first 10 documents? > > What are the cache statistics like? Can they be shrunk? The stats are > shown the Statistics page off of the main solr/admin page. > > Facets come from something called the Lucene Field Cache, which is not > controlled out of Solr. It has no eviction policy. When you do a facet > request, the memory used to load up the facets for a particular field > will not be evicted. So if you have lots and lots of facets, this > could be a problem. > > On Wed, Apr 7, 2010 at 3:45 PM, Victoria Kagansky > wrote: > > Hi, > > We are using Solr 1.4 running 2 cores each containing ~90M documents. > Each > > core index size on the disk is ~ 120 G. > > The machine is a 64-bit quad-core 64G RAM running Windows Server 2008. > > Max heap size is set to 9G for the Tomcat process. Default caches are > used. > > > > Our queries are complex and involve 8 facet fields (3 of them are > boolean) > > and sorting on up 2 fields in addition to Solr score. > > I noticed a new behavior that didn't happened before: the first core > being > > queried after startup answers all the queries, even the ones bringing ten > > millions of the documents, while the other core (that was queried the > > second) causes OutOfMemory exceptions for any query, even the "smallest" > > one. The heap is shown as not fully used 6-5 G out of 9. This is very > > strange because till recently both cores were working well, handling the > > heaviest queries, while the heap usage was on 8 G. > > > > Any ideas? > > > > Thanks > > > > > > -- > Lance Norskog > goks...@gmail.com >
Re: Multi-core memory problem
I noticed now that the OutOfMemory exception occurs upon faceting queries. Queries without facets do return successfully. There are two log types upon the exception. The queries causing them differ only in q parameter, the faceting and sorting parameters are the same. I guess this has something to do with the result set size influencing the faceting mechanism. 1) Apr 8, 2010 9:18:21 AM org.apache.solr.common.SolrException log SEVERE: java.lang.OutOfMemoryError: Java heap space 2) Apr 8, 2010 9:18:13 AM org.apache.solr.common.SolrException log SEVERE: java.lang.OutOfMemoryError: Java heap space at org.apache.solr.request.UnInvertedField.uninvert(UnInvertedField.java:191) at org.apache.solr.request.UnInvertedField.(UnInvertedField.java:178) at org.apache.solr.request.UnInvertedField.getUnInvertedField(UnInvertedField.java:839) at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:250) at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:283) at org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:166) at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:72) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:859) at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:574) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1527) at java.lang.Thread.run(Thread.java:619) On Thu, Apr 8, 2010 at 9:51 AM, Victoria Kagansky < victoria.kagan...@gmail.com> wrote: > The queries do require sorting (on int) and faceting. They should fetch > first 200 docs. > The current problematic core has 10 entries in fieldCache and 5 entries in > filterCache. The other caches are empty. Is there any way to know how much > memory specific cache takes? > > The problem is that one core behaves well, while the other one throws > OutOfMemory exceptions right from the restart. This behavior is consistent > if I switch the order of the cores initialization. It feels like there the > core initialized second has no memory resources assigned. > > > > On Thu, Apr 8, 2010 at 4:26 AM, Lance Norskog wrote: > >> Sorting takes memory. What data types are the fields sorted on? If >> they're strings, that could be a space-eater. If they are ints or >> dates, not a problem. >> >> Do the queries pull all of the documents found? Or do they just fetch >> the, for example, first 10 documents? >> >> What are the cache statistics like? Can they be shrunk? The stats are >> shown the Statistics page off of the main solr/admin page. >> >> Facets come from something called the Lucene Field Cache, which is not >> controlled out of Solr. It has no eviction policy. When you do a facet >> request, the memory used to load up the facets for a particular field >> will not be evicted. So if you have lots and lots of facets, this >> could be a problem. >> >> On Wed, Apr 7, 2010 at 3:45 PM, Victoria Kagansky >> wrote: >> > Hi, >> > We are using Solr 1.4 running 2 cores each containing ~90M documents. >> Each >> > core index size on the disk is ~ 120 G. >> > The machine is a 64-bit quad-core 64G RAM running Windows Server 2008. >> > Max heap size is set to 9G for the Tomcat process. Default caches are >> used. >> > >> > Our queries are complex and involve 8 facet fields (3 of them are >> boolean) >> > and sorting on up 2 fields in addition to Solr score. >> > I n
Re: Multi-core memory problem
Thanks! We enlarged the max heap size and it looks ok so far. On Fri, Apr 9, 2010 at 4:23 AM, Lance Norskog wrote: > Since the facet "cache" is hard-allocated and has not eviction policy, > you could do a facet query on each core as part of the wam-up. This > way, the facets will not fail. At that point, you can tune the Solr > cache sizes. > > Solr caches documents, searches, and filter queries. Filter queries > are sets of documents cached as a bitmap, one bit for every document > (I think). Searches are cached as a sorted list of document numbers > (they include the relevance order, where filters don't care). > Documents cache all of the fields in a document. > > On Thu, Apr 8, 2010 at 3:46 AM, Victoria Kagansky > wrote: > > I noticed now that the OutOfMemory exception occurs upon faceting > queries. > > Queries without facets do return successfully. > > > > There are two log types upon the exception. The queries causing them > differ > > only in q parameter, the faceting and sorting parameters are the same. I > > guess this has something to do with the result set size influencing the > > faceting mechanism. > > > > 1) Apr 8, 2010 9:18:21 AM org.apache.solr.common.SolrException log > > SEVERE: java.lang.OutOfMemoryError: Java heap space > > > > 2) Apr 8, 2010 9:18:13 AM org.apache.solr.common.SolrException log > > SEVERE: java.lang.OutOfMemoryError: Java heap space > >at > > > org.apache.solr.request.UnInvertedField.uninvert(UnInvertedField.java:191) > >at > > org.apache.solr.request.UnInvertedField.(UnInvertedField.java:178) > >at > > > org.apache.solr.request.UnInvertedField.getUnInvertedField(UnInvertedField.java:839) > >at > > org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:250) > >at > > > org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:283) > >at > > > org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:166) > >at > > > org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:72) > >at > > > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) > >at > > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) > >at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) > >at > > > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) > >at > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) > >at > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > >at > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > >at > > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > >at > > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > >at > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) > >at > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > >at > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567) > >at > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > >at > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) > >at > > > org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:859) > >at > > > org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:574) > >at > > org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1527) > >at java.lang.Thread.run(Thread.java:619) > > > > > > > > > > On Thu, Apr 8, 2010 at 9:51 AM, Victoria Kagansky < > > victoria.kagan...@gmail.com> wrote: > > > >> The queries do require sorting (on int) and faceting. They should fetch > >> first 200 docs. > >> The current problematic core has 10 entries in fieldCache and 5 entries > in > >> filterCache. The other caches are empty. Is there any way to know how > much > >> memory specific cache takes? > >> > >> The problem is that one core behaves well, while the other one throws > >> OutOfMemory exceptions right from the restart. This behavior is > consistent > >> if I switch the order of the cores initializ
Combining Dismax and payload boosting
Hi, We are using payloads for score boosting. For this purpose we've implemented custom boosting QueryParser and similarity function. We followed http://www.lucidimagination.com/blog/2009/08/05/getting-started-with-payloads/ . On the other hand, we'd like to use dismax query handling because of its benefits in several fields search. How can we make dismax use our custom QueryParser? Thanks!
