I noticed now that the OutOfMemory exception occurs upon faceting queries.
Queries without facets do return successfully.
There are two log types upon the exception. The queries causing them differ
only in q parameter, the faceting and sorting parameters are the same. I
guess this has something to do with the result set size influencing the
faceting mechanism.
1) Apr 8, 2010 9:18:21 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
2) Apr 8, 2010 9:18:13 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
at
org.apache.solr.request.UnInvertedField.uninvert(UnInvertedField.java:191)
at
org.apache.solr.request.UnInvertedField.<init>(UnInvertedField.java:178)
at
org.apache.solr.request.UnInvertedField.getUnInvertedField(UnInvertedField.java:839)
at
org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:250)
at
org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:283)
at
org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:166)
at
org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:72)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:859)
at
org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:574)
at
org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1527)
at java.lang.Thread.run(Thread.java:619)
On Thu, Apr 8, 2010 at 9:51 AM, Victoria Kagansky <
victoria.kagan...@gmail.com> wrote:
> The queries do require sorting (on int) and faceting. They should fetch
> first 200 docs.
> The current problematic core has 10 entries in fieldCache and 5 entries in
> filterCache. The other caches are empty. Is there any way to know how much
> memory specific cache takes?
>
> The problem is that one core behaves well, while the other one throws
> OutOfMemory exceptions right from the restart. This behavior is consistent
> if I switch the order of the cores initialization. It feels like there the
> core initialized second has no memory resources assigned.
>
>
>
> On Thu, Apr 8, 2010 at 4:26 AM, Lance Norskog <goks...@gmail.com> wrote:
>
>> Sorting takes memory. What data types are the fields sorted on? If
>> they're strings, that could be a space-eater. If they are ints or
>> dates, not a problem.
>>
>> Do the queries pull all of the documents found? Or do they just fetch
>> the, for example, first 10 documents?
>>
>> What are the cache statistics like? Can they be shrunk? The stats are
>> shown the Statistics page off of the main solr/admin page.
>>
>> Facets come from something called the Lucene Field Cache, which is not
>> controlled out of Solr. It has no eviction policy. When you do a facet
>> request, the memory used to load up the facets for a particular field
>> will not be evicted. So if you have lots and lots of facets, this
>> could be a problem.
>>
>> On Wed, Apr 7, 2010 at 3:45 PM, Victoria Kagansky
>> <victoria.kagan...@gmail.com> wrote:
>> > Hi,
>> > We are using Solr 1.4 running 2 cores each containing ~90M documents.
>> Each
>> > core index size on the disk is ~ 120 G.
>> > The machine is a 64-bit quad-core 64G RAM running Windows Server 2008.
>> > Max heap size is set to 9G for the Tomcat process. Default caches are
>> used.
>> >
>> > Our queries are complex and involve 8 facet fields (3 of them are
>> boolean)
>> > and sorting on up 2 fields in addition to Solr score.
>> > I noticed a new behavior that didn't happened before: the first core
>> being
>> > queried after startup answers all the queries, even the ones bringing
>> ten
>> > millions of the documents, while the other core (that was queried the
>> > second) causes OutOfMemory exceptions for any query, even the "smallest"
>> > one. The heap is shown as not fully used 6-5 G out of 9. This is very
>> > strange because till recently both cores were working well, handling the
>> > heaviest queries, while the heap usage was on 8 G.
>> >
>> > Any ideas?
>> >
>> > Thanks
>> >
>>
>>
>>
>> --
>> Lance Norskog
>> goks...@gmail.com
>>
>
>
