In fact this (the root cause NPE) is a Lucene bug -- I have a small
test case showing it.
It can happen when you have compressed text fields (Store.COMPRESS) in
the index. I'll open an issue and fix it.
Thank you for raising this!
Mike
Chris Harris wrote:
I've made some changes to my Solr setup, and now I'm getting the
"background merge hit exception" pasted at the end of this message.
The most notable changes I've made are:
Update to r690989 (Lucene r688745)
Change a few things in my schema. In particular, I was previously
storing my main document text and the metadata fields in a single
"body" field, like this:
*****
<!-- This is what we use for the default text search and for
highlighting : -->
<field name="body" type="shingleText" indexed="true" stored="true"
omitNorms="false" multiValued="true" />
<copyField source="companyname" dest="body"/>
<copyField source="date" dest="body"/>
*****
whereas I'm now using "body" as a sort of alias that just gets
redirected to other fields, like this:
*****
<!-- This is only an alias used to populate other fields using
copyField. -->
<field name="body" type="shingleText" indexed="false"
stored="false" />
<!-- This is what we use for the default text search: -->
<field name="everything" type="shingleText" indexed="true"
stored="false" omitNorms="false" multiValued="true" />
<!-- Stored text for use with highlighting -->
<field name="kwic" type="shingleText" indexed="false" stored="true"
compressed="true" multiValued="false" />
<!-- Note: The maxLength parameter is for the copyField max length
patch (SOLR-538). At first I thought
that patch was causing my difficulties, but then
I had the same trouble when I built without that patch -->
<copyField source="body" dest="kwic" maxLength="100000" />
<copyField source="body" dest="everything"/>
<copyField source="companyname" dest="everything"/>
<copyField source="date" dest="everything"/>
*****
When I was indexing with this new setup, things were initially fine,
and segments seemed to be merging fine. I ran into trouble when I sent
an <optimize/>, though. I think in an earlier run I also got a very
similar exception just from document adds, without an explicit
<optimize/>.
I'm also running with a shingle-related patch
(https://issues.apache.org/jira/browse/LUCENE-1370 /
https://issues.apache.org/jira/browse/SOLR-744) and the rich document
handler patch, though I've used these before without trouble.
Is it possible that my schema change is illegitimate? Am I not allowed
to have non-indexed, non-stored fields, for example?
Anyway, here is my stack trace:
*****
background merge hit exception: _1h:C2552 _1i:C210->_1i _1j:C266->_1i
_1k:C214->_1i _1l:C329->_1i _1m:C231->_1i _1n:C379->_1i _1o:C447
_1p:C453->_1p _1q:C485->_1p into _1r [optimize]
java.io.IOException: background merge hit exception: _1h:C2552
_1i:C210->_1i _1j:C266->_1i _1k:C214->_1i _1l:C329->_1i _1m:C231->_1i
_1n:C379->_1i _1o:C447 _1p:C453->_1p _1q:C485->_1p into _1r [optimize]
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:
2303)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:
2233)
at
org
.apache
.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:
355)
at
org
.apache
.solr
.update
.processor
.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:77)
at
org
.apache
.solr
.handler
.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:
228)
at
org
.apache
.solr
.handler
.XmlUpdateRequestHandler
.handleRequestBody(XmlUpdateRequestHandler.java:125)
at
org
.apache
.solr
.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1156)
at
org
.apache
.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:341)
at
org
.apache
.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:272)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1089)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:
365)
at
org
.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:
216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:
181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:
712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:
405)
at
org
.mortbay
.jetty
.handler
.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at
org
.mortbay
.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:
502)
at org.mortbay.jetty.HttpConnection
$RequestHandler.content(HttpConnection.java:835)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at org.mortbay.jetty.bio.SocketConnector
$Connection.run(SocketConnector.java:226)
at org.mortbay.thread.BoundedThreadPool
$PoolThread.run(BoundedThreadPool.java:442)
Caused by: java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at
org
.apache
.lucene
.store.BufferedIndexOutput.writeBytes(BufferedIndexOutput.java:49)
at
org.apache.lucene.index.FieldsWriter.writeField(FieldsWriter.java:215)
at
org.apache.lucene.index.FieldsWriter.addDocument(FieldsWriter.java:
268)
at
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:
359)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:
138)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:
3988)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3637)
at
org
.apache
.lucene
.index
.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:214)
at org.apache.lucene.index.ConcurrentMergeScheduler
$MergeThread.run(ConcurrentMergeScheduler.java:269)
