bq: maxWarmingSearchers is set to 6
Red flag ref. If this was done to avoid the warning in the logs about
too many warming searchers, it's a clear indication that you're
committing far too often. Let's see exactly what you're using to post
when you say you're "using the REST API". My bet: each one does a
commit. If this is the post.jar tool there's an option to _not_ commit
and I'd be sure to set that and let your autocommit settings handle
committing.
My guess is that you are adding more and more documents to Solr, thus
making it more likely that you are opening a bunch of searchers at
once (see above) and running into a race condition. So the first thing
I'd do is straighten that out and see if the problem goes away.
Best,
Erick
On Tue, Dec 6, 2016 at 1:34 PM, Alexandre Drouin
<alexandre.dro...@orckestra.com> wrote:
> Hello,
>
> I have an error that has been popping up randomly since 3 weeks ago and the
> randomness of the issue makes it hard to troubleshoot.
>
> I have a service that use the REST API to index documents (1000 docs at a
> time) and in this process I often call the core status API
> (/solr/admin/cores?action=STATUS) to get the statuses of the different cores.
> This process has been working flawlessly since 2014 however it has been
> failing recently with the exception: " this IndexWriter is closed".
>
> I did a few search on Google for this exception but I did not see anything
> relevant. Does anyone have an idea how to troubleshoot/fix this issue?
>
> This is my configuration:
> - Solr 4.10.2 on Windows. I am not using SolrCloud.
> - Java 1.7.0_79 24.79-b02
> - useColdSearcher is set to true
> - maxWarmingSearchers is set to 6
> - I changed my Solr configuration about 2-3 months ago: I disabled HTTPS and
> enabled the logging (INFO level) but I do not think this could cause the
> issue.
>
> Relevant stack trace:
>
> INFO - 2016-12-06 18:43:23.854; org.apache.solr.update.CommitTracker; Hard
> AutoCommit: if uncommited for 90000ms; if 75000 uncommited docs
> INFO - 2016-12-06 18:43:23.856; org.apache.solr.update.CommitTracker; Soft
> AutoCommit: if uncommited for 15000ms;
> INFO - 2016-12-06 18:43:23.929;
> org.apache.solr.update.processor.LogUpdateProcessor; [coreENCA] webapp=/solr
> path=/update params={commit=false} {add=[Global_44235 (1552993270510911488),
> Global_44236Pony (1552993270516154368), Global_44236Magnum
> (1552993270518251520), Global_44237Pony (1552993270519300096),
> Global_44237Split (1552993270521397249), Global_44237Standard
> (1552993270523494401), Global_44238Pony (1552993270525591553),
> Global_44238Standard (1552993270527688704), Global_44238Magnum
> (1552993270529785856), Global_44239Standard (1552993270531883008), ... (2102
> adds)]} 0 8292
> INFO - 2016-12-06 18:43:23.933; org.apache.solr.core.SolrCore; [coreENCA]
> CLOSING SolrCore org.apache.solr.core.SolrCore@5730eaaf
> INFO - 2016-12-06 18:43:23.935; org.apache.solr.update.DirectUpdateHandler2;
> closing DirectUpdateHandler2{commits=0,autocommit maxDocs=75000,autocommit
> maxTime=90000ms,autocommits=0,soft autocommit maxTime=15000ms,soft
> autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=4176,adds=4176,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=4176,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=84547858,transaction_logs_total_number=1}
> INFO - 2016-12-06 18:43:23.936; org.apache.solr.core.SolrCore; [coreENCA]
> Closing main searcher on request.
> INFO - 2016-12-06 18:43:24.044; org.apache.solr.search.SolrIndexSearcher;
> Opening Searcher@73a40fbb[coreENCA] main
> ERROR - 2016-12-06 18:43:24.045; org.apache.solr.common.SolrException;
> org.apache.solr.common.SolrException: Error handling 'status' action
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:710)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:214)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:188)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at
> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:729)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:258)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:522)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
> at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter
> is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:698)
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:712)
> at
> org.apache.lucene.index.IndexWriter.nrtIsCurrent(IndexWriter.java:4653)
> at
> org.apache.lucene.index.StandardDirectoryReader.isCurrent(StandardDirectoryReader.java:357)
> at
> org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:562)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.getCoreStatus(CoreAdminHandler.java:1152)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:698)
> ... 31 more
>
>
> Alexandre Drouin
