Hi All,
We are trying to load 14+ Billion documents into Solr. But we are failing to
load them into Solr.
Solr version: 4.8.0
Analyzer used: ClassicTokenizer for index as well as query.
Can someone help me in getting into the core of this issue?
For 14+ Billion document load, we are loading 2Billion batches using the
dataimport with single thread.
First batch completed successfully & added 2 Billion documents
Second batch, dataimport is showing as successful completion.
But the no of documents is still 2 Billion with the following exception in the
logs.
Screenshot of Dataimport handler:
[cid:[email protected]]
Exception:
2/4/2015, 1:09:06 PM ERROR SolrWriter Exception while solr commit.
org.apache.solr.common.SolrException: Error opening new searcher
at
org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1550)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1662)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1406)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:611)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1458)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1435)
at
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157)
at
org.apache.solr.handler.dataimport.SolrWriter.commit(SolrWriter.java:108)
at
org.apache.solr.handler.dataimport.DocBuilder.finish(DocBuilder.java:309)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:257)
at
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:444)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:485)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:464)
Caused by: java.lang.IllegalArgumentException: Too many documents, composite
IndexReaders cannot exceed 2147483647
at
org.apache.lucene.index.BaseCompositeReader.<init>(BaseCompositeReader.java:77)
at
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:369)
at
org.apache.lucene.index.StandardDirectoryReader.<init>(StandardDirectoryReader.java:43)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:117)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:385)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:289)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:274)
at
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:251)
at
org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1461)
Please let me know if we need more details on this.
Thanks in advance.
Regards,
Suresh.A,
EMC Corporation.
