What's your cluster size? The 2 billion limit is per-node. My personal recommendation is that you don't load more than 100 million documents per node. You need to do a proof of concept test to verify whether your particular data would support a higher number or not. Ultimately, it will not be a matter of how many documents you can load, but whether the query response latency for those documents is sufficient.
-- Jack Krupansky On Wed, Feb 4, 2015 at 4:54 PM, Arumugam, Suresh <suresh.arumu...@emc.com> wrote: > Hi All, > > > > We are trying to load 14+ Billion documents into Solr. But we are failing > to load them into Solr. > > > > Solr version: *4.8.0* > > Analyzer used: *ClassicTokenizer for index as well as query.* > > > > Can someone help me in getting into the core of this issue? > > > > For 14+ Billion document load, we are loading 2Billion batches using the > dataimport with single thread. > > > > First batch completed successfully & added 2 Billion > documents > > Second batch, dataimport is showing as successful > completion. But the no of documents is still 2 Billion with the following > exception in the logs. > > > > *Screenshot of Dataimport handler:* > > > > > > *Exception:* > > > > 2/4/2015, 1:09:06 PM ERROR SolrWriter Exception while solr > commit. > > org.apache.solr.common.SolrException: Error opening new searcher > > at > org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1550) > > at > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1662) > > at > org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1406) > > at > org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:611) > > at > org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95) > > at > org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64) > > at > org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1458) > > at > org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1435) > > at > org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157) > > at > org.apache.solr.handler.dataimport.SolrWriter.commit(SolrWriter.java:108) > > at > org.apache.solr.handler.dataimport.DocBuilder.finish(DocBuilder.java:309) > > at > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:257) > > at > org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:444) > > at > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:485) > > at > org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:464) > > Caused by: java.lang.IllegalArgumentException: Too many documents, > composite IndexReaders cannot exceed 2147483647 > > at > org.apache.lucene.index.BaseCompositeReader.<init>(BaseCompositeReader.java:77) > > at > org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:369) > > at > org.apache.lucene.index.StandardDirectoryReader.<init>(StandardDirectoryReader.java:43) > > at > org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:117) > > at > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:385) > > at > org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:289) > > at > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:274) > > at > org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:251) > > at > org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1461) > > > > > Please let me know if we need more details on this. > > > > Thanks in advance. > > > > Regards, > > Suresh.A, > > EMC Corporation. >
