See: https://issues.apache.org/jira/browse/SOLR-12646
On Wed, Aug 8, 2018 at 11:24 AM, Bjarke Buur Mortensen <morten...@eluence.com> wrote: > OK, thanks. > > As long as it's my dev box, reindexing is fine. > I just hope that my assumption holds, that our prod solr is 7x segments > only. > > Thanks again, > Bjarke > > 2018-08-08 20:03 GMT+02:00 Erick Erickson <erickerick...@gmail.com>: > >> Bjarke: >> >> Using SPLITSHARD on an index with 6x segments just seems to not work, >> even outside the standalone-> cloud issue. I'll raise a JIRA. >> Meanwhile I think you'll have to re-index I'm afraid. >> >> Thanks for raising the issue. >> >> Erick >> >> On Wed, Aug 8, 2018 at 6:34 AM, Bjarke Buur Mortensen >> <morten...@eluence.com> wrote: >> > Erick, >> > >> > thanks, that is of course something I left out of the original question. >> > Our Solr is 7.1, so that should not present a problem (crossing fingers). >> > >> > However, on my dev box I'm trying out the steps, and here I have some >> > segments created with version 6 of Solr. >> > >> > After having copied data from my non-cloud solr into my >> > single-shard-single-replica collection and verified that Solr Cloud works >> > with this collection, I then submit the splitshard command >> > >> > http://172.17.0.4:8984 >> > /solr/admin/collections?action=SPLITSHARD&collection= >> procurement&shard=shard1 >> > >> > However, this gives me the error: >> > org.apache.solr.client.solrj.impl.HttpSolrClient$ >> RemoteSolrException:Error >> > from server at http://172.17.0.4:8984/solr: >> > java.lang.IllegalArgumentException: Cannot merge a segment that has been >> > created with major version 6 into this index which has been created by >> > major version 7"} >> > >> > I have tried running both optimize and IndexUpgrader on the index before >> > shard splitting, but the same error still occurs. >> > >> > Any ideas as to why this happens? >> > >> > Below is an output from running IndexUpgrader, which I cannot decipher. >> > It both states that "All segments upgraded to version 7.1.0" and ''all >> > running merges have aborted" ¯\_(ツ)_/¯ >> > >> > Thanks a lot, >> > Bjarke >> > >> > >> > ====================== >> > java -cp >> > /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/lucene- >> backward-codecs-7.1.0.jar:/opt/solr/server/solr-webapp/ >> webapp/WEB-INF/lib/lucene-core-7.1.0.jar >> > org.apache.lucene.index.IndexUpgrader -delete-prior-commits -verbose >> > /var/solr/cloud/procurement_shard1_replica_n1/data/index >> > IFD 0 [2018-08-08T13:00:18.244Z; main]: init: current segments file is >> > "segments_4vs"; >> > deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPoli >> cy@721e0f4f >> > IFD 0 [2018-08-08T13:00:18.266Z; main]: init: load commit "segments_4vs" >> > IFD 0 [2018-08-08T13:00:18.270Z; main]: now checkpoint >> > "_bhg(7.1.0):C108396" [1 segments ; isCommit = false] >> > IFD 0 [2018-08-08T13:00:18.270Z; main]: 0 msec to checkpoint >> > IW 0 [2018-08-08T13:00:18.270Z; main]: init: create=false >> > IW 0 [2018-08-08T13:00:18.273Z; main]: >> > dir=MMapDirectory@/var/solr/cloud/procurement_shard1_ >> replica_n1/data/index >> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@6debcae2 >> > index=_bhg(7.1.0):C108396 >> > version=7.1.0 >> > analyzer=null >> > ramBufferSizeMB=16.0 >> > maxBufferedDocs=-1 >> > mergedSegmentWarmer=null >> > delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy >> > commit=null >> > openMode=CREATE_OR_APPEND >> > similarity=org.apache.lucene.search.similarities.BM25Similarity >> > mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=-1, >> > maxMergeCount=-1, ioThrottle=true >> > codec=Lucene70 >> > infoStream=org.apache.lucene.util.PrintStreamInfoStream >> > mergePolicy=UpgradeIndexMergePolicy([TieredMergePolicy: >> maxMergeAtOnce=10, >> > maxMergeAtOnceExplicit=30, maxMergedSegmentMB=5120.0, floorSegmentMB=2.0, >> > forceMergeDeletesPctAllowed=10.0, segmentsPerTier=10.0, >> > maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1) >> > indexerThreadPool=org.apache.lucene.index.DocumentsWriterPerThreadPool@ >> 5ba23b66 >> > readerPooling=true >> > perThreadHardLimitMB=1945 >> > useCompoundFile=true >> > commitOnClose=true >> > indexSort=null >> > writer=org.apache.lucene.index.IndexWriter@2ff4f00f >> > >> > IW 0 [2018-08-08T13:00:18.273Z; main]: MMapDirectory.UNMAP_SUPPORTED= >> true >> > IndexUpgrader 0 [2018-08-08T13:00:18.274Z; main]: Upgrading all pre-7.1.0 >> > segments of index directory >> > 'MMapDirectory@/var/solr/cloud/procurement_shard1_replica_n1/data/index >> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@6debcae2' to >> > version 7.1.0... >> > IW 0 [2018-08-08T13:00:18.274Z; main]: forceMerge: index now >> > _bhg(7.1.0):C108396 >> > IW 0 [2018-08-08T13:00:18.274Z; main]: now flush at forceMerge >> > IW 0 [2018-08-08T13:00:18.274Z; main]: start flush: >> applyAllDeletes=true >> > IW 0 [2018-08-08T13:00:18.274Z; main]: index before flush >> > _bhg(7.1.0):C108396 >> > DW 0 [2018-08-08T13:00:18.274Z; main]: startFullFlush >> > DW 0 [2018-08-08T13:00:18.275Z; main]: main finishFullFlush success=true >> > IW 0 [2018-08-08T13:00:18.275Z; main]: now apply all deletes for all >> > segments buffered updates bytesUsed=0 reader pool bytesUsed=0 >> > BD 0 [2018-08-08T13:00:18.275Z; main]: waitApply: no deletes to apply >> > UPGMP 0 [2018-08-08T13:00:18.276Z; main]: findForcedMerges: >> > segmentsToUpgrade={} >> > MS 0 [2018-08-08T13:00:18.282Z; main]: initDynamicDefaults spins=true >> > maxThreadCount=1 maxMergeCount=6 >> > MS 0 [2018-08-08T13:00:18.282Z; main]: now merge >> > MS 0 [2018-08-08T13:00:18.282Z; main]: index: _bhg(7.1.0):C108396 >> > MS 0 [2018-08-08T13:00:18.282Z; main]: no more merges pending; now >> return >> > IndexUpgrader 0 [2018-08-08T13:00:18.282Z; main]: All segments upgraded >> to >> > version 7.1.0 >> > IndexUpgrader 0 [2018-08-08T13:00:18.283Z; main]: Enforcing commit to >> > rewrite all index metadata... >> > IW 0 [2018-08-08T13:00:18.283Z; main]: commit: start >> > IW 0 [2018-08-08T13:00:18.283Z; main]: commit: enter lock >> > IW 0 [2018-08-08T13:00:18.283Z; main]: commit: now prepare >> > IW 0 [2018-08-08T13:00:18.283Z; main]: prepareCommit: flush >> > IW 0 [2018-08-08T13:00:18.283Z; main]: index before flush >> > _bhg(7.1.0):C108396 >> > DW 0 [2018-08-08T13:00:18.283Z; main]: startFullFlush >> > IW 0 [2018-08-08T13:00:18.283Z; main]: now apply all deletes for all >> > segments buffered updates bytesUsed=0 reader pool bytesUsed=0 >> > BD 0 [2018-08-08T13:00:18.283Z; main]: waitApply: no deletes to apply >> > DW 0 [2018-08-08T13:00:18.284Z; main]: main finishFullFlush success=true >> > IW 0 [2018-08-08T13:00:18.284Z; main]: startCommit(): start >> > IW 0 [2018-08-08T13:00:18.284Z; main]: startCommit >> > index=_bhg(7.1.0):C108396 changeCount=2 >> > IW 0 [2018-08-08T13:00:18.293Z; main]: startCommit: wrote pending >> segments >> > file "pending_segments_4vt" >> > IW 0 [2018-08-08T13:00:18.295Z; main]: done all syncs: >> > [_bhg_Lucene50_0.tip, _bhg.fdx, _bhg.fnm, _bhg.nvm, _bhg.fdt, _bhg.si, >> > _bhg_Lucene50_0.pos, _bhg.nvd, _bhg_Lucene50_0.doc, _bhg_Lucene50_0.tim] >> > IW 0 [2018-08-08T13:00:18.295Z; main]: commit: pendingCommit != null >> > IW 0 [2018-08-08T13:00:18.298Z; main]: commit: done writing segments file >> > "segments_4vt" >> > IFD 0 [2018-08-08T13:00:18.298Z; main]: now checkpoint >> > "_bhg(7.1.0):C108396" [1 segments ; isCommit = true] >> > IFD 0 [2018-08-08T13:00:18.298Z; main]: deleteCommits: now decRef commit >> > "segments_4vs" >> > IFD 0 [2018-08-08T13:00:18.298Z; main]: delete [segments_4vs] >> > IFD 0 [2018-08-08T13:00:18.299Z; main]: 0 msec to checkpoint >> > IW 0 [2018-08-08T13:00:18.319Z; main]: commit: took 16.0 msec >> > IW 0 [2018-08-08T13:00:18.319Z; main]: commit: done >> > IndexUpgrader 0 [2018-08-08T13:00:18.319Z; main]: Committed upgraded >> > metadata to index. >> > IW 0 [2018-08-08T13:00:18.319Z; main]: now flush at close >> > IW 0 [2018-08-08T13:00:18.319Z; main]: start flush: >> applyAllDeletes=true >> > IW 0 [2018-08-08T13:00:18.319Z; main]: index before flush >> > _bhg(7.1.0):C108396 >> > DW 0 [2018-08-08T13:00:18.319Z; main]: startFullFlush >> > DW 0 [2018-08-08T13:00:18.320Z; main]: main finishFullFlush success=true >> > IW 0 [2018-08-08T13:00:18.320Z; main]: now apply all deletes for all >> > segments buffered updates bytesUsed=0 reader pool bytesUsed=0 >> > BD 0 [2018-08-08T13:00:18.320Z; main]: waitApply: no deletes to apply >> > MS 0 [2018-08-08T13:00:18.320Z; main]: updateMergeThreads ioThrottle=true >> > targetMBPerSec=10240.0 MB/sec >> > MS 0 [2018-08-08T13:00:18.320Z; main]: now merge >> > MS 0 [2018-08-08T13:00:18.321Z; main]: index: _bhg(7.1.0):C108396 >> > MS 0 [2018-08-08T13:00:18.321Z; main]: no more merges pending; now >> return >> > IW 0 [2018-08-08T13:00:18.321Z; main]: waitForMerges >> > IW 0 [2018-08-08T13:00:18.321Z; main]: waitForMerges done >> > IW 0 [2018-08-08T13:00:18.321Z; main]: commit: start >> > IW 0 [2018-08-08T13:00:18.321Z; main]: commit: enter lock >> > IW 0 [2018-08-08T13:00:18.321Z; main]: commit: now prepare >> > IW 0 [2018-08-08T13:00:18.321Z; main]: prepareCommit: flush >> > IW 0 [2018-08-08T13:00:18.321Z; main]: index before flush >> > _bhg(7.1.0):C108396 >> > DW 0 [2018-08-08T13:00:18.321Z; main]: startFullFlush >> > IW 0 [2018-08-08T13:00:18.321Z; main]: now apply all deletes for all >> > segments buffered updates bytesUsed=0 reader pool bytesUsed=0 >> > BD 0 [2018-08-08T13:00:18.322Z; main]: waitApply: no deletes to apply >> > DW 0 [2018-08-08T13:00:18.322Z; main]: main finishFullFlush success=true >> > IW 0 [2018-08-08T13:00:18.322Z; main]: startCommit(): start >> > IW 0 [2018-08-08T13:00:18.322Z; main]: skip startCommit(): no changes >> > pending >> > IW 0 [2018-08-08T13:00:18.322Z; main]: commit: pendingCommit == null; >> skip >> > IW 0 [2018-08-08T13:00:18.322Z; main]: commit: took 0.9 msec >> > IW 0 [2018-08-08T13:00:18.322Z; main]: commit: done >> > IW 0 [2018-08-08T13:00:18.322Z; main]: rollback >> > IW 0 [2018-08-08T13:00:18.322Z; main]: all running merges have aborted >> > IW 0 [2018-08-08T13:00:18.323Z; main]: rollback: done finish merges >> > DW 0 [2018-08-08T13:00:18.323Z; main]: abort >> > DW 0 [2018-08-08T13:00:18.323Z; main]: done abort success=true >> > IW 0 [2018-08-08T13:00:18.323Z; main]: rollback: >> infos=_bhg(7.1.0):C108396 >> > IFD 0 [2018-08-08T13:00:18.323Z; main]: now checkpoint >> > "_bhg(7.1.0):C108396" [1 segments ; isCommit = false] >> > IFD 0 [2018-08-08T13:00:18.323Z; main]: 0 msec to checkpoint >> > >> > 2018-08-07 16:38 GMT+02:00 Erick Erickson <erickerick...@gmail.com>: >> > >> >> Bjarke: >> >> >> >> One thing, what version of Solr are you moving _from_ and _to_? >> >> Solr/Lucene only guarantee one major backward revision so you can copy >> >> an index created with Solr 6 to another Solr 6 or Solr 7, but you >> >> couldn't copy an index created with Solr 5 to Solr 7... >> >> >> >> Also note that shard splitting is a very expensive operation, so be >> >> patient.... >> >> >> >> Best, >> >> Erick >> >> >> >> On Tue, Aug 7, 2018 at 6:17 AM, Rahul Singh >> >> <rahul.xavier.si...@gmail.com> wrote: >> >> > Bjarke, >> >> > >> >> > I am imagining that at some point you may need to shard that data if >> it >> >> grows. Or do you imagine this data to remain stagnant? >> >> > >> >> > Generally you want to add solrcloud to do two things : 1. Increase >> >> availability with replicas 2. Increase available data via shards 3. >> >> Increase fault tolerance with leader and replicas being spread around >> the >> >> cluster. >> >> > >> >> > You would be bypassing general High availability / distributed >> computing >> >> processes by trying to not reindex. >> >> > >> >> > Rahul >> >> > On Aug 7, 2018, 7:06 AM -0400, Bjarke Buur Mortensen < >> >> morten...@eluence.com>, wrote: >> >> >> Hi List, >> >> >> >> >> >> is there a cookbook recipe for moving an existing solr core to a solr >> >> cloud >> >> >> collection. >> >> >> >> >> >> We currently have a single machine with a large core (~150gb), and we >> >> would >> >> >> like to move to solr cloud. >> >> >> >> >> >> I haven't been able to find anything that reuses an existing index, >> so >> >> any >> >> >> pointers much appreciated. >> >> >> >> >> >> Thanks, >> >> >> Bjarke >> >> >>
