Hi Erick, Is there any workaround for this? My requirement is to move solr data from solr 6 to solr 7.
I have thought to use /replication handler to do so. But, as per the case, the solr 6 collection has 1 shard & 1 replica and we wish to have 2 shards and 1 replica on solr 7. If I enable replication on collection having 2 shards, the same data from solr 6 is copied to both the shards. i.e data becomes double. However, if I enable replication on collection with 1 shard and later try to SPLITSHARD, it failed with below error even after using Index Upgrade Tool/Optimization: *Caused by: java.lang.IllegalArgumentException: Cannot merge a segment that has been created with major version 6 into this index which has been created by major version 7* Regards Kamal On Wed, Aug 8, 2018 at 11:58 PM Erick Erickson <erickerick...@gmail.com> wrote: > See: https://issues.apache.org/jira/browse/SOLR-12646 > > On Wed, Aug 8, 2018 at 11:24 AM, Bjarke Buur Mortensen > <morten...@eluence.com> wrote: > > OK, thanks. > > > > As long as it's my dev box, reindexing is fine. > > I just hope that my assumption holds, that our prod solr is 7x segments > > only. > > > > Thanks again, > > Bjarke > > > > 2018-08-08 20:03 GMT+02:00 Erick Erickson <erickerick...@gmail.com>: > > > >> Bjarke: > >> > >> Using SPLITSHARD on an index with 6x segments just seems to not work, > >> even outside the standalone-> cloud issue. I'll raise a JIRA. > >> Meanwhile I think you'll have to re-index I'm afraid. > >> > >> Thanks for raising the issue. > >> > >> Erick > >> > >> On Wed, Aug 8, 2018 at 6:34 AM, Bjarke Buur Mortensen > >> <morten...@eluence.com> wrote: > >> > Erick, > >> > > >> > thanks, that is of course something I left out of the original > question. > >> > Our Solr is 7.1, so that should not present a problem (crossing > fingers). > >> > > >> > However, on my dev box I'm trying out the steps, and here I have some > >> > segments created with version 6 of Solr. > >> > > >> > After having copied data from my non-cloud solr into my > >> > single-shard-single-replica collection and verified that Solr Cloud > works > >> > with this collection, I then submit the splitshard command > >> > > >> > http://172.17.0.4:8984 > >> > /solr/admin/collections?action=SPLITSHARD&collection= > >> procurement&shard=shard1 > >> > > >> > However, this gives me the error: > >> > org.apache.solr.client.solrj.impl.HttpSolrClient$ > >> RemoteSolrException:Error > >> > from server at http://172.17.0.4:8984/solr: > >> > java.lang.IllegalArgumentException: Cannot merge a segment that has > been > >> > created with major version 6 into this index which has been created by > >> > major version 7"} > >> > > >> > I have tried running both optimize and IndexUpgrader on the index > before > >> > shard splitting, but the same error still occurs. > >> > > >> > Any ideas as to why this happens? > >> > > >> > Below is an output from running IndexUpgrader, which I cannot > decipher. > >> > It both states that "All segments upgraded to version 7.1.0" and ''all > >> > running merges have aborted" ¯\_(ツ)_/¯ > >> > > >> > Thanks a lot, > >> > Bjarke > >> > > >> > > >> > ====================== > >> > java -cp > >> > /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/lucene- > >> backward-codecs-7.1.0.jar:/opt/solr/server/solr-webapp/ > >> webapp/WEB-INF/lib/lucene-core-7.1.0.jar > >> > org.apache.lucene.index.IndexUpgrader -delete-prior-commits -verbose > >> > /var/solr/cloud/procurement_shard1_replica_n1/data/index > >> > IFD 0 [2018-08-08T13:00:18.244Z; main]: init: current segments file is > >> > "segments_4vs"; > >> > deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPoli > >> cy@721e0f4f > >> > IFD 0 [2018-08-08T13:00:18.266Z; main]: init: load commit > "segments_4vs" > >> > IFD 0 [2018-08-08T13:00:18.270Z; main]: now checkpoint > >> > "_bhg(7.1.0):C108396" [1 segments ; isCommit = false] > >> > IFD 0 [2018-08-08T13:00:18.270Z; main]: 0 msec to checkpoint > >> > IW 0 [2018-08-08T13:00:18.270Z; main]: init: create=false > >> > IW 0 [2018-08-08T13:00:18.273Z; main]: > >> > dir=MMapDirectory@/var/solr/cloud/procurement_shard1_ > >> replica_n1/data/index > >> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@6debcae2 > >> > index=_bhg(7.1.0):C108396 > >> > version=7.1.0 > >> > analyzer=null > >> > ramBufferSizeMB=16.0 > >> > maxBufferedDocs=-1 > >> > mergedSegmentWarmer=null > >> > delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy > >> > commit=null > >> > openMode=CREATE_OR_APPEND > >> > similarity=org.apache.lucene.search.similarities.BM25Similarity > >> > mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=-1, > >> > maxMergeCount=-1, ioThrottle=true > >> > codec=Lucene70 > >> > infoStream=org.apache.lucene.util.PrintStreamInfoStream > >> > mergePolicy=UpgradeIndexMergePolicy([TieredMergePolicy: > >> maxMergeAtOnce=10, > >> > maxMergeAtOnceExplicit=30, maxMergedSegmentMB=5120.0, > floorSegmentMB=2.0, > >> > forceMergeDeletesPctAllowed=10.0, segmentsPerTier=10.0, > >> > maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1) > >> > > indexerThreadPool=org.apache.lucene.index.DocumentsWriterPerThreadPool@ > >> 5ba23b66 > >> > readerPooling=true > >> > perThreadHardLimitMB=1945 > >> > useCompoundFile=true > >> > commitOnClose=true > >> > indexSort=null > >> > writer=org.apache.lucene.index.IndexWriter@2ff4f00f > >> > > >> > IW 0 [2018-08-08T13:00:18.273Z; main]: MMapDirectory.UNMAP_SUPPORTED= > >> true > >> > IndexUpgrader 0 [2018-08-08T13:00:18.274Z; main]: Upgrading all > pre-7.1.0 > >> > segments of index directory > >> > 'MMapDirectory@ > /var/solr/cloud/procurement_shard1_replica_n1/data/index > >> > lockFactory=org.apache.lucene.store.NativeFSLockFactory@6debcae2' to > >> > version 7.1.0... > >> > IW 0 [2018-08-08T13:00:18.274Z; main]: forceMerge: index now > >> > _bhg(7.1.0):C108396 > >> > IW 0 [2018-08-08T13:00:18.274Z; main]: now flush at forceMerge > >> > IW 0 [2018-08-08T13:00:18.274Z; main]: start flush: > >> applyAllDeletes=true > >> > IW 0 [2018-08-08T13:00:18.274Z; main]: index before flush > >> > _bhg(7.1.0):C108396 > >> > DW 0 [2018-08-08T13:00:18.274Z; main]: startFullFlush > >> > DW 0 [2018-08-08T13:00:18.275Z; main]: main finishFullFlush > success=true > >> > IW 0 [2018-08-08T13:00:18.275Z; main]: now apply all deletes for all > >> > segments buffered updates bytesUsed=0 reader pool bytesUsed=0 > >> > BD 0 [2018-08-08T13:00:18.275Z; main]: waitApply: no deletes to apply > >> > UPGMP 0 [2018-08-08T13:00:18.276Z; main]: findForcedMerges: > >> > segmentsToUpgrade={} > >> > MS 0 [2018-08-08T13:00:18.282Z; main]: initDynamicDefaults spins=true > >> > maxThreadCount=1 maxMergeCount=6 > >> > MS 0 [2018-08-08T13:00:18.282Z; main]: now merge > >> > MS 0 [2018-08-08T13:00:18.282Z; main]: index: _bhg(7.1.0):C108396 > >> > MS 0 [2018-08-08T13:00:18.282Z; main]: no more merges pending; now > >> return > >> > IndexUpgrader 0 [2018-08-08T13:00:18.282Z; main]: All segments > upgraded > >> to > >> > version 7.1.0 > >> > IndexUpgrader 0 [2018-08-08T13:00:18.283Z; main]: Enforcing commit to > >> > rewrite all index metadata... > >> > IW 0 [2018-08-08T13:00:18.283Z; main]: commit: start > >> > IW 0 [2018-08-08T13:00:18.283Z; main]: commit: enter lock > >> > IW 0 [2018-08-08T13:00:18.283Z; main]: commit: now prepare > >> > IW 0 [2018-08-08T13:00:18.283Z; main]: prepareCommit: flush > >> > IW 0 [2018-08-08T13:00:18.283Z; main]: index before flush > >> > _bhg(7.1.0):C108396 > >> > DW 0 [2018-08-08T13:00:18.283Z; main]: startFullFlush > >> > IW 0 [2018-08-08T13:00:18.283Z; main]: now apply all deletes for all > >> > segments buffered updates bytesUsed=0 reader pool bytesUsed=0 > >> > BD 0 [2018-08-08T13:00:18.283Z; main]: waitApply: no deletes to apply > >> > DW 0 [2018-08-08T13:00:18.284Z; main]: main finishFullFlush > success=true > >> > IW 0 [2018-08-08T13:00:18.284Z; main]: startCommit(): start > >> > IW 0 [2018-08-08T13:00:18.284Z; main]: startCommit > >> > index=_bhg(7.1.0):C108396 changeCount=2 > >> > IW 0 [2018-08-08T13:00:18.293Z; main]: startCommit: wrote pending > >> segments > >> > file "pending_segments_4vt" > >> > IW 0 [2018-08-08T13:00:18.295Z; main]: done all syncs: > >> > [_bhg_Lucene50_0.tip, _bhg.fdx, _bhg.fnm, _bhg.nvm, _bhg.fdt, _bhg.si > , > >> > _bhg_Lucene50_0.pos, _bhg.nvd, _bhg_Lucene50_0.doc, > _bhg_Lucene50_0.tim] > >> > IW 0 [2018-08-08T13:00:18.295Z; main]: commit: pendingCommit != null > >> > IW 0 [2018-08-08T13:00:18.298Z; main]: commit: done writing segments > file > >> > "segments_4vt" > >> > IFD 0 [2018-08-08T13:00:18.298Z; main]: now checkpoint > >> > "_bhg(7.1.0):C108396" [1 segments ; isCommit = true] > >> > IFD 0 [2018-08-08T13:00:18.298Z; main]: deleteCommits: now decRef > commit > >> > "segments_4vs" > >> > IFD 0 [2018-08-08T13:00:18.298Z; main]: delete [segments_4vs] > >> > IFD 0 [2018-08-08T13:00:18.299Z; main]: 0 msec to checkpoint > >> > IW 0 [2018-08-08T13:00:18.319Z; main]: commit: took 16.0 msec > >> > IW 0 [2018-08-08T13:00:18.319Z; main]: commit: done > >> > IndexUpgrader 0 [2018-08-08T13:00:18.319Z; main]: Committed upgraded > >> > metadata to index. > >> > IW 0 [2018-08-08T13:00:18.319Z; main]: now flush at close > >> > IW 0 [2018-08-08T13:00:18.319Z; main]: start flush: > >> applyAllDeletes=true > >> > IW 0 [2018-08-08T13:00:18.319Z; main]: index before flush > >> > _bhg(7.1.0):C108396 > >> > DW 0 [2018-08-08T13:00:18.319Z; main]: startFullFlush > >> > DW 0 [2018-08-08T13:00:18.320Z; main]: main finishFullFlush > success=true > >> > IW 0 [2018-08-08T13:00:18.320Z; main]: now apply all deletes for all > >> > segments buffered updates bytesUsed=0 reader pool bytesUsed=0 > >> > BD 0 [2018-08-08T13:00:18.320Z; main]: waitApply: no deletes to apply > >> > MS 0 [2018-08-08T13:00:18.320Z; main]: updateMergeThreads > ioThrottle=true > >> > targetMBPerSec=10240.0 MB/sec > >> > MS 0 [2018-08-08T13:00:18.320Z; main]: now merge > >> > MS 0 [2018-08-08T13:00:18.321Z; main]: index: _bhg(7.1.0):C108396 > >> > MS 0 [2018-08-08T13:00:18.321Z; main]: no more merges pending; now > >> return > >> > IW 0 [2018-08-08T13:00:18.321Z; main]: waitForMerges > >> > IW 0 [2018-08-08T13:00:18.321Z; main]: waitForMerges done > >> > IW 0 [2018-08-08T13:00:18.321Z; main]: commit: start > >> > IW 0 [2018-08-08T13:00:18.321Z; main]: commit: enter lock > >> > IW 0 [2018-08-08T13:00:18.321Z; main]: commit: now prepare > >> > IW 0 [2018-08-08T13:00:18.321Z; main]: prepareCommit: flush > >> > IW 0 [2018-08-08T13:00:18.321Z; main]: index before flush > >> > _bhg(7.1.0):C108396 > >> > DW 0 [2018-08-08T13:00:18.321Z; main]: startFullFlush > >> > IW 0 [2018-08-08T13:00:18.321Z; main]: now apply all deletes for all > >> > segments buffered updates bytesUsed=0 reader pool bytesUsed=0 > >> > BD 0 [2018-08-08T13:00:18.322Z; main]: waitApply: no deletes to apply > >> > DW 0 [2018-08-08T13:00:18.322Z; main]: main finishFullFlush > success=true > >> > IW 0 [2018-08-08T13:00:18.322Z; main]: startCommit(): start > >> > IW 0 [2018-08-08T13:00:18.322Z; main]: skip startCommit(): no > changes > >> > pending > >> > IW 0 [2018-08-08T13:00:18.322Z; main]: commit: pendingCommit == null; > >> skip > >> > IW 0 [2018-08-08T13:00:18.322Z; main]: commit: took 0.9 msec > >> > IW 0 [2018-08-08T13:00:18.322Z; main]: commit: done > >> > IW 0 [2018-08-08T13:00:18.322Z; main]: rollback > >> > IW 0 [2018-08-08T13:00:18.322Z; main]: all running merges have aborted > >> > IW 0 [2018-08-08T13:00:18.323Z; main]: rollback: done finish merges > >> > DW 0 [2018-08-08T13:00:18.323Z; main]: abort > >> > DW 0 [2018-08-08T13:00:18.323Z; main]: done abort success=true > >> > IW 0 [2018-08-08T13:00:18.323Z; main]: rollback: > >> infos=_bhg(7.1.0):C108396 > >> > IFD 0 [2018-08-08T13:00:18.323Z; main]: now checkpoint > >> > "_bhg(7.1.0):C108396" [1 segments ; isCommit = false] > >> > IFD 0 [2018-08-08T13:00:18.323Z; main]: 0 msec to checkpoint > >> > > >> > 2018-08-07 16:38 GMT+02:00 Erick Erickson <erickerick...@gmail.com>: > >> > > >> >> Bjarke: > >> >> > >> >> One thing, what version of Solr are you moving _from_ and _to_? > >> >> Solr/Lucene only guarantee one major backward revision so you can > copy > >> >> an index created with Solr 6 to another Solr 6 or Solr 7, but you > >> >> couldn't copy an index created with Solr 5 to Solr 7... > >> >> > >> >> Also note that shard splitting is a very expensive operation, so be > >> >> patient.... > >> >> > >> >> Best, > >> >> Erick > >> >> > >> >> On Tue, Aug 7, 2018 at 6:17 AM, Rahul Singh > >> >> <rahul.xavier.si...@gmail.com> wrote: > >> >> > Bjarke, > >> >> > > >> >> > I am imagining that at some point you may need to shard that data > if > >> it > >> >> grows. Or do you imagine this data to remain stagnant? > >> >> > > >> >> > Generally you want to add solrcloud to do two things : 1. Increase > >> >> availability with replicas 2. Increase available data via shards 3. > >> >> Increase fault tolerance with leader and replicas being spread around > >> the > >> >> cluster. > >> >> > > >> >> > You would be bypassing general High availability / distributed > >> computing > >> >> processes by trying to not reindex. > >> >> > > >> >> > Rahul > >> >> > On Aug 7, 2018, 7:06 AM -0400, Bjarke Buur Mortensen < > >> >> morten...@eluence.com>, wrote: > >> >> >> Hi List, > >> >> >> > >> >> >> is there a cookbook recipe for moving an existing solr core to a > solr > >> >> cloud > >> >> >> collection. > >> >> >> > >> >> >> We currently have a single machine with a large core (~150gb), > and we > >> >> would > >> >> >> like to move to solr cloud. > >> >> >> > >> >> >> I haven't been able to find anything that reuses an existing > index, > >> so > >> >> any > >> >> >> pointers much appreciated. > >> >> >> > >> >> >> Thanks, > >> >> >> Bjarke > >> >> > >> >
