OK, thanks. As long as it's my dev box, reindexing is fine. I just hope that my assumption holds, that our prod solr is 7x segments only.
Thanks again, Bjarke 2018-08-08 20:03 GMT+02:00 Erick Erickson <erickerick...@gmail.com>: > Bjarke: > > Using SPLITSHARD on an index with 6x segments just seems to not work, > even outside the standalone-> cloud issue. I'll raise a JIRA. > Meanwhile I think you'll have to re-index I'm afraid. > > Thanks for raising the issue. > > Erick > > On Wed, Aug 8, 2018 at 6:34 AM, Bjarke Buur Mortensen > <morten...@eluence.com> wrote: > > Erick, > > > > thanks, that is of course something I left out of the original question. > > Our Solr is 7.1, so that should not present a problem (crossing fingers). > > > > However, on my dev box I'm trying out the steps, and here I have some > > segments created with version 6 of Solr. > > > > After having copied data from my non-cloud solr into my > > single-shard-single-replica collection and verified that Solr Cloud works > > with this collection, I then submit the splitshard command > > > > http://172.17.0.4:8984 > > /solr/admin/collections?action=SPLITSHARD&collection= > procurement&shard=shard1 > > > > However, this gives me the error: > > org.apache.solr.client.solrj.impl.HttpSolrClient$ > RemoteSolrException:Error > > from server at http://172.17.0.4:8984/solr: > > java.lang.IllegalArgumentException: Cannot merge a segment that has been > > created with major version 6 into this index which has been created by > > major version 7"} > > > > I have tried running both optimize and IndexUpgrader on the index before > > shard splitting, but the same error still occurs. > > > > Any ideas as to why this happens? > > > > Below is an output from running IndexUpgrader, which I cannot decipher. > > It both states that "All segments upgraded to version 7.1.0" and ''all > > running merges have aborted" ¯\_(ツ)_/¯ > > > > Thanks a lot, > > Bjarke > > > > > > ====================== > > java -cp > > /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/lucene- > backward-codecs-7.1.0.jar:/opt/solr/server/solr-webapp/ > webapp/WEB-INF/lib/lucene-core-7.1.0.jar > > org.apache.lucene.index.IndexUpgrader -delete-prior-commits -verbose > > /var/solr/cloud/procurement_shard1_replica_n1/data/index > > IFD 0 [2018-08-08T13:00:18.244Z; main]: init: current segments file is > > "segments_4vs"; > > deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPoli > cy@721e0f4f > > IFD 0 [2018-08-08T13:00:18.266Z; main]: init: load commit "segments_4vs" > > IFD 0 [2018-08-08T13:00:18.270Z; main]: now checkpoint > > "_bhg(7.1.0):C108396" [1 segments ; isCommit = false] > > IFD 0 [2018-08-08T13:00:18.270Z; main]: 0 msec to checkpoint > > IW 0 [2018-08-08T13:00:18.270Z; main]: init: create=false > > IW 0 [2018-08-08T13:00:18.273Z; main]: > > dir=MMapDirectory@/var/solr/cloud/procurement_shard1_ > replica_n1/data/index > > lockFactory=org.apache.lucene.store.NativeFSLockFactory@6debcae2 > > index=_bhg(7.1.0):C108396 > > version=7.1.0 > > analyzer=null > > ramBufferSizeMB=16.0 > > maxBufferedDocs=-1 > > mergedSegmentWarmer=null > > delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy > > commit=null > > openMode=CREATE_OR_APPEND > > similarity=org.apache.lucene.search.similarities.BM25Similarity > > mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=-1, > > maxMergeCount=-1, ioThrottle=true > > codec=Lucene70 > > infoStream=org.apache.lucene.util.PrintStreamInfoStream > > mergePolicy=UpgradeIndexMergePolicy([TieredMergePolicy: > maxMergeAtOnce=10, > > maxMergeAtOnceExplicit=30, maxMergedSegmentMB=5120.0, floorSegmentMB=2.0, > > forceMergeDeletesPctAllowed=10.0, segmentsPerTier=10.0, > > maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1) > > indexerThreadPool=org.apache.lucene.index.DocumentsWriterPerThreadPool@ > 5ba23b66 > > readerPooling=true > > perThreadHardLimitMB=1945 > > useCompoundFile=true > > commitOnClose=true > > indexSort=null > > writer=org.apache.lucene.index.IndexWriter@2ff4f00f > > > > IW 0 [2018-08-08T13:00:18.273Z; main]: MMapDirectory.UNMAP_SUPPORTED= > true > > IndexUpgrader 0 [2018-08-08T13:00:18.274Z; main]: Upgrading all pre-7.1.0 > > segments of index directory > > 'MMapDirectory@/var/solr/cloud/procurement_shard1_replica_n1/data/index > > lockFactory=org.apache.lucene.store.NativeFSLockFactory@6debcae2' to > > version 7.1.0... > > IW 0 [2018-08-08T13:00:18.274Z; main]: forceMerge: index now > > _bhg(7.1.0):C108396 > > IW 0 [2018-08-08T13:00:18.274Z; main]: now flush at forceMerge > > IW 0 [2018-08-08T13:00:18.274Z; main]: start flush: > applyAllDeletes=true > > IW 0 [2018-08-08T13:00:18.274Z; main]: index before flush > > _bhg(7.1.0):C108396 > > DW 0 [2018-08-08T13:00:18.274Z; main]: startFullFlush > > DW 0 [2018-08-08T13:00:18.275Z; main]: main finishFullFlush success=true > > IW 0 [2018-08-08T13:00:18.275Z; main]: now apply all deletes for all > > segments buffered updates bytesUsed=0 reader pool bytesUsed=0 > > BD 0 [2018-08-08T13:00:18.275Z; main]: waitApply: no deletes to apply > > UPGMP 0 [2018-08-08T13:00:18.276Z; main]: findForcedMerges: > > segmentsToUpgrade={} > > MS 0 [2018-08-08T13:00:18.282Z; main]: initDynamicDefaults spins=true > > maxThreadCount=1 maxMergeCount=6 > > MS 0 [2018-08-08T13:00:18.282Z; main]: now merge > > MS 0 [2018-08-08T13:00:18.282Z; main]: index: _bhg(7.1.0):C108396 > > MS 0 [2018-08-08T13:00:18.282Z; main]: no more merges pending; now > return > > IndexUpgrader 0 [2018-08-08T13:00:18.282Z; main]: All segments upgraded > to > > version 7.1.0 > > IndexUpgrader 0 [2018-08-08T13:00:18.283Z; main]: Enforcing commit to > > rewrite all index metadata... > > IW 0 [2018-08-08T13:00:18.283Z; main]: commit: start > > IW 0 [2018-08-08T13:00:18.283Z; main]: commit: enter lock > > IW 0 [2018-08-08T13:00:18.283Z; main]: commit: now prepare > > IW 0 [2018-08-08T13:00:18.283Z; main]: prepareCommit: flush > > IW 0 [2018-08-08T13:00:18.283Z; main]: index before flush > > _bhg(7.1.0):C108396 > > DW 0 [2018-08-08T13:00:18.283Z; main]: startFullFlush > > IW 0 [2018-08-08T13:00:18.283Z; main]: now apply all deletes for all > > segments buffered updates bytesUsed=0 reader pool bytesUsed=0 > > BD 0 [2018-08-08T13:00:18.283Z; main]: waitApply: no deletes to apply > > DW 0 [2018-08-08T13:00:18.284Z; main]: main finishFullFlush success=true > > IW 0 [2018-08-08T13:00:18.284Z; main]: startCommit(): start > > IW 0 [2018-08-08T13:00:18.284Z; main]: startCommit > > index=_bhg(7.1.0):C108396 changeCount=2 > > IW 0 [2018-08-08T13:00:18.293Z; main]: startCommit: wrote pending > segments > > file "pending_segments_4vt" > > IW 0 [2018-08-08T13:00:18.295Z; main]: done all syncs: > > [_bhg_Lucene50_0.tip, _bhg.fdx, _bhg.fnm, _bhg.nvm, _bhg.fdt, _bhg.si, > > _bhg_Lucene50_0.pos, _bhg.nvd, _bhg_Lucene50_0.doc, _bhg_Lucene50_0.tim] > > IW 0 [2018-08-08T13:00:18.295Z; main]: commit: pendingCommit != null > > IW 0 [2018-08-08T13:00:18.298Z; main]: commit: done writing segments file > > "segments_4vt" > > IFD 0 [2018-08-08T13:00:18.298Z; main]: now checkpoint > > "_bhg(7.1.0):C108396" [1 segments ; isCommit = true] > > IFD 0 [2018-08-08T13:00:18.298Z; main]: deleteCommits: now decRef commit > > "segments_4vs" > > IFD 0 [2018-08-08T13:00:18.298Z; main]: delete [segments_4vs] > > IFD 0 [2018-08-08T13:00:18.299Z; main]: 0 msec to checkpoint > > IW 0 [2018-08-08T13:00:18.319Z; main]: commit: took 16.0 msec > > IW 0 [2018-08-08T13:00:18.319Z; main]: commit: done > > IndexUpgrader 0 [2018-08-08T13:00:18.319Z; main]: Committed upgraded > > metadata to index. > > IW 0 [2018-08-08T13:00:18.319Z; main]: now flush at close > > IW 0 [2018-08-08T13:00:18.319Z; main]: start flush: > applyAllDeletes=true > > IW 0 [2018-08-08T13:00:18.319Z; main]: index before flush > > _bhg(7.1.0):C108396 > > DW 0 [2018-08-08T13:00:18.319Z; main]: startFullFlush > > DW 0 [2018-08-08T13:00:18.320Z; main]: main finishFullFlush success=true > > IW 0 [2018-08-08T13:00:18.320Z; main]: now apply all deletes for all > > segments buffered updates bytesUsed=0 reader pool bytesUsed=0 > > BD 0 [2018-08-08T13:00:18.320Z; main]: waitApply: no deletes to apply > > MS 0 [2018-08-08T13:00:18.320Z; main]: updateMergeThreads ioThrottle=true > > targetMBPerSec=10240.0 MB/sec > > MS 0 [2018-08-08T13:00:18.320Z; main]: now merge > > MS 0 [2018-08-08T13:00:18.321Z; main]: index: _bhg(7.1.0):C108396 > > MS 0 [2018-08-08T13:00:18.321Z; main]: no more merges pending; now > return > > IW 0 [2018-08-08T13:00:18.321Z; main]: waitForMerges > > IW 0 [2018-08-08T13:00:18.321Z; main]: waitForMerges done > > IW 0 [2018-08-08T13:00:18.321Z; main]: commit: start > > IW 0 [2018-08-08T13:00:18.321Z; main]: commit: enter lock > > IW 0 [2018-08-08T13:00:18.321Z; main]: commit: now prepare > > IW 0 [2018-08-08T13:00:18.321Z; main]: prepareCommit: flush > > IW 0 [2018-08-08T13:00:18.321Z; main]: index before flush > > _bhg(7.1.0):C108396 > > DW 0 [2018-08-08T13:00:18.321Z; main]: startFullFlush > > IW 0 [2018-08-08T13:00:18.321Z; main]: now apply all deletes for all > > segments buffered updates bytesUsed=0 reader pool bytesUsed=0 > > BD 0 [2018-08-08T13:00:18.322Z; main]: waitApply: no deletes to apply > > DW 0 [2018-08-08T13:00:18.322Z; main]: main finishFullFlush success=true > > IW 0 [2018-08-08T13:00:18.322Z; main]: startCommit(): start > > IW 0 [2018-08-08T13:00:18.322Z; main]: skip startCommit(): no changes > > pending > > IW 0 [2018-08-08T13:00:18.322Z; main]: commit: pendingCommit == null; > skip > > IW 0 [2018-08-08T13:00:18.322Z; main]: commit: took 0.9 msec > > IW 0 [2018-08-08T13:00:18.322Z; main]: commit: done > > IW 0 [2018-08-08T13:00:18.322Z; main]: rollback > > IW 0 [2018-08-08T13:00:18.322Z; main]: all running merges have aborted > > IW 0 [2018-08-08T13:00:18.323Z; main]: rollback: done finish merges > > DW 0 [2018-08-08T13:00:18.323Z; main]: abort > > DW 0 [2018-08-08T13:00:18.323Z; main]: done abort success=true > > IW 0 [2018-08-08T13:00:18.323Z; main]: rollback: > infos=_bhg(7.1.0):C108396 > > IFD 0 [2018-08-08T13:00:18.323Z; main]: now checkpoint > > "_bhg(7.1.0):C108396" [1 segments ; isCommit = false] > > IFD 0 [2018-08-08T13:00:18.323Z; main]: 0 msec to checkpoint > > > > 2018-08-07 16:38 GMT+02:00 Erick Erickson <erickerick...@gmail.com>: > > > >> Bjarke: > >> > >> One thing, what version of Solr are you moving _from_ and _to_? > >> Solr/Lucene only guarantee one major backward revision so you can copy > >> an index created with Solr 6 to another Solr 6 or Solr 7, but you > >> couldn't copy an index created with Solr 5 to Solr 7... > >> > >> Also note that shard splitting is a very expensive operation, so be > >> patient.... > >> > >> Best, > >> Erick > >> > >> On Tue, Aug 7, 2018 at 6:17 AM, Rahul Singh > >> <rahul.xavier.si...@gmail.com> wrote: > >> > Bjarke, > >> > > >> > I am imagining that at some point you may need to shard that data if > it > >> grows. Or do you imagine this data to remain stagnant? > >> > > >> > Generally you want to add solrcloud to do two things : 1. Increase > >> availability with replicas 2. Increase available data via shards 3. > >> Increase fault tolerance with leader and replicas being spread around > the > >> cluster. > >> > > >> > You would be bypassing general High availability / distributed > computing > >> processes by trying to not reindex. > >> > > >> > Rahul > >> > On Aug 7, 2018, 7:06 AM -0400, Bjarke Buur Mortensen < > >> morten...@eluence.com>, wrote: > >> >> Hi List, > >> >> > >> >> is there a cookbook recipe for moving an existing solr core to a solr > >> cloud > >> >> collection. > >> >> > >> >> We currently have a single machine with a large core (~150gb), and we > >> would > >> >> like to move to solr cloud. > >> >> > >> >> I haven't been able to find anything that reuses an existing index, > so > >> any > >> >> pointers much appreciated. > >> >> > >> >> Thanks, > >> >> Bjarke > >> >
