Oops, I somehow forgot to mention that. The errors I'm seeing are with the release version of Solr 4.4.0. I mentioned 4.1.0 as that's what we currently have in prod, and we want to upgrade to 4.4.0 so we can do shard splitting. Towards that end, I'm testing shard splitting in 4.4.0 and seeing these errors.
-Greg On Sun, Aug 11, 2013 at 7:51 AM, Erick Erickson <erickerick...@gmail.com>wrote: > The very first thing I'd do is go to Solr 4.4. There have been > a lot of improvements in this code in the intervening 3 > versions. > > If the problem still occurs in 4.4, it'll get a lot more attention > than 4.1.... > > FWIW, > Erick > > > On Fri, Aug 9, 2013 at 7:32 PM, Greg Preston <gpres...@marinsoftware.com > >wrote: > > > Howdy, > > > > I'm trying to test shard splitting, and it's not working for me. I've > got > > a 4 node cloud with a single collection and 2 shards. > > > > I've indexed 170k small documents, and I'm using the compositeId router, > > with an internal "client id" as the shard key, with 4 distinct values > > across the data set. For my testing, the values of the shard keys are 1 > > through 4. Before splitting, shard1 contains 100k docs (all of the docs > > for shard keys 1 and 4) and shard2 contains 70k docs (all of the docs for > > shard keys 2 and 3). > > > > In prod, we're going to have thousands of unique shard keys, but for now, > > I'm testing at a smaller scale. I attempt to split shard2 with > > > > > http://host0:8983/solr/admin/collections?action=SPLITSHARD&collection=coll&shard=shard2 > > > > I understand the shard splitting is on hash range, not document count, > and > > it shouldn't split up documents within a single shard key, so I'm ok with > > it if both shard keys end up in the same sub-shard. > > > > I see the following in the logs: > > > > 689524 [qtp259549756-119] ERROR > org.apache.solr.servlet.SolrDispatchFilter > > – null:java.lang.RuntimeException: java.lang.IllegalArgumentException: > > maxValue must be non-negative (got: -1) > > at > > > > > org.apache.solr.handler.admin.CoreAdminHandler.handleSplitAction(CoreAdminHandler.java:290) > > at > > > > > org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:186) > > at > > > > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) > > at > > > > > org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:611) > > at > > > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:209) > > at > > > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158) > > at > > > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) > > at > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) > > at > > > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > > at > > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) > > at > > > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) > > at > > > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) > > at > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) > > at > > > > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > > at > > > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) > > at > > > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > > at > > > > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) > > at > > > > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) > > at > > > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > > at org.eclipse.jetty.server.Server.handle(Server.java:368) > > at > > > > > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) > > at > > > > > org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) > > at > > > > > org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953) > > at > > > > > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014) > > at > org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) > > at > > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) > > at > > > > > org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) > > at > > > > > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) > > at > > > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > > at > > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > > at java.lang.Thread.run(Thread.java:662) > > Caused by: java.lang.IllegalArgumentException: maxValue must be > > non-negative (got: -1) > > at > > > org.apache.lucene.util.packed.PackedInts.bitsRequired(PackedInts.java:1184) > > at > > > > > org.apache.lucene.codecs.lucene42.Lucene42DocValuesConsumer.addNumericField(Lucene42DocValuesConsumer.java:140) > > at > > > > > org.apache.lucene.codecs.lucene42.Lucene42DocValuesConsumer.addNumericField(Lucene42DocValuesConsumer.java:92) > > at > > > > > org.apache.lucene.codecs.DocValuesConsumer.mergeNumericField(DocValuesConsumer.java:112) > > at > > org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:221) > > at > > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:119) > > at > > org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:2488) > > at > > > org.apache.solr.update.SolrIndexSplitter.split(SolrIndexSplitter.java:125) > > at > > > > > org.apache.solr.update.DirectUpdateHandler2.split(DirectUpdateHandler2.java:766) > > at > > > > > org.apache.solr.handler.admin.CoreAdminHandler.handleSplitAction(CoreAdminHandler.java:284) > > ... 30 more > > > > > > > > I thought maybe there was an issue with performing a split where all docs > > go to one sub-shard, so I wiped everything clean and re-indexed the same > > data using default routing (no shard key). Now when I perform a split, I > > get a replication error on the second replica of both sub-shards: > > > > 453616 [RecoveryThread] WARN org.apache.solr.update.PeerSync – > PeerSync: > > core=coll_shard2_0_replica2 url=http://host2:8983/solr exception > talking > > to http://host1:8983/solr/coll_shard2_0_replica1/, failed > > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: > > Server at http://host1:8983/solr/coll_shard2_0_replica1 returned non ok > > status:404, message:Not Found > > at > > > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385) > > at > > > > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) > > at > > > > > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156) > > at > > > > > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119) > > at > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > > at > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > > > > if I browse to http://host1:8983/solr/coll_shard2_0_replica1< > > http://solrcloud1:8983/solr/#/marin_shard2_0_replica1> > > I > > do get a 404, but > > http://host1:8983/solr/#/coll_shard2_0_replica1< > > http://solrcloud1:8983/solr/#/marin_shard2_0_replica1> > > does > > bring up the admin UI for that core. > > > > I've been running solr cloud 4.1.0 for a while now, and our prod cloud > has > > ~1.8 billion docs indexed into 3 shards. I'd really like to get > splitting > > working so we can add more hardware. > > > > Any ideas? > > > > -Greg > > >
