Hi Erick, @ichattopadhyaya beat me to it already yesterday. So we are good -cheers Vijay
On Wed, Jan 28, 2015 at 1:30 PM, Erick Erickson <erickerick...@gmail.com> wrote: > Vijay: > > Thanks for reporting this back! Could I ask you to post a new patch with > your correction? Please use the same patch name > (SOLR-5850.patch), and include a note about what you found (I've already > added a comment). > > Thanks! > Erick > > On Wed, Jan 28, 2015 at 9:18 AM, Vijay Sekhri <sekhrivi...@gmail.com> > wrote: > > > Hi Shawn, > > Thank you so much for the assistance. Building is not a problem . Back in > > the days I have worked with linking, compiling and building C , C++ > > software . Java is a piece of cake. > > We have built the new war from the source version 4.10.3 and our > > preliminary tests have shown that our issue (replicas in recovery on high > > load)* is resolved *. We will continue to do more testing and confirm . > > Please note that the *patch is BUGGY*. > > > > It removed the break statement within while loop because of which, > whenever > > we send a list of docs it would hang (API CloudSolrServer.add) , but it > > would work if send one doc at a time. > > > > It took a while to figure out why that is happening. Once we put the > break > > statement back it worked like a charm. > > Furthermore the patch has > > > > > solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java > > which should be > > > > > solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.java > > > > Finally checking if(!offer) is sufficient than using if(offer == false) > > Last but not the least having a configurable queue size and timeouts > > (managed via solrconfig) would be quite helpful > > Thank you once again for your help. > > > > Vijay > > > > On Tue, Jan 27, 2015 at 6:20 PM, Shawn Heisey <apa...@elyograg.org> > wrote: > > > > > On 1/27/2015 2:52 PM, Vijay Sekhri wrote: > > > > Hi Shawn, > > > > Here is some update. We found the main issue > > > > We have configured our cluster to run under jetty and when we tried > > full > > > > indexing, we did not see the original Invalid Chunk error. However > the > > > > replicas still went into recovery > > > > All this time we been trying to look into replicas logs to diagnose > the > > > > issue. The problem seem to be at the leader side. When we looked into > > > > leader logs, we found the following on all the leaders > > > > > > > > 3439873 [qtp1314570047-92] WARN > > > > org.apache.solr.update.processor.DistributedUpdateProcessor – Error > > > > sending update > > > > *java.lang.IllegalStateException: Queue full* > > > > > > <snip> > > > > > > > There is a similar bug reported around this > > > > https://issues.apache.org/jira/browse/SOLR-5850 > > > > > > > > and it seem to be in OPEN status. Is there a way we can configure the > > > queue > > > > size and increase it ? or is there a version of solr that has this > > issue > > > > resolved already? > > > > Can you suggest where we go from here to resolve this ? We can > repatch > > > the > > > > war file if that is what you would recommend . > > > > In the end our initial speculation about solr unable to handle so > many > > > > update is correct. We do not see this issue when the update load is > > less. > > > > > > Are you in a position where you can try the patch attached to > > > SOLR-5850? You would need to get the source code for the version > you're > > > on (or perhaps a newer 4.x version), patch it, and build Solr yourself. > > > If you have no experience building java packages from source, this > might > > > prove to be difficult. > > > > > > Thanks, > > > Shawn > > > > > > > > > > > > -- > > ********************************************* > > Vijay Sekhri > > ********************************************* > > > -- ********************************************* Vijay Sekhri *********************************************
