I clean the entire index and re-indexed it with SOLRJ 3.6. Still I get the same error every single day. How can I see if the container returned partial/nonconforming response since it may be hidden by solrj ?
Thanks Ravi Kiran Bhaskar On Mon, May 7, 2012 at 2:16 PM, Ravi Solr <ravis...@gmail.com> wrote: > Hello Mr. Miller and Mr. Erickson, > Found yet another inconsistency on the query server that > might be causing this issue. Today morning also I got a similar error > as shown in stacktrace below. So I tried querying for that > "d101dd3a-979a-11e1-927c-291130c98dff" which is our unique key in the > schema. > > On the server having issue it returned more than 10 docs with > numFound="1051273" and on all other sane servers it returned only 1 > doc with numFound="1". This is really weird, as, we copied the entire > index from a sane server onto the server having issues now just 2 days > ago. Do you have any idea why this would happen ? > > [#|2012-05-07T12:58:54.055-0400|SEVERE|sun-appserver2.1.1|com.wpost.ipad.feeds.FeedController|_ThreadID=22;_ThreadName=httpSSLWorkerThread-9001-3;_RequestID=4203e3e5-c39d-4df7-a32a-600d0169c81f;|Error > searching for thumbnails for d101dd3a-979a-11e1-927c-291130c98dff > org.apache.solr.client.solrj.SolrServerException: Error executing query > at > org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) > at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311) > at xxx.xxx.xxx.xxx.populateThumbnails(FeedController.java:1184) > at xxx.xxx.xxx.xxx..findNewsBySection(FeedController.java:509) > at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source) > .. > ... > .. > Caused by: java.lang.RuntimeException: Invalid version (expected 2, > but 60) or the data in not in 'javabin' format > at > org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99) > at > org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41) > at > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333) > at > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) > at > org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89) > ... 43 more > |#] > > Ravi Kiran Bhaskar > Principal Software Engineer > Washington Post Digital > 1150 15th Street NW, Washington, DC 20071 > > On Mon, May 7, 2012 at 9:36 AM, Mark Miller <markrmil...@gmail.com> wrote: >> Normally this specific error is caused by a non success http error page and >> response is returned. The response parser tries to parse HTML as javabin. >> >> Sent from my iPhone >> >> On May 7, 2012, at 7:37 AM, Erick Erickson <erickerick...@gmail.com> wrote: >> >>> Well, I'm guessing that the version of Solr (and perhaps there are >>> classpath issues in here?) are different, somehow, on the machine >>> slave that is showing the error. >>> >>> It's also possible that your config files have a different LUCENE_VERSION >>> in them, although I don't think this should really create the errors you're >>> reporting. >>> >>> The thing that leads me in this direction is your statement that things >>> are fine for a while and then go bad later. If replication happens just >>> before you get the index version error, that would point a finger at >>> something like different Solr versions. >>> >>> If there is no replication before this error, then this probably isn't >>> the problem >>> and we'll have to look elsewhere... >>> >>> But this is all guesswork, just like every bug... things are only obvious >>> after >>> you find the problem! >>> >>> Best >>> Erick >>> >>> >>> On Sun, May 6, 2012 at 11:08 AM, Ravi Solr <ravis...@gmail.com> wrote: >>>> Thank you very much for responding Mr.Erickson. You may be right on >>>> old version index, I will reindex. However we have a 2 >>>> separate/disjoint master-slave setup...only one query node/slave has >>>> this issue. if it was really incompatible indexes why isnt the other >>>> query server also throwing errors? that's what is throwing my >>>> debugging thought process off. >>>> >>>> Thanks >>>> >>>> Ravi Kiran Bhaskar >>>> Principal Software Engineer >>>> Washington Post Digital >>>> 1150 15th Street NW, Washington, DC 20071 >>>> >>>> On Sat, May 5, 2012 at 12:53 PM, Erick Erickson <erickerick...@gmail.com> >>>> wrote: >>>>> The first thing I'd check is if, in the log, there is a replication >>>>> happening >>>>> immediately prior to the error. I confess I'm not entirely up on the >>>>> version thing, but is it possible you're replicating an index that >>>>> is built with some other version of Solr? >>>>> >>>>> That would at least explain your statement that it runs OK, but then >>>>> fails sometime later. >>>>> >>>>> Best >>>>> Erick >>>>> >>>>> On Fri, May 4, 2012 at 1:50 PM, Ravi Solr <ravis...@gmail.com> wrote: >>>>>> Hello, >>>>>> We Recently we migrated our SOLR 3.6 server OS from Solaris >>>>>> to CentOS and from then on we started seeing "Invalid version >>>>>> (expected 2, but 60)" errors on one of the query servers (oddly one >>>>>> other query server seems fine). If we restart the server having issue >>>>>> everything will be alright, but the next day in the morning again we >>>>>> get the same exception. I made sure that all the client applications >>>>>> are using SOLR 3.6 version. >>>>>> >>>>>> The Glassfish on which all the applications and SOLR are deployed use >>>>>> Java 1.6.0_29. The only difference I could see >>>>>> >>>>>> 1. The process indexing to the server having issues is using java1.6.0_31 >>>>>> 2. The process indexing to the server that DOES NOT have issues is >>>>>> using java1.6.0_29 >>>>>> >>>>>> Could the Java minor version being greater than the SOLR instance be >>>>>> the cause of this issue ??? >>>>>> >>>>>> Can anybody please help me debug this a bit more ? what else can I >>>>>> look at to understand the underlying problem. The stack trace is given >>>>>> below >>>>>> >>>>>> >>>>>> [#|2012-05-04T09:58:43.985-0400|SEVERE|sun-appserver2.1.1|xxx.xxxx.xxxx.xxxx|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-7;_RequestID=a19f92cc-2a8c-47e8-b159-a20330f14af5; >>>>>> org.apache.solr.client.solrj.SolrServerException: Error executing query >>>>>> at >>>>>> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95) >>>>>> at >>>>>> org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311) >>>>>> at >>>>>> com.wpost.ipad.feeds.FeedController.findLinksetNewsBySection(FeedController.java:743) >>>>>> at >>>>>> com.wpost.ipad.feeds.FeedController.findNewsBySection(FeedController.java:347) >>>>>> at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source) >>>>>> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>> at >>>>>> org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175) >>>>>> at >>>>>> org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421) >>>>>> at >>>>>> org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409) >>>>>> at >>>>>> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774) >>>>>> at >>>>>> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) >>>>>> at >>>>>> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644) >>>>>> at >>>>>> org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549) >>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) >>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) >>>>>> at >>>>>> org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427) >>>>>> at >>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315) >>>>>> at >>>>>> org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287) >>>>>> at >>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218) >>>>>> at >>>>>> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) >>>>>> at >>>>>> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) >>>>>> at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) >>>>>> at >>>>>> com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98) >>>>>> at >>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222) >>>>>> at >>>>>> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) >>>>>> at >>>>>> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) >>>>>> at >>>>>> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587) >>>>>> at >>>>>> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093) >>>>>> at >>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166) >>>>>> at >>>>>> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) >>>>>> at >>>>>> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) >>>>>> at >>>>>> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587) >>>>>> at >>>>>> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093) >>>>>> at >>>>>> org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291) >>>>>> at >>>>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670) >>>>>> at >>>>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601) >>>>>> at >>>>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875) >>>>>> at >>>>>> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365) >>>>>> at >>>>>> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285) >>>>>> at >>>>>> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221) >>>>>> at >>>>>> com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:393) >>>>>> at >>>>>> com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269) >>>>>> at >>>>>> com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111) >>>>>> Caused by: java.lang.RuntimeException: Invalid version (expected 2, >>>>>> but 60) or the data in not in 'javabin' format >>>>>> at >>>>>> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99) >>>>>> at >>>>>> org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41) >>>>>> at >>>>>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333) >>>>>> at >>>>>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) >>>>>> at >>>>>> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89) >>>>>> ... 43 more >>>>>> >>>>>> >>>>>> Thanking you in anticipation, >>>>>> >>>>>> Ravi Kiran Bhaskar >>>>>> Principal Software Engineer >>>>>> Washington Post Digital >>>>>> 1150 15th Street NW, Washington, DC 20071
