Hello Mr. Miller and Mr. Erickson,
Found yet another inconsistency on the query server that
might be causing this issue. Today morning also I got a similar error
as shown in stacktrace below. So I tried querying for that
"d101dd3a-979a-11e1-927c-291130c98dff" which is our unique key in the
schema.
On the server having issue it returned more than 10 docs with
numFound="1051273" and on all other sane servers it returned only 1
doc with numFound="1". This is really weird, as, we copied the entire
index from a sane server onto the server having issues now just 2 days
ago. Do you have any idea why this would happen ?
[#|2012-05-07T12:58:54.055-0400|SEVERE|sun-appserver2.1.1|com.wpost.ipad.feeds.FeedController|_ThreadID=22;_ThreadName=httpSSLWorkerThread-9001-3;_RequestID=4203e3e5-c39d-4df7-a32a-600d0169c81f;|Error
searching for thumbnails for d101dd3a-979a-11e1-927c-291130c98dff
org.apache.solr.client.solrj.SolrServerException: Error executing query
at
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
at xxx.xxx.xxx.xxx.populateThumbnails(FeedController.java:1184)
at xxx.xxx.xxx.xxx..findNewsBySection(FeedController.java:509)
at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source)
..
...
..
Caused by: java.lang.RuntimeException: Invalid version (expected 2,
but 60) or the data in not in 'javabin' format
at
org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
at
org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
at
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
... 43 more
|#]
Ravi Kiran Bhaskar
Principal Software Engineer
Washington Post Digital
1150 15th Street NW, Washington, DC 20071
On Mon, May 7, 2012 at 9:36 AM, Mark Miller <markrmil...@gmail.com> wrote:
> Normally this specific error is caused by a non success http error page and
> response is returned. The response parser tries to parse HTML as javabin.
>
> Sent from my iPhone
>
> On May 7, 2012, at 7:37 AM, Erick Erickson <erickerick...@gmail.com> wrote:
>
>> Well, I'm guessing that the version of Solr (and perhaps there are
>> classpath issues in here?) are different, somehow, on the machine
>> slave that is showing the error.
>>
>> It's also possible that your config files have a different LUCENE_VERSION
>> in them, although I don't think this should really create the errors you're
>> reporting.
>>
>> The thing that leads me in this direction is your statement that things
>> are fine for a while and then go bad later. If replication happens just
>> before you get the index version error, that would point a finger at
>> something like different Solr versions.
>>
>> If there is no replication before this error, then this probably isn't
>> the problem
>> and we'll have to look elsewhere...
>>
>> But this is all guesswork, just like every bug... things are only obvious
>> after
>> you find the problem!
>>
>> Best
>> Erick
>>
>>
>> On Sun, May 6, 2012 at 11:08 AM, Ravi Solr <ravis...@gmail.com> wrote:
>>> Thank you very much for responding Mr.Erickson. You may be right on
>>> old version index, I will reindex. However we have a 2
>>> separate/disjoint master-slave setup...only one query node/slave has
>>> this issue. if it was really incompatible indexes why isnt the other
>>> query server also throwing errors? that's what is throwing my
>>> debugging thought process off.
>>>
>>> Thanks
>>>
>>> Ravi Kiran Bhaskar
>>> Principal Software Engineer
>>> Washington Post Digital
>>> 1150 15th Street NW, Washington, DC 20071
>>>
>>> On Sat, May 5, 2012 at 12:53 PM, Erick Erickson <erickerick...@gmail.com>
>>> wrote:
>>>> The first thing I'd check is if, in the log, there is a replication
>>>> happening
>>>> immediately prior to the error. I confess I'm not entirely up on the
>>>> version thing, but is it possible you're replicating an index that
>>>> is built with some other version of Solr?
>>>>
>>>> That would at least explain your statement that it runs OK, but then
>>>> fails sometime later.
>>>>
>>>> Best
>>>> Erick
>>>>
>>>> On Fri, May 4, 2012 at 1:50 PM, Ravi Solr <ravis...@gmail.com> wrote:
>>>>> Hello,
>>>>> We Recently we migrated our SOLR 3.6 server OS from Solaris
>>>>> to CentOS and from then on we started seeing "Invalid version
>>>>> (expected 2, but 60)" errors on one of the query servers (oddly one
>>>>> other query server seems fine). If we restart the server having issue
>>>>> everything will be alright, but the next day in the morning again we
>>>>> get the same exception. I made sure that all the client applications
>>>>> are using SOLR 3.6 version.
>>>>>
>>>>> The Glassfish on which all the applications and SOLR are deployed use
>>>>> Java 1.6.0_29. The only difference I could see
>>>>>
>>>>> 1. The process indexing to the server having issues is using java1.6.0_31
>>>>> 2. The process indexing to the server that DOES NOT have issues is
>>>>> using java1.6.0_29
>>>>>
>>>>> Could the Java minor version being greater than the SOLR instance be
>>>>> the cause of this issue ???
>>>>>
>>>>> Can anybody please help me debug this a bit more ? what else can I
>>>>> look at to understand the underlying problem. The stack trace is given
>>>>> below
>>>>>
>>>>>
>>>>> [#|2012-05-04T09:58:43.985-0400|SEVERE|sun-appserver2.1.1|xxx.xxxx.xxxx.xxxx|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-7;_RequestID=a19f92cc-2a8c-47e8-b159-a20330f14af5;
>>>>> org.apache.solr.client.solrj.SolrServerException: Error executing query
>>>>> at
>>>>> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
>>>>> at
>>>>> org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
>>>>> at
>>>>> com.wpost.ipad.feeds.FeedController.findLinksetNewsBySection(FeedController.java:743)
>>>>> at
>>>>> com.wpost.ipad.feeds.FeedController.findNewsBySection(FeedController.java:347)
>>>>> at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at
>>>>> org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175)
>>>>> at
>>>>> org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421)
>>>>> at
>>>>> org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409)
>>>>> at
>>>>> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774)
>>>>> at
>>>>> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
>>>>> at
>>>>> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
>>>>> at
>>>>> org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
>>>>> at
>>>>> org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
>>>>> at
>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315)
>>>>> at
>>>>> org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
>>>>> at
>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
>>>>> at
>>>>> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>>> at
>>>>> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>>> at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
>>>>> at
>>>>> com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
>>>>> at
>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
>>>>> at
>>>>> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>>> at
>>>>> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>>> at
>>>>> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
>>>>> at
>>>>> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
>>>>> at
>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
>>>>> at
>>>>> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>>> at
>>>>> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>>> at
>>>>> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
>>>>> at
>>>>> org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
>>>>> at
>>>>> org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
>>>>> at
>>>>> com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:393)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
>>>>> at
>>>>> com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
>>>>> Caused by: java.lang.RuntimeException: Invalid version (expected 2,
>>>>> but 60) or the data in not in 'javabin' format
>>>>> at
>>>>> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
>>>>> at
>>>>> org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
>>>>> at
>>>>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333)
>>>>> at
>>>>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>>>>> at
>>>>> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
>>>>> ... 43 more
>>>>>
>>>>>
>>>>> Thanking you in anticipation,
>>>>>
>>>>> Ravi Kiran Bhaskar
>>>>> Principal Software Engineer
>>>>> Washington Post Digital
>>>>> 1150 15th Street NW, Washington, DC 20071
