Hello,
Unfortunately it seems like I spoke too early. Today morning I
received the same error again even after disabling the iptables. The
weird thing is only one out of 6 or 7 queries fails as evidenced in
the stack traces below. The query below the stack trace gave a
'status=500' subsequent queries look fine
[#|2012-05-15T08:12:38.703-0400|SEVERE|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-8;_RequestID=9f54ea89-357a-4c1b-87a1-fbaacc9fd0ee;|org.apache.solr.common.SolrException
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:275)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:313)
at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at
com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
at
com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
at
com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
Caused by: java.lang.RuntimeException: Invalid version (expected 2,
but 60) or the data in not in 'javabin' format
at
org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
at
org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:469)
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:249)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:129)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:103)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[#|2012-05-15T08:12:38.704-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-8;|[sitesearchcore]
webapp=/solr-admin path=/select
params={q=dwts&start=0&rows=10&shards=localhost:9001/solr-admin/sitesearchcore,localhost:9001/solr-admin/deathnoticescore&tracking=sitesearch&fl=headline,contenturl,blurb,sourcenav,contenttype,byline,primarysection,smallthumburl,displaydatetime,pubdatetime&facet=true&facet.field=contenttype&facet.mincount=1&facet.method=enum&fq=displaydatetime:[2005-01-01T00:00:00Z+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+24+Hours"}displaydatetime:[NOW/DAY-1DAY+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+7+Days"}displaydatetime:[NOW/DAY-7DAYS+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+60+Days"}displaydatetime:[NOW/DAY-60DAYS+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+12+Months"}displaydatetime:[NOW/DAY-1YEAR+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"All+Since+2005"}displaydatetime:[*+TO+NOW/DAY%2B1DAY]&wt=javabin&version=2}
status=500 QTime=25 |#]
[#|2012-05-15T08:12:39.905-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=27;_ThreadName=httpSSLWorkerThread-9001-6;|[sitesearchcore]
webapp=/solr-admin path=/admin/ping params={} hits=35257 status=0
QTime=5 |#]
[#|2012-05-15T08:12:39.906-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=27;_ThreadName=httpSSLWorkerThread-9001-6;|[sitesearchcore]
webapp=/solr-admin path=/admin/ping params={} status=0 QTime=6 |#]
[#|2012-05-15T08:12:41.109-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=22;_ThreadName=httpSSLWorkerThread-9001-2;|[spellsuggestcore]
webapp=/solr-admin path=/select
params={q=north+korea&spellcheck=true&spellcheck.count=5&spellcheck.collate=true&spellcheck.maxCollations=5&spellcheck.collateExtendedResults=true&tracking=sitesearch&wt=javabin&version=2}
hits=248 status=0 QTime=2 |#]
[#|2012-05-15T08:12:43.349-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=33;_ThreadName=httpSSLWorkerThread-9001-9;|[deathnoticescore]
webapp=/solr-admin path=/select
params={mm=2<-1+5<-3+6<60%25&tie=0.01&qf=headline^50.0+webheadline^50.0+mobileheadline^50.0+subheadline^50.0+blurb^40.0+bylinetext^30.0+keywordLower^10.0+personLower^10.0+organizationLower^10.0+conceptLower^10.0+subjectLower^10.0+neighborhoodLower^10.0+addressLower^10.0+cityLower^10.0+zipcodeLower^10.0+countyLower^10.0+countryLower^10.0+countrycodeLower^10.0+body^5.0&q.alt=*:*&hl.fl=keyword&defType=edismax&f.body.hl.fragsize=0&pf=headline^55.0+webheadline^55.0+mobileheadline^55.0+subheadline^55.0+blurb^45.0+bylinetext^35.0+keywordLower^15.0+personLower^15.0+organizationLower^15.0+conceptLower^15.0+subjectLower^15.0+neighborhoodLower^15.0+addressLower^15.0+cityLower^15.0+zipcodeLower^15.0+countyLower^15.0+countryLower^15.0+countrycodeLower^15.0+body^10.0&fl=systemid,score&bf=ord(displaydatetime)^1.0&f.text.hl.fragmenter=regex&f.name.hl.alternateField=keyword&ps=5&q=annalynne&start=0&rows=10&tracking=sitesearch&facet=true&facet.field=contenttype&facet.method=enum&fq=displaydatetime:[2005-01-01T00:00:00Z+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+24+Hours"}displaydatetime:[NOW/DAY-1DAY+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+7+Days"}displaydatetime:[NOW/DAY-7DAYS+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+60+Days"}displaydatetime:[NOW/DAY-60DAYS+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+12+Months"}displaydatetime:[NOW/DAY-1YEAR+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"All+Since+2005"}displaydatetime:[*+TO+NOW/DAY%2B1DAY]&fsv=true&f.contenttype.facet.limit=160&isShard=true&NOW=1337083963338&wt=javabin&version=2}
hits=0 status=0 QTime=4 |#]
[#|2012-05-15T08:12:43.356-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=28;_ThreadName=httpSSLWorkerThread-9001-7;|[sitesearchcore]
webapp=/solr-admin path=/select
params={mm=2<-1+5<-3+6<60%25&tie=0.01&qf=headline^50.0+webheadline^50.0+mobileheadline^50.0+subheadline^50.0+blurb^40.0+bylinetext^30.0+keywordLower^10.0+personLower^10.0+organizationLower^10.0+conceptLower^10.0+subjectLower^10.0+neighborhoodLower^10.0+addressLower^10.0+cityLower^10.0+zipcodeLower^10.0+countyLower^10.0+countryLower^10.0+countrycodeLower^10.0+body^5.0&q.alt=*:*&hl.fl=keyword&defType=edismax&f.body.hl.fragsize=0&pf=headline^55.0+webheadline^55.0+mobileheadline^55.0+subheadline^55.0+blurb^45.0+bylinetext^35.0+keywordLower^15.0+personLower^15.0+organizationLower^15.0+conceptLower^15.0+subjectLower^15.0+neighborhoodLower^15.0+addressLower^15.0+cityLower^15.0+zipcodeLower^15.0+countyLower^15.0+countryLower^15.0+countrycodeLower^15.0+body^10.0&fl=systemid,score&bf=ord(displaydatetime)^1.0&f.text.hl.fragmenter=regex&f.name.hl.alternateField=keyword&ps=5&q=annalynne&start=0&rows=10&tracking=sitesearch&facet=true&facet.field=contenttype&facet.method=enum&fq=displaydatetime:[2005-01-01T00:00:00Z+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+24+Hours"}displaydatetime:[NOW/DAY-1DAY+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+7+Days"}displaydatetime:[NOW/DAY-7DAYS+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+60+Days"}displaydatetime:[NOW/DAY-60DAYS+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+12+Months"}displaydatetime:[NOW/DAY-1YEAR+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"All+Since+2005"}displaydatetime:[*+TO+NOW/DAY%2B1DAY]&fsv=true&f.contenttype.facet.limit=160&isShard=true&NOW=1337083963338&wt=javabin&version=2}
hits=5 status=0 QTime=11 |#]
Thanks,
Ravi Kiran Bhaskar
On Fri, May 11, 2012 at 11:04 PM, Mark Miller <markrmil...@gmail.com> wrote:
> Yeah, 9 times out of 10, this error is a 404 - which wouldn't be logged
> anywhere.
>
> On May 11, 2012, at 6:12 PM, Ravi Solr wrote:
>
>> Guys, just to give you an update, we think we "might" have found the
>> issue. iptables was enabled on one query server and disabled on the
>> other. The server where iptables is enabled is the one having issues,
>> we disabled the iptables today to test out the theory that the
>> iptables might be causing this issue of null/empty response. If the
>> server holds up during the weekend then we have the culprit :-)
>>
>> Thanks to all of you who helped me out. Stay tuned.
>>
>> Ravi Kiran
>>
>> On Fri, May 11, 2012 at 1:23 AM, Shawn Heisey <s...@elyograg.org> wrote:
>>> On 5/10/2012 4:17 PM, Ravi Solr wrote:
>>>>
>>>> Thanks for responding Mr. Heisey... I don't see any parsing errors in
>>>> my log but I see lot of exceptions like the one listed below....once
>>>> an exception like this happens weirdness ensues. For example - To
>>>> check sanity I queried for uniquekey:"111" from the solr admin GUI it
>>>> gave back numFound equal to all docs in that index i.e. its not
>>>> searching for that uniquekey at all, it blindly matched all docs.
>>>> However, once you restart the server the same index without any change
>>>> works perfectly returning only one doc in numFound when you search for
>>>> uniquekey:"111"...I tried everything from reindexing, copying index
>>>> from another sane server, delete entire index and reindex from scratch
>>>> etc but in vain, it works for roughly 24 hours and then starts
>>>> throwing the same error no matter what the query is.
>>>>
>>>>
>>>>
>>>> [#|2012-05-10T13:27:14.071-0400|SEVERE|sun-appserver2.1.1|xxx.xxx.xxx.xxx|_ThreadID=21;_ThreadName=httpSSLWorkerThread-9001-6;_RequestID=d44462e7-576b-4391-a499-c65da33e3293;|Error
>>>> searching data for section Local
>>>> org.apache.solr.client.solrj.SolrServerException: Error executing query
>>>> at
>>>> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
>>>> at
>>>> org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
>>>> at xxx.xxx.xxx.xxx(FeedController.java:621)
>>>> at xxx.xxx.xxx.xxx(FeedController.java:402)
>>>
>>>
>>> This is still saying solrj. Unless I am completely misunderstanding the way
>>> things work, which I will freely admit is possible, this is the client code.
>>> Do you have anything in the log files from Solr (the server)? I don't have
>>> a lot of experience with Tomcat, because I run my Solr under jetty as
>>> included in the example. It looks like the client is running under Tomcat,
>>> though I suppose you might be running Solr under a different container.
>>>
>>> Thanks,
>>> Shawn
>>>
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
