We are using Solr 3.6.2 with a single core setup on a glassfish server, every 4-5 hours the server gradually gets into a some kind of a inconsistent state and stops accepting any queries giving back cached results. Even the core reload fails giving the following. Has anybody experienced such behavior ? Can anybody help me understand why this might happen ?
http://searchserver:80/solr/admin/cores?action=RELOAD&core=core1 <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">9</int> </lst> <lst name="status"> <lst name="core1"> <str name="name">core1</str> <str name="instanceDir">/data/solr/core1-home/</str> <str name="dataDir">/data/solr/core/core1-data/</str> <date name="startTime">2013-05-01T19:16:31.32Z</date> <long name="uptime">137850</long> <lst name="index"> <int name="numDocs">21479</int> <int name="maxDoc">25170</int> <long name="version">1367184551418</long> <int name="segmentCount">4</int> <bool name="current">true</bool> <bool name="hasDeletions">true</bool> <str name="directory">org.apache.lucene.store.MMapDirectory:org.apache.lucene.store.MMapDirectory@/data/solr/core/core1-data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@71d9673</str> <date name="lastModified">2013-05-01T19:15:04Z</date> </lst> </lst> </lst> </response> During the inconsistent state any queries being issued to the server loose the query parameters. We can see the proper queries in the container's http access logs but solr somehow solr doesn't get the query params at all. Also note that "content length" on the container's access logs is always 68935, which implies its always giving the same docs irrespective of the query. If we restart the server everything is back to normal and the same queries run properly. SOLR Log -------------- [#|2013-05-01T15:20:02.031-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=20;_ThreadName=httpSSLWorkerThread-9001-1;|[core1] webapp=/solr path=/select params={} hits=21479 status=0 QTime=17 |#] [#|2013-05-01T15:20:02.034-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=24;_ThreadName=httpSSLWorkerThread-9001-4;|[core1] webapp=/solr path=/select params={} hits=21479 status=0 QTime=13 |#] [#|2013-05-01T15:20:02.055-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=23;_ThreadName=httpSSLWorkerThread-9001-3;|[core1] webapp=/solr path=/select params={} hits=21479 status=0 QTime=13 |#] [#|2013-05-01T15:20:02.081-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=25;_ThreadName=httpSSLWorkerThread-9001-5;|[core1] webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#] [#|2013-05-01T15:20:02.106-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=19;_ThreadName=httpSSLWorkerThread-9001-0;|[core1] webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#] [#|2013-05-01T15:20:02.136-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=22;_ThreadName=httpSSLWorkerThread-9001-2;|[core1] webapp=/solr path=/select params={} hits=21479 status=0 QTime=16 |#] [#|2013-05-01T15:20:02.161-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=20;_ThreadName=httpSSLWorkerThread-9001-1;|[core1] webapp=/solr path=/select params={} hits=21479 status=0 QTime=15 |#] [#|2013-05-01T15:20:02.185-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=24;_ThreadName=httpSSLWorkerThread-9001-4;|[core1] webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#] [#|2013-05-01T15:20:02.209-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=23;_ThreadName=httpSSLWorkerThread-9001-3;|[core1] webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#] [#|2013-05-01T15:20:02.241-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=25;_ThreadName=httpSSLWorkerThread-9001-5;|[core1] webapp=/solr path=/select params={} hits=21479 status=0 QTime=16 |#] [#|2013-05-01T15:20:02.266-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=19;_ThreadName=httpSSLWorkerThread-9001-0;|[core1] webapp=/solr path=/select params={} hits=21479 status=0 QTime=15 |#] [#|2013-05-01T15:20:02.288-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=22;_ThreadName=httpSSLWorkerThread-9001-2;|[core1] webapp=/solr path=/select params={} hits=21479 status=0 QTime=14 |#] [#|2013-05-01T15:20:02.291-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=20;_ThreadName=httpSSLWorkerThread-9001-1;|[core1] webapp=/solr path=/select params={} hits=21479 status=0 QTime=15 |#] Container Access Logs --------------------------------- "xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET /solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A% 22site.com%22&fq=categories%3A%28%22Music+Venues%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2 HTTP/1.1" 200 68935 "xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET /solr/core1/select?q=*%3A*&rows=5&start=0&facet=true&fq=source%3A%22site.com%22&fq=categories%3A%28%22Exhibits%22%29&fq=types%3A%28%22Painting%2FDrawing%22%29&sort=closingdate+asc%2C+openingdate+desc%2C+namestring+asc&wt=javabin&version=2&wt=javabin&version=2 HTTP/1.1" 200 68935 "xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET /solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=categories%3A%28%22Music+Events%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2 HTTP/1.1" 200 68935 "xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET /solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A% 22site.com%22&fq=categories%3A%28%22Museums%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2 HTTP/1.1" 200 68935 "xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET /solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A% 22site.com%22&fq=categories%3A%28%22Exhibits%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2 HTTP/1.1" 200 68935 "xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET /solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A% 22site.com%22&fq=categories%3A%28%22Performing+Arts%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2 HTTP/1.1" 200 68935 "xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET /solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A% 22site.com%22&fq=categories%3A%28%22Theaters%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2 HTTP/1.1" 200 68935 "xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET /solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A% 22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=group%3A%22Events%22&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2 HTTP/1.1" 200 68935 "xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET /solr/core1/select?q=*%3A*&rows=250&start=0&facet=true&fq=source%3A% 22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=group%3A%22Locations%22&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2 HTTP/1.1" 200 68935 "xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET /solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Performing+Arts%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2 HTTP/1.1" 200 68935 "xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET /solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Music+Events%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2 HTTP/1.1" 200 68935 "xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET /solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Exhibits%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2 HTTP/1.1" 200 68935 "xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET /solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Restaurants%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2 HTTP/1.1" 200 68935 "xx.xxx.xx.xx" "UUUU" "01/May/2013:15:20:02 -0500" "GET /solr/core1/select?q=*%3A*&rows=1&start=0&facet=true&fq=source%3A%22site.com%22&fq=specialinterests%3A%28%22Kid-Friendly%22%29&fq=categories%3A%28%22Museums%22%29&fl=name%2Cnamestring%2Cscore&sort=namestring+desc&wt=javabin&version=2&wt=javabin&version=2 HTTP/1.1" 200 68935 Thanks, Ravi
