Ever increasing open searchers

[Manish Kumar]

Hi,

I am actually struggling with one of my custom search plugins is causing a
SolrCore to keep many open searchers. This is causing my slave boxes to run
out of disk space in couple of days.

Following is the problemm description :

My solr set-up comprises a master, a repeater and two slaves. Slaves poll
their respective masters every 10 secs.Searches are handled only by slave
nodes while updates are done only to the master node. These are deployed on
Jboss application servers on Linux boxes. The sorl version is 4.0.

The problem we are facing is that the slave boxes run out of hard-disk
space in couple of days which we are currently solving by restarting the
slave nodes once every week. The reason for running out of hard-disk space
is the Jboss holding reference to hundreds of deleted index files.

Deleted file reference are alive because I see SolrCore having many open
searchers. There should be maximum 2 open searchers at any time as per
solrconfig.xml. [image: enter image description here]

Another thing I have been able to establish that SolrCore holds the open
searcher only when a query has been running and a replication cycle changes
the registered searcher for the core while the last query hasn't finished
yet. The searcher when the query began is held in memory by the SolrCore
forever.

This problem comes only when the query uses a SearchPlugin that we have
written. In the search plugin we are closing the *SolrRequest* and
*SolrCore* object. But still it doesn't seem to close the associated
searcher.

SolrCore portfolioIndexCore = container.getCore(portfolioCoreName);
SolrIndexSearcher portfolioIndexSearcher =
portfolioIndexCore.getSearcher().get();

When search is complete :

finally {

  if (null != portfolioSolrReq) portfolioSolrReq.close();

  if (null != portfolioIndexCore) {
portfolioIndexCore.close();
  }
}

I tried changing the above finally block to :

finally {

  if (null != portfolioSolrReq) portfolioSolrReq.close();

  if (null != portfolioIndexCore) {
RefCounted searcher =
portfolioIndexCore.getNewestSearcher(false);
if (searcher.get() != portfolioIndexSearcher) {
  log.warn("Current Searcher for the Core " + portfolioIndexCore
  + " has changed. Old Searcher=[" +
portfolioIndexSearcher + "], new Searcher=["
  + searcher.get() + "]");
  portfolioIndexSearcher.close();
  portfolioIndexSearcher = null;
}
searcher.decref();
portfolioIndexCore.close();
  }
}

But this also doesn't seem to help.

-- 
Regards,
Manish

Re: program termination in solrj

[Manish Kumar]

I think you are missing a call to *add
*
(Collection
http://lucene.apache.org/solr/4_1_0/solr-solrj/org/apache/solr/common/SolrInputDocument.html>>
docs)
or one of the similar methods.

SolrInputDocument doc1 = new SolrInputDocument();
doc1.addField( "id", 23);
doc1.addField( "title", "doc1" );
doc1.addField( "author","Chetan Bhagat" );
doc1.addField( "contents", "I am the best." );
doc1.addField( "date_modified", "12-12-2014" );
server.add(doc1);
server.commit();



On Sat, Dec 21, 2013 at 5:43 PM, Nutan  wrote:

> i am trying to index documents with solrj, this is my code,
>
> import org.apache.solr.client.solrj.SolrServerException;
> import org.apache.solr.client.solrj.impl.*;
> import org.apache.solr.common.SolrInputDocument;
> import java.io.IOException;
>
> public class index {
>
> public static void main(String[] args) throws IOException,
> SolrServerException {
> String url = "http://localhost:8080/solr/document/";;
> HttpSolrServer server = new HttpSolrServer( url );
> server.setMaxRetries(1); // defaults to 0.  > 1 not recommended.
> server.setConnectionTimeout(5000);
> server.setSoTimeout(1000);  // socket read timeout
> server.setDefaultMaxConnectionsPerHost(100);
> server.setMaxTotalConnections(100);
> server.setFollowRedirects(false);  // defaults to false
> SolrInputDocument doc1 = new SolrInputDocument();
> doc1.addField( "id", 23);
> doc1.addField( "title", "doc1" );
> doc1.addField( "author","Chetan Bhagat" );
> doc1.addField( "contents", "I am the best." );
> doc1.addField( "date_modified", "12-12-2014" );
> server.commit();
> }
> }
> After running,(running as a java application) the console on eclipse shows
> this:
>
> Dec 21, 2013 2:07:25 AM org.apache.solr.client.solrj.impl.HttpClientUtil
> createClient INFO: Creating new http client,
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
>
>
> My schema is:
> 
> 
>
>  multiValued="false"/>
>  required="true"
> multiValued="false"/>
>  multiValued="false"/>
>  multiValued="true"/>
> 
>  multivalued="true"/>
>  multiValued="false"/>
>  multiValued="true"/>
>
>  multiValued="true" />
> 
>
>  stored="false" />
> 
>  stored="true" />
> 
> 
>
> Logs does not show any error.Am i following the right process?
> What causes this abnormal termination??
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/program-termination-in-solrj-tp4107706.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
Regards,
Manish