From: Matthew Runo <[EMAIL PROTECTED]>
To: solr-user@lucene.apache.org
Sent: Tuesday, August 19, 2008 4:26:59 PM
Subject: Re: Deadlock in lucene?
I know this isn't really the place for this, so please forgive me -
but does this patch look reasonably safe to use to skip the isDeleted
check inside of FunctionQuery?
My reasoning behind this is that many people (us included) will be
building the index on a separate server, and then using the
replication scripts to publish the files out to several read-only
servers. On those instances, deletedDocs would always be empty, since
it's a read only instance - and so we can conveniently skip the
Lucene
code in question. This flag would also be good for other
optimizations
that can only be made when you assume the index is read-only.
Solr seems to work with the flag set - any reasons why this will
crash
and/or kill my kitten?
(please forgive my posting this here instead of in solr-dev!)
Index: src/java/org/apache/solr/search/FunctionQParser.java
===================================================================
--- src/java/org/apache/solr/search/FunctionQParser.java (revision
687135)
+++ src/java/org/apache/solr/search/FunctionQParser.java Tue Aug
19
11:08:45 PDT 2008
@@ -49,7 +49,7 @@
}
***/
- return new FunctionQuery(vs);
+ return new FunctionQuery(vs,
req.getSchema().getSolrConfig().isReadOnly() );
}
/**
Index: src/java/org/apache/solr/search/function/FunctionQuery.java
===================================================================
--- src/java/org/apache/solr/search/function/FunctionQuery.java
(revision 687135)
+++ src/java/org/apache/solr/search/function/FunctionQuery.java
Tue
Aug 19 11:08:45 PDT 2008
@@ -31,12 +31,14 @@
*/
public class FunctionQuery extends Query {
ValueSource func;
+ Boolean readOnly;
/**
* @param func defines the function to be used for scoring
*/
- public FunctionQuery(ValueSource func) {
+ public FunctionQuery(ValueSource func, Boolean readOnly) {
this.func=func;
+ this.readOnly=readOnly;
}
/** @return The associated ValueSource */
@@ -113,7 +115,7 @@
if (doc>=maxDoc) {
return false;
}
- if (reader.isDeleted(doc)) continue;
+ if (!readOnly && reader.isDeleted(doc)) continue;
// todo: maybe allow score() to throw a specific exception
// and continue on to the next document if it is thrown...
// that may be useful, but exceptions aren't really good
Index: src/java/org/apache/solr/core/Config.java
===================================================================
--- src/java/org/apache/solr/core/Config.java (revision 687135)
+++ src/java/org/apache/solr/core/Config.java Tue Aug 19
11:08:45 PDT
2008
@@ -45,6 +45,8 @@
private final String name;
private final SolrResourceLoader loader;
+ private Boolean readOnly;
+
/**
* @deprecated Use [EMAIL PROTECTED] #Config(SolrResourceLoader, String,
InputStream, String)} instead.
*/
@@ -254,6 +256,19 @@
return val!=null ? Double.parseDouble(val) : def;
}
+ /**
+ * Is the index set up to be readOnly? If so, this will cause the
FunctionQuery stuff to not check
+ * for deleted documents.
+ * @return boolean readOnly
+ */
+ public boolean isReadOnly() {
+ if( this.readOnly == null ){
+ readOnly = getBool("/mainIndex/readOnly", false);
+ }
+
+ return readOnly;
+ }
+
// The following functions were moved to ResourceLoader
//-----------------------------------------------------------------------------
Index: example/solr/conf/solrconfig.xml
===================================================================
--- example/solr/conf/solrconfig.xml (revision 687135)
+++ example/solr/conf/solrconfig.xml Tue Aug 19 11:13:13 PDT 2008
@@ -114,6 +114,12 @@
This is not needed if lock type is 'none' or 'single'
-->
false
+
+
+ false
use
--- end patch ---
On Aug 18, 2008, at 8:04 PM, Yonik Seeley wrote:
It's not a deadlock (just a synchronization bottleneck) , but it
is a
known issue in Lucene and there has been some progress in improving
the situation.
-Yonik
On Mon, Aug 18, 2008 at 10:55 PM, Matthew Runo
wrote:
Hello folks!
I was just wondering if anyone else has seen this issue under heavy
load. We
had some servers set to very high thread limits (12 core servers
with 32
gigs of ram), and found several threads would end up in this
state....
Name: http-8080-891
State: BLOCKED on [EMAIL PROTECTED]
owned by:
http-8080-191
Total blocked: 97,926 Total waited: 16
Stack trace:
org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:
674)
org.apache.solr.search.function.FunctionQuery
$AllScorer.next(FunctionQuery.java:116)
org
.apache
.lucene
.util.ScorerDocQueue.topNextAndAdjustElsePop(ScorerDocQueue.java:
116)
org
.apache
.lucene
.search
.DisjunctionSumScorer
.advanceAfterCurrent(DisjunctionSumScorer.java:
175)
org
.apache
.lucene
.search.DisjunctionSumScorer.skipTo(DisjunctionSumScorer.java:228)
org
.apache.lucene.search.ReqOptSumScorer.score(ReqOptSumScorer.java:
76)
org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:
357)
org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:
320)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
137)
org.apache.lucene.search.Searcher.search(Searcher.java:126)
org.apache.lucene.search.Searcher.search(Searcher.java:105)
org
.apache
.solr
.search
.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:
1148)
org
.apache
.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:
834)
org
.apache
.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:
269)
org
.apache
.solr.handler.component.QueryComponent.process(QueryComponent.java:
160)
org
.apache
.solr
.handler
.component.SearchHandler.handleRequestBody(SearchHandler.java:169)
org
.apache
.solr
.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:
128)
org.apache.solr.core.SolrCore.execute(SolrCore.java:1143)
org
.apache
.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:
338)
org
.apache
.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
272)
org
.apache
.catalina
.core
.ApplicationFilterChain
.internalDoFilter(ApplicationFilterChain.java:235)
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
206)
org
.apache
.catalina
.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
org
.apache
.catalina
.core.StandardContextValve.invoke(StandardContextValve.java:175)
org
.apache
.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
org
.apache
.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
org
.apache
.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:
109)
org
.apache
.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
286)
org
.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
844)
org.apache.coyote.http11.Http11Protocol
$Http11ConnectionHandler.process(Http11Protocol.java:583)
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:
447)
java.lang.Thread.run(Thread.java:619)
Thanks for your time!
Matthew Runo
Software Engineer, Zappos.com
[EMAIL PROTECTED] - 702-943-7833
