I'm using trunk, but I set a breakpoint on SegmentReader.isDeleted()
on an index with deletions, and I couldn't get it to be called.
numDocs : 26
maxDoc : 130
reader:SolrIndexReader{this=1935e6f,r=readonlymultisegmentrea...@1935e6f,segments=5}
-Yonik
http://www.lucidimagination.com
On Thu, Feb 26, 2009 at 4:55 PM, Matthew Runo <mr...@zappos.com> wrote:
> I see a ReadOnlySegmentReader now - we're on an optimized index now which
> gets around the isDeleted() check.
>
> (solr4, optimized)
> searcherName : searc...@260f8e27 main
> caching : true
> numDocs : 139583
> maxDoc : 139583
> readerImpl : ReadOnlySegmentReader
> readerDir :
> org.apache.lucene.store.NIOFSDirectory@/opt/solr-data/zeta-main/index
> indexVersion : 1233423823917
> openedAt : Thu Feb 26 13:29:25 PST 2009
> registeredAt : Thu Feb 26 13:29:42 PST 2009
> warmupTime : 16910
>
> (solr1, non optimized)
> searcherName : searc...@36be11a1 main
> caching : true
> numDocs : 139561
> maxDoc : 139591
> readerImpl : ReadOnlyMultiSegmentReader
> readerDir :
> org.apache.lucene.store.NIOFSDirectory@/opt/solr-data/zeta-main/index
> indexVersion : 1233423823924
> openedAt : Thu Feb 26 13:48:16 PST 2009
> registeredAt : Thu Feb 26 13:49:11 PST 2009
> warmupTime : 54785
>
> I did a thread dump against the optimized server just now, but didn't find
> anything blocked to check which reader was actually in use this time.
>
> Thanks for your time!
>
> Matthew Runo
> Software Engineer, Zappos.com
> mr...@zappos.com - 702-943-7833
>
> On Feb 26, 2009, at 1:39 PM, Yonik Seeley wrote:
>
>> That's interesting.
>> We should be using read-only readers, which should not synchronize on
>> the deleted docs check. But as your stack trace shows, you're using
>> SegmentReader and MultiSegmentReader.
>>
>> Right now, if I look at the admin/statistics page at the searcher, it
>> shows the following for the reader:
>>
>>
>> reader:SolrIndexReader{this=42f352,r=readonlymultisegmentrea...@42f352,segments=6}
>>
>> Hopefully the fact that it's a ReadOnlyMultiSegmentReader means that
>> it contains ReadOnlySegmentReader instances, which don't synchronize
>> on isDeleted.
>>
>> What do you see?
>>
>> -Yonik
>>
>> On Thu, Feb 26, 2009 at 4:09 PM, Matthew Runo <mr...@zappos.com> wrote:
>>>
>>> Hello folks!
>>>
>>> I was under the impression that this sync bottleneck was fixed in recent
>>> versions of Solr/Lucene, but we're seeing it with 1.4-dev right now. When
>>> we
>>> load test a server with >100 threads (using jmeter), we see several
>>> threads
>>> all blocked at the same spot:
>>>
>>> "http-8080-exec-505" - Thread t...@594
>>> java.lang.Thread.State: BLOCKED on
>>> org.apache.lucene.index.segmentrea...@2b6f5d18 owned by:
>>> http-8080-exec-434
>>> at
>>> org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:737)
>>> at
>>>
>>> org.apache.lucene.index.MultiSegmentReader.isDeleted(MultiSegmentReader.java:266)
>>> at
>>>
>>> org.apache.solr.search.function.FunctionQuery$AllScorer.next(FunctionQuery.java:118)
>>> at
>>>
>>> org.apache.solr.search.function.FunctionQuery$AllScorer.skipTo(FunctionQuery.java:137)
>>> at
>>>
>>> org.apache.lucene.search.BooleanScorer2$SingleMatchScorer.skipTo(BooleanScorer2.java:170)
>>> at
>>> org.apache.lucene.search.ReqOptSumScorer.score(ReqOptSumScorer.java:76)
>>> at
>>> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:357)
>>> at
>>> org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:320)
>>> at
>>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:136)
>>> at org.apache.lucene.search.Searcher.search(Searcher.java:126)
>>> at org.apache.lucene.search.Searcher.search(Searcher.java:105)
>>> at
>>>
>>> org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1231)
>>> at
>>>
>>> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:917)
>>> at
>>>
>>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:338)
>>> at
>>>
>>> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:164)
>>> at
>>>
>>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:171)
>>> at
>>>
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
>>> at
>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
>>> at
>>>
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
>>> at
>>>
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>> at
>>>
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>> at
>>>
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>> at
>>>
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
>>> at
>>>
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>>> at
>>>
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>> at
>>>
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>> at
>>>
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>> at
>>>
>>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:879)
>>> at
>>>
>>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:719)
>>> at
>>>
>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2080)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>> at java.lang.Thread.run(Thread.java:619)
>>>
>>> Locked ownable synchronizers:
>>> - locked
>>> java.util.concurrent.locks.reentrantlock$nonfairs...@4d54c7be
>>>
>>>
>>> I checked the Lucene SVN and it looks like that's still appearing to be a
>>> bottleneck.
>>>
>>>
>>> http://svn.apache.org/viewvc/lucene/java/trunk/src/java/org/apache/lucene/index/SegmentReader.java?view=markup
>>>
>>> Any tips?
>>>
>>> Thanks for your time!
>>>
>>> Matthew Runo
>>> Software Engineer, Zappos.com
>>> mr...@zappos.com - 702-943-7833
>>>
>>>
>>
>>
>>
>> --
>> -Yonik
>> http://www.lucidimagination.com
>>
>
>
