Snapinstaller + Overlapping onDeckSearchers Problems
We have been running our solr slaves without autowarming our new searchers
for a long time, but that was causing us 50-75 requests in 20+ seconds
timeframe after every update on the slaves. I have turned on autowarming and
that has fixed our slow response times, but I'm running into occasional
Overlapping onDeckSearchers.
We have replication setup and are using the snapinstaller script every 10
minutes:
/home/solr/bin/snappuller -M util01 -P 18984 -D /home/solr/write/data -S
/home/solr/logs -d /home/solr/read/data -u instruct;
/home/solr/bin/snapinstaller -M util01 -S /home/solr/write/logs -d
/home/solr/read/data -u instruct
Here's what a successful update/commit log looks like:
[14:13:02.510] start
commit(optimize=false,waitFlush=false,waitSearcher=true)
[14:13:02.522] Opening searc...@e9b4bb main
[14:13:02.524] end_commit_flush
[14:13:02.525] autowarming searc...@e9b4bb main from searc...@159e6e8 main
[14:13:02.525]
filterCache{lookups=1809739,hits=1766607,hitratio=0.97,inserts=43211,evictions=0,
size=43154,cumulative_lookups=1809739,cumulative_hits=1766607,cumulative_hitratio=0.97,cumulative_inserts=43211,cumulative_evictions=0}
--
[14:15:42.372] {commit=} 0 159964
[14:15:42.373] /update 0 159964
Here's what a unsuccessful update/commit log looks like, where the /update
took too long and we started another commit:
[21:03:03.829] start
commit(optimize=false,waitFlush=false,waitSearcher=true)
[21:03:03.836] Opening searc...@b2f2d6 main
[21:03:03.836] end_commit_flush
[21:03:03.836] autowarming searc...@b2f2d6 main from searc...@103c520 main
[21:03:03.836]
filterCache{lookups=1062196,hits=1062160,hitratio=0.99,inserts=49144,evictions=0,size=48353,cumulative_lookups=259485564,cumulative_hits=259426904,cumulative_hitratio=0.99,cumulative_inserts=68467,cumulative_evictions=0}
--
[21:23:04.794] start
commit(optimize=false,waitFlush=false,waitSearcher=true)
[21:23:04.794] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
[21:23:04.802] Opening searc...@f11bc main
[21:23:04.802] end_commit_flush
--
[21:24:55.987] {commit=} 0 1312158
[21:24:55.987] /update 0 1312158
I don't understand why this sometimes takes two minutes between the start
commit & /update and sometimes takes 20 minutes? One of our caches has about
~40,000 items, but I can't imagine it taking 20 minutes to autowarm a
searcher.
It would be super handy if the Snapinstaller script would wait until the
previous one was done before starting a new one, but I'm not sure how to
make that happen.
Thanks for any help with this.
best,
cloude
--
VP of Product Development
Instructables.com
http://www.instructables.com/member/lebowski
Re: Snapinstaller + Overlapping onDeckSearchers Problems
Yes, I guess I'm running 40k queries when it starts :) I didn't know that each count was equal to a query. I thought it was just copying the cache entries from the previous searcher, but I guess that wouldn't include new entries. I set it to the size of our filterCache. What should I set the the autowarmCount to if I want to try and fill up the caches? lookups : 8720372 hits : 8676170 hitratio : 0.99 inserts : 44551 evictions : 0 size : 44417 cumulative_lookups : 8720372 cumulative_hits : 8676170 cumulative_hitratio : 0.99 cumulative_inserts : 44551 cumulative_evictions : 0 best, cloude On Wed, Mar 25, 2009 at 8:38 AM, Ryan McKinley wrote: >> >> I don't understand why this sometimes takes two minutes between the start >> commit & /update and sometimes takes 20 minutes? One of our caches has about >> ~40,000 items, but I can't imagine it taking 20 minutes to autowarm a >> searcher. > > > What do your cache configs look like? > > How big is the autowarm count? > > If you have: > class="solr.LRUCache" > size="512" > initialSize="512" > autowarmCount="32"/> > > that will run 32 queries when solr starts. Are you running 40K queries when it starts? > > > ryan > > -- VP of Product Development Instructables.com http://www.instructables.com/member/lebowski
Re: Snapinstaller + Overlapping onDeckSearchers Problems
I set the autowarm to 2000, which only takes about two minutes and resolves my issues. Thanks for your help! best, cloude On Wed, Mar 25, 2009 at 9:34 AM, Ryan McKinley wrote: > It looks like the cache is configured big enough, but the autowarm count is > too big to have good performance. > > Try something smaller and see if that fixes both problems. I imagine even > just warming the most recent 100 queries would precache the most important > ones, but try some higher numbers and see if the performance is acceptable. > > for the filterCache and queryCache, autowarm queries the new index and > caches the results. > > > > > On Mar 25, 2009, at 11:48 AM, Cloude Porteus wrote: > > Yes, I guess I'm running 40k queries when it starts :) I didn't know that >> each count was equal to a query. I thought it was just copying the cache >> entries from the previous searcher, but I guess that wouldn't include new >> entries. I set it to the size of our filterCache. What should I set the >> the >> autowarmCount to if I want to try and fill up the caches? >> >> lookups : 8720372 >> hits : 8676170 >> hitratio : 0.99 >> inserts : 44551 >> evictions : 0 >> size : 44417 >> cumulative_lookups : 8720372 >> cumulative_hits : 8676170 >> cumulative_hitratio : 0.99 >> cumulative_inserts : 44551 >> cumulative_evictions : 0 >> >> best, >> cloude >> >> On Wed, Mar 25, 2009 at 8:38 AM, Ryan McKinley wrote: >> >>> >>>> I don't understand why this sometimes takes two minutes between the >>>> start >>>> commit & /update and sometimes takes 20 minutes? One of our caches has >>>> >>> about >> >>> ~40,000 items, but I can't imagine it taking 20 minutes to autowarm a >>>> searcher. >>>> >>> >>> >>> What do your cache configs look like? >>> >>> How big is the autowarm count? >>> >>> If you have: >>> >>class="solr.LRUCache" >>>size="512" >>>initialSize="512" >>>autowarmCount="32"/> >>> >>> that will run 32 queries when solr starts. Are you running 40K queries >>> >> when it starts? >> >>> >>> >>> ryan >>> >>> >>> >> >> >> -- >> VP of Product Development >> Instructables.com >> >> http://www.instructables.com/member/lebowski >> > > -- VP of Product Development Instructables.com http://www.instructables.com/member/lebowski
