With such a short replication period, the statistics for autowarming won’t be
very good. They will be recent, but not the most popular and likely, because of
the tiny sampling period.
You might try some static warming queries instead.
Find a fairly stable set of most popular queries and put those in
solrconfig.xml like this. We use these for firstSearcher, but you might want to
do it for newSearcher.
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst>
<!-- Top non-numeric queries from August 2011 rush -->
<str name="q">introduction</str>
<str name="q">intermediate</str>
<str name="q">fundamentals</str>
<str name="q">understanding</str>
<str name="q">introductory</str>
<str name="q">precalculus</str>
<str name="q">foundations</str>
<str name="q">microeconomics</str>
<str name="q">microbiology</str>
<str name="q">macroeconomics</str>
<str name="q">discovering</str>
<str name="q">international</str>
<str name="q">mathematics</str>
<str name="q">organizational</str>
<str name="q">criminology</str>
<str name="q">developmental</str>
<str name="q">engineering</str>
</lst>
</arr>
</listener>
wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/
On Aug 27, 2014, at 11:46 AM, Scott Rankin <sran...@crsinc.com> wrote:
> Thanks for the suggestions, Erick. I took a look in the config and it
> turns out that we didn¹t have any auto warming going on. I set the
> filterCache to be about 75% autowarm and the document and query result
> cache to be 50%. That had a marked impact on the performance, with the
> average response time going down from about 1600ms to about 800ms. So
> thank you for that. I also bumped the replication interval up to 20
> seconds to give the autowarm time to go.
>
> However - we¹re still looking at a 4-6x difference in performance between
> the master and the replicas. Once we get past the autowarming, is there
> anything else that you would recommend us looking at?
>
> Thanks,
> Scott
>
>
> Scott Rankin
> Corporate Reimbursement Services, Inc.
> Phone: 617-467-1931
> Email: sran...@crsinc.com
>
>
>
>
> On 8/26/14, 9:01 PM, "Erick Erickson" <erickerick...@gmail.com> wrote:
>
>> What are your autowarm settings? You should be able to alleviate this by
>> configuring these in solrconfig.xml
>> 1> your cache autowarm settings, particularly filterCache and
>> documentResultCache.
>> 2> your newSearcher settings.
>>
>> The point of all the autowarming is that these queries are executed after
>> a
>> commit
>> (or replication in your situation) but _before_ the new searcher serves
>> queries.
>> So here's the sequence
>> 1> a replication (or commit) happens
>> 2> a new searcher is opened and autowarming occurs (cache autowarm and
>> executing any newSearcher queries defined)
>> 3> while <2> is going on, incoming queries are served by the old seacher.
>> 4> when <2> is completed, incoming queries are served by the new searcher
>> 5> the old searcher fulfills its last query and closes itself.
>>
>> Don't be confused by firstSearcher and newSearcher. firstSearcher is fired
>> when the entire server is started (and there are no cache entries to
>> autowarm). newSearcher queries are fired whenever a commit (or
>> replication)
>> happen.
>>
>> HOWEVER.
>> at a 10 second refresh interval, you have the chance of <2> taking more
>> than 10 seconds. you'll see warnings about "too many on deck searchers" if
>> this is the case. For a M/R setup, 10 seconds is _very_ aggressive.
>> Seriously consider SolrCloud in this case if your latency requirements are
>> truly that low.
>>
>> Having your master serve queries is also suspect. It's _busy_ indexing and
>> you're asking it to add query load too. May not really matter a lot
>> depends
>> on the indexing rate...
>>
>> Best,
>> Erick
>>
>>
>> On Tue, Aug 26, 2014 at 1:32 PM, Scott Rankin <sran...@crsinc.com> wrote:
>>
>>> Hi all,
>>>
>>> I have a scenario here and I'd love some advice on what my options are.
>>> We have one Solr master and two read replicas. The replicas query the
>>> master every 10 seconds because we need relatively quick availability of
>>> new documents. We balance read queries across all three servers and
>>> send
>>> writes only to the master.
>>>
>>> The problem that I'm seeing is that average query time is vastly higher
>>> on
>>> the replicas than on the master - 150ms vs 1600ms. What I've noticed is
>>> that immediately after a replication, a query against the replica can
>>> take
>>> up to 5 seconds. Then subsequent queries are faster until the next
>>> replication. On one level this makes sense, since when there's a
>>> change
>>> to the index I imagine that the caches get flushed. But this is really
>>> bogging down performance on a pretty high number of queries.
>>>
>>> We should have plenty of OS disk cache - the server has 12 GB of RAM,
>>> and
>>> the apps on the server are only using up about 6. Our index is 2.7 GB,
>>> so
>>> it should fit in the OS disk cache. Are there any other factors that I
>>> can
>>> look at to eliminate these slow queries?
>>>
>>> Thanks,
>>> Scott
>>>
>>> Scott Rankin
>>> Corporate Reimbursement Services, Inc.
>>> Phone: 617-467-1931
>>> Email: sran...@crsinc.com<mailto:sran...@crsinc.com>
>>>
>>> This email message contains information that Corporate Reimbursement
>>> Services, Inc. considers confidential and/or proprietary, or may later
>>> designate as confidential and proprietary. It is intended only for use
>>> of
>>> the individual or entity named above and should not be forwarded to any
>>> other persons or entities without the express consent of Corporate
>>> Reimbursement Services, Inc., nor should it be used for any purpose
>>> other
>>> than in the course of any potential or actual business relationship with
>>> Corporate Reimbursement Services, Inc. If the reader of this message is
>>> not
>>> the intended recipient, or the employee or agent responsible to deliver
>>> it
>>> to the intended recipient, you are hereby notified that any
>>> dissemination,
>>> distribution, or copying of this communication is strictly prohibited.
>>> If
>>> you have received this communication in error, please notify sender
>>> immediately and destroy the original message.
>>>
>>> Internal Revenue Service regulations require that certain types of
>>> written
>>> advice include a disclaimer. To the extent the preceding message
>>> contains
>>> advice relating to a Federal tax issue, unless expressly stated
>>> otherwise
>>> the advice is not intended or written to be used, and it cannot be used
>>> by
>>> the recipient or any other taxpayer, for the purpose of avoiding Federal
>>> tax penalties, and was not written to support the promotion or
>>> marketing of
>>> any transaction or matter discussed herein.
>>>
>
> This email message contains information that Corporate Reimbursement
> Services, Inc. considers confidential and/or proprietary, or may later
> designate as confidential and proprietary. It is intended only for use of the
> individual or entity named above and should not be forwarded to any other
> persons or entities without the express consent of Corporate Reimbursement
> Services, Inc., nor should it be used for any purpose other than in the
> course of any potential or actual business relationship with Corporate
> Reimbursement Services, Inc. If the reader of this message is not the
> intended recipient, or the employee or agent responsible to deliver it to the
> intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication is strictly prohibited. If you
> have received this communication in error, please notify sender immediately
> and destroy the original message.
>
> Internal Revenue Service regulations require that certain types of written
> advice include a disclaimer. To the extent the preceding message contains
> advice relating to a Federal tax issue, unless expressly stated otherwise the
> advice is not intended or written to be used, and it cannot be used by the
> recipient or any other taxpayer, for the purpose of avoiding Federal tax
> penalties, and was not written to support the promotion or marketing of any
> transaction or matter discussed herein.
