Based on your solrconfig.xml settings for the filter and queryResult caches, I
believe Chris's initial guess is correct. After a commit, there is likely
plenty of time spent warming these caches due to the significantly high
autowarm counts.
<filterCache class="solr.FastLRUCache"
size="16384"
initialSize="4096"
autowarmCount="4096"/>
<queryResultCache class="solr.FastLRUCache"
size="8192"
initialSize="8192"
autowarmCount="2048"/>
Suggest you try setting the autowarmcount very low or to zero, and then testing
to confirm the problem.
You might want to monitor if any JVM garbage collections are occurring during
this time, and causing system pauses. With such large caches (nominally stored
in Old Gen) you may be setting yourself up for GCs that take a significant
amount of time and thus add to your delay.
Matt
-----Original Message-----
From: cwhit [mailto:cwhi...@solinkcorp.com]
Sent: Tuesday, August 12, 2014 11:18 AM
To: solr-user@lucene.apache.org
Subject: Re: Updates to index not available immediately as index scales, even
with autoSoftCommit at 1 second
Immediately after triggering the update, this is what is in the logs:
/2014-08-12 12:58:48,774 | [71] | 153414367 [qtp2038499066-4772] INFO
org.apache.solr.update.processor.LogUpdateProcessor – [collection1]
webapp=/solr path=/update params={wt=json} {add=[52627624
(1476251068652322816)]} 0 34
2014-08-12 12:58:49,773 | [71] | 153415369 [commitScheduler-7-thread-1] INFO
org.apache.solr.update.UpdateHandler – start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
2014-08-12 12:58:49,862 | [71] | 153415459 [commitScheduler-7-thread-1] INFO
org.apache.solr.search.SolrIndexSearcher – Opening Searcher@65c48c06 main
2014-08-12 12:58:49,874 | [71] | 153415472 [commitScheduler-7-thread-1] INFO
org.apache.solr.update.UpdateHandler – end_commit_flush/
The end_commit_flush leads me to believe that the soft commit has completed,
but perhaps that thought is wrong. There are no other logs for a while, until
....
/
2014-08-12 13:03:49,556 | [71] | 153715147 [commitScheduler-6-thread-1] INFO
org.apache.solr.update.UpdateHandler – start
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2014-08-12 13:03:49,805 | [71] | 153715402 [commitScheduler-6-thread-1] INFO
org.apache.solr.core.SolrCore – SolrDeletionPolicy.onCommit: commits: num=2
2014-08-12 13:03:49,805 | [71] |
commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@E:\Program
Files (x86)\SolrLive\SolrFiles\Solr\service\solr\data\index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fac1a3c;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2we,generation=3758}
2014-08-12 13:03:49,805 | [71] |
commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@E:\Program
Files (x86)\SolrLive\SolrFiles\Solr\service\solr\data\index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fac1a3c;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2wf,generation=3759}
2014-08-12 13:03:49,805 | [34] | 153715403 [commitScheduler-6-thread-1] INFO
org.apache.solr.core.SolrCore – newest commit generation = 3759
2014-08-12 13:03:49,818 | [34] | 153715415 [commitScheduler-6-thread-1] INFO
org.apache.solr.update.UpdateHandler – end_commit_flush / At this point, the
update is still not present...
/2014-08-12 13:11:45,279 | [81] | 154190876 [searcherExecutor-4-thread-1]
INFO org.apache.solr.core.SolrCore – QuerySenderListener sending requests
to Searcher@65c48c06 main{StandardDirectoryReader(segments_2we:82217:nrt
_qkc(4.6):C8161558/879724:delGen=275 _sra(4.6):C2943436/247953:delGen=51
_r2w(4.6):C1149753/18376:delGen=55 _rgs(4.6):C1468449/648612:delGen=107
_tdl(4.6):C583431/7873:delGen=94 _svo(4.6):C197286/7:delGen=5
_t4d(4.6):C247031/2928:delGen=36 _tkf(4.6):C111429/761:delGen=23
_tch(4.6):C6014/81:delGen=22 _tk5(4.6):C3907/242:delGen=21
_tjv(4.6):C3492/119:delGen=13 _thd(4.6):C5014/241:delGen=24
_tdh(4.6):C5375/437:delGen=30 _tj1(4.6):C5989/15:delGen=6
_tkq(4.6):C1749/36:delGen=6 _tmj(4.6):C961/1:delGen=1
_tlm(4.6):C714/9:delGen=5 _tm6(4.6):C2616 _tlx(4.6):C1105/273:delGen=3
_tly(4.6):C5/2:delGen=1 _tm2(4.6):C1 _tm4(4.6):C1 _tmb(4.6):C1 _tmk(4.6):C5
_tml(4.6):C12 _tmm(4.6):C1 _tmn(4.6):C2/1:delGen=1 _tmo(4.6):C1 _tmp(4.6):C1
_tmr(4.6):C1 _tms(4.6):C1)}
2014-08-12 13:11:45,280 | [81] | 154190877 [searcherExecutor-4-thread-1]
INFO org.apache.solr.core.SolrCore – QuerySenderListener done.
2014-08-12 13:11:45,280 | [81] | 154190877 [searcherExecutor-4-thread-1]
INFO org.apache.solr.handler.component.SpellCheckComponent – Building
spell index for spell checker: suggest
2014-08-12 13:11:45,280 | [81] | 154190877 [searcherExecutor-4-thread-1]
INFO org.apache.solr.spelling.suggest.Suggester – build()/
Still no update...
/2014-08-12 13:12:58,424 | [81] | 154264021 [searcherExecutor-4-thread-1]
INFO org.apache.solr.core.SolrCore – [collection1] Registered new searcher
Searcher@65c48c06 main{StandardDirectoryReader(segments_2we:82217:nrt
_qkc(4.6):C8161558/879724:delGen=275 _sra(4.6):C2943436/247953:delGen=51
_r2w(4.6):C1149753/18376:delGen=55 _rgs(4.6):C1468449/648612:delGen=107
_tdl(4.6):C583431/7873:delGen=94 _svo(4.6):C197286/7:delGen=5
_t4d(4.6):C247031/2928:delGen=36 _tkf(4.6):C111429/761:delGen=23
_tch(4.6):C6014/81:delGen=22 _tk5(4.6):C3907/242:delGen=21
_tjv(4.6):C3492/119:delGen=13 _thd(4.6):C5014/241:delGen=24
_tdh(4.6):C5375/437:delGen=30 _tj1(4.6):C5989/15:delGen=6
_tkq(4.6):C1749/36:delGen=6 _tmj(4.6):C961/1:delGen=1
_tlm(4.6):C714/9:delGen=5 _tm6(4.6):C2616 _tlx(4.6):C1105/273:delGen=3
_tly(4.6):C5/2:delGen=1 _tm2(4.6):C1 _tm4(4.6):C1 _tmb(4.6):C1 _tmk(4.6):C5
_tml(4.6):C12 _tmm(4.6):C1 _tmn(4.6):C2/1:delGen=1 _tmo(4.6):C1 _tmp(4.6):C1
_tmr(4.6):C1 _tms(4.6):C1)}/
There we go! Finally an update! Almost 15 minutes after making the update,
it is visible to queries.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Updates-to-index-not-available-immediately-as-index-scales-even-with-autoSoftCommit-at-1-second-tp4152511p4152554.html
Sent from the Solr - User mailing list archive at Nabble.com.
