Thanks Craig,
I'm beginning to think that frequent commits are causing the performance
degradation, even though there shouldn't be any *concurrent* commits in
our system. There's just one indexing thread per Solr core,
waitFlush/waitSearcher are set to true.
For now I'd pretty much rule out RAM shortage (-Xmx10096m) and disk
activity, which never really went above 1 MB/s during my tests.
This is our solrconfig.xml. To reproduce the issue, we create ~300
indexing jobs. For each job 1 document is indexed before committing.
This should result in many commit requests in serial order. Below the
config is the log output created by one of these jobs.
So I guess we'll just have to avoid short time spans between every two
commit requests.
=================================================================
<?xml version="1.0" encoding="UTF-8" ?>
<config>
<abortOnConfigurationError>${solr.abortOnConfigurationError:true}
</abortOnConfigurationError>
<dataDir>${solr.core.dataDir}</dataDir>
<indexDefaults>
<useCompoundFile>false</useCompoundFile>
<mergeFactor>10</mergeFactor>
<ramBufferSizeMB>32</ramBufferSizeMB>
<maxFieldLength>10000</maxFieldLength>
<writeLockTimeout>1000</writeLockTimeout>
<commitLockTimeout>10000</commitLockTimeout>
<lockType>native</lockType>
</indexDefaults>
<mainIndex>
<useCompoundFile>false</useCompoundFile>
<ramBufferSizeMB>32</ramBufferSizeMB>
<mergeFactor>10</mergeFactor>
<unlockOnStartup>false</unlockOnStartup>
<reopenReaders>true</reopenReaders>
<deletionPolicy class="solr.SolrDeletionPolicy">
<str name="maxCommitsToKeep">1</str>
<str name="maxOptimizedCommitsToKeep">0</str>
</deletionPolicy>
<infoStream file="INFOSTREAM.txt">false</infoStream>
</mainIndex>
<jmx />
<updateHandler class="solr.DirectUpdateHandler2">
</updateHandler>
<query>
<maxBooleanClauses>50000</maxBooleanClauses>
<filterCache class="solr.FastLRUCache" size="512"
initialSize="512" autowarmCount="0" />
<queryResultCache class="solr.LRUCache" size="512"
initialSize="512" autowarmCount="0" />
<documentCache class="solr.LRUCache" size="512"
initialSize="512" autowarmCount="0" />
<queryResultWindowSize>20</queryResultWindowSize>
<queryResultMaxDocsCached>200</queryResultMaxDocsCached>
<useColdSearcher>false</useColdSearcher>
<maxWarmingSearchers>2</maxWarmingSearchers>
</query>
<requestDispatcher handleSelect="true">
<requestParsers enableRemoteStreaming="true"
multipartUploadLimitInKB="2048000" />
<httpCaching lastModifiedFrom="openTime">
</httpCaching>
</requestDispatcher>
<!--searchComponent name="geoDistance"
class="nl.jteam.search.solrext.spatial.GeoDistanceComponent">
<str name="latField">lat</str>
<str name="lngField">lng</str>
</searchComponent-->
<requestHandler name="dismax" class="solr.DisMaxRequestHandler"
default="true">
<lst name="defaults">
<str name="facet">true</str>
<str name="q.alt">*:*</str>
<str name="qs">1</str>
<int name="ps">0</int>
<str name="mm">0</str>
</lst>
</requestHandler>
<requestHandler name="standard" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
</lst>
</requestHandler>
<requestHandler name="/update/javabin"
class="solr.BinaryUpdateRequestHandler" />
<requestHandler name="/update" class="solr.XmlUpdateRequestHandler" />
...
</config>
=================================================================
[#|2011-05-01T13:42:10.407+0200|INFO|glassfish3.0.1|dd.doubled.jobservice.PipelineImpl|_ThreadID=494;_ThreadName=Thread-1;|Attempting
to execute job (id=16643,type=solr-index)|#]
[#|2011-05-01T13:42:10.424+0200|INFO|glassfish3.0.1|dd.doubled.jobservice.handler.solrindexer.IndexDataCommand|_ThreadID=494;_ThreadName=Thread-1;|Indexed
1 documents in 0.0 seconds (333 dps)|#]
[#|2011-05-01T13:42:10.474+0200|INFO|glassfish3.0.1|org.apache.solr.core.SolrCore|_ThreadID=31;_ThreadName=Thread-1;|SolrDeletionPolicy.onInit:
commits:num=1
commit{dir=/opt/solr/cores/item/data/index,segFN=segments_4we,version=1300132289520,generation=6350,filenames=[_bxr.fnm,
_bn1.fdt, _8z3.frq, _bu6.prx, _bxs.fnm, _bn1.fdx, _2se.fnm, _bxt.fnm,
_cnz_1.del, _bxs.tii, _bn1.tii, _co0.fnm, _bxs.tis, _bxo.fnm, _bql.tis,
_8z3.fdt, _bn1.tis, _8z3.fdx, _bxm.nrm, _bxs.nrm, _5ex.fnm, _bxp.fnm,
_bql.tii, _bk3.fnm, _bug.frq, _btv.nrm, _co0.prx, _cny.fdt, _cny.fdx,
_bxu.fnm, _bu6.fnm, _cny.nrm, _2se.prx, _bxv.tii, _bur.fnm, _cny.tis,
_aku.nrm, _cnz.prx, _btv.tii, _bql_1.del, _bur.prx, _5ex.prx, _cny.tii,
_bxt.prx, _4od.fdx, _bxp.tii, _aku.fdt, _4od.fdt, _bxv.tis, _9pt.fnm,
_btv.tis, _aku.fdx, _bxp.tis, _bvn.prx, _bxr.frq, _8z3.fnm, _bur.nrm,
_bxu.prx, _bug.tis, _bxm.fdt, _bf9.fdt, _5ex.fdt, _bxm.fdx, _5ex.fdx,
_bxt.frq, _2se.frq, _bk3.prx, _bxm.frq, _5ex.nrm, _bxn.tii, _9pt.prx,
_bxr.nrm, _bve.frq, _5zc.fnm, _bxv.prx, _bxv.fdx, _5ex.frq, _bv2.fnm,
_bxp.frq, _bxv.fdt, _bug.tii, _cny.frq, _cnz.fdt, _bxq.fnm, _btv.frq,
_bxn.nrm, _co0.tis, _btv.fdt, _bxn.fdt, _cnz.fdx, _bxr_1.del, _bxn.fdx,
_cnz.tii, _bxq.prx, _bql.fnm, _8z3.prx, _bxp.nrm, _bve.fdx, _bve.fdt,
_aku.frq, _bu6.frq, _bf9_c.del, _co0.tii, _btv.fdx, _4od.nrm, _bur.frq,
_bxn.tis, _5zc.prx, _8z3_dx.del, _bf9.fdx, _bxo.prx, _cnz.tis, _bxs.prx,
_bu6.nrm, _bve.prx, _co0.fdt, _9pt_9w.del, _bf9.prx, _co0.fdx, _bvn.tii,
_bve.tii, _bf9.nrm, _bvn.tis, _bxp.prx, _bql.nrm, _cnz.frq, _cny.fnm,
_9pt.nrm, _bxm.prx, _bxn.frq, _bxt.nrm, _bvn.nrm, _bql.frq, _4od.frq,
_bve.fnm, _bve.nrm, _5zc.fdt, _aku.fnm, _cny.prx, _bxt.tis, _4od_8.del,
_bv2.frq, _bn1.prx, _5ex.tis, _bv2.tis, _btv.prx, _2se.fdx, _bur.fdx,
_bur.tis, _bv2.tii, _bxt.tii, _5zc.fdx, _bur.fdt, _cnz.nrm, _5zc_b.del,
_5zc.tii, _5ex.tii, _2se.fdt, _bxr.tis, _9pt.fdx, _bxv.frq, _5zc.frq,
_5zc.tis, _9pt.fdt, _btv.fnm, _bur.tii, _bxr.tii, _bvn.fnm, _2se.nrm,
_bxr.fdx, _bf9.frq, _aku.tis, _aku.tii, _bu6.tii, _bu6.fdx, _bn1.fnm,
_bxr.fdt, _bk3.tis, _bxo.fdt, _4od.tii, _bu6.tis, _bu6.fdt, _bxo.fdx,
_bxp.fdt, _bk3.tii, _aku.prx, _bxp.fdx, _4od.tis, _bxq.frq, _bxm.tis,
_bug.fdx, _bvn.fdx, _bxp_2.del, _9pt.tii, _bvn.fdt, _bxs.frq,
_aku_a.del, _2se_kw.del, _bk3.nrm, _9pt.tis, _4od.prx, _co0.nrm,
_bv2.prx, _bxo.tis, _bf9.tis, _5ex_8.del, _bxq.nrm, _bug.prx, _bxu.nrm,
_bk3.fdx, _bxo.tii, _bk3_3.del, _bk3.fdt, segments_4we, _bxq.tis,
_bxu.tis, _bxr.prx, _bk3.frq, _bug.fnm, _bxn.fnm, _2se.tis, _bxo.frq,
_cnz.fnm, _bxq.tii, _bf9.tii, _bxu.tii, _4od.fnm, _2se.tii, _bxu.frq,
_bn1.nrm, _8z3.nrm, _bxv.nrm, _9pt.frq, _bxo.nrm, _8z3.tis, _bxn.prx,
_bxv.fnm, _bvn.frq, _bv2.fdt, _btv_1.del, _co0.frq, _bv2.fdx, _bql.fdx,
_bv2.nrm, _bn1.frq, _bve.tis, _bf9.fnm, _bql.fdt, _bxm.tii, _bxm.fnm,
_bql.prx, _8z3.tii, _bug.fdt, _cny_1.del, _bxq_1.del, _bug.nrm, _5zc.nrm]|#]
[#|2011-05-01T13:42:10.474+0200|INFO|glassfish3.0.1|org.apache.solr.core.SolrCore|_ThreadID=31;_ThreadName=Thread-1;|newest
commit = 1300132289520|#]
[#|2011-05-01T13:42:10.475+0200|INFO|glassfish3.0.1|org.apache.solr.update.processor.UpdateRequestProcessor|_ThreadID=31;_ThreadName=Thread-1;|{add=[137756535]}
0 8|#]
[#|2011-05-01T13:42:10.475+0200|INFO|glassfish3.0.1|org.apache.solr.core.SolrCore|_ThreadID=31;_ThreadName=Thread-1;|[item]
webapp=/solr path=/update/javabin params={wt=javabin&version=1} status=0
QTime=8 |#]
[#|2011-05-01T13:42:10.576+0200|INFO|glassfish3.0.1|dd.doubled.jobservice.handler.solrindexer.IndexDataCommand|_ThreadID=494;_ThreadName=Thread-1;|Overall
indexing performance: 1 documents in 0.2 seconds (6 dps)|#]
[#|2011-05-01T13:42:10.578+0200|INFO|glassfish3.0.1|org.apache.solr.update.UpdateHandler|_ThreadID=19;_ThreadName=Thread-1;|start
commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false)|#]
[#|2011-05-01T13:42:11.972+0200|INFO|glassfish3.0.1|org.apache.solr.core.SolrCore|_ThreadID=19;_ThreadName=Thread-1;|SolrDeletionPolicy.onCommit:
commits:num=2
commit{dir=/opt/solr/cores/item/data/index,segFN=segments_4we,version=1300132289520,generation=6350,filenames=[_bxr.fnm,
_bn1.fdt, _8z3.frq, _bu6.prx, _bxs.fnm, _bn1.fdx, _2se.fnm, _bxt.fnm,
_cnz_1.del, _bxs.tii, _bn1.tii, _co0.fnm, _bxs.tis, _bxo.fnm, _bql.tis,
_8z3.fdt, _bn1.tis, _8z3.fdx, _bxm.nrm, _bxs.nrm, _5ex.fnm, _bxp.fnm,
_bql.tii, _bk3.fnm, _bug.frq, _btv.nrm, _co0.prx, _cny.fdt, _cny.fdx,
_bxu.fnm, _bu6.fnm, _cny.nrm, _2se.prx, _bxv.tii, _bur.fnm, _cny.tis,
_aku.nrm, _cnz.prx, _btv.tii, _bql_1.del, _bur.prx, _5ex.prx, _cny.tii,
_bxt.prx, _4od.fdx, _bxp.tii, _aku.fdt, _4od.fdt, _bxv.tis, _9pt.fnm,
_btv.tis, _aku.fdx, _bxp.tis, _bvn.prx, _bxr.frq, _8z3.fnm, _bur.nrm,
_bxu.prx, _bug.tis, _bxm.fdt, _bf9.fdt, _5ex.fdt, _bxm.fdx, _5ex.fdx,
_bxt.frq, _2se.frq, _bk3.prx, _bxm.frq, _5ex.nrm, _bxn.tii, _9pt.prx,
_bxr.nrm, _bve.frq, _5zc.fnm, _bxv.prx, _bxv.fdx, _5ex.frq, _bv2.fnm,
_bxp.frq, _bxv.fdt, _bug.tii, _cny.frq, _cnz.fdt, _bxq.fnm, _btv.frq,
_bxn.nrm, _co0.tis, _btv.fdt, _bxn.fdt, _cnz.fdx, _bxr_1.del, _bxn.fdx,
_cnz.tii, _bxq.prx, _bql.fnm, _8z3.prx, _bxp.nrm, _bve.fdx, _bve.fdt,
_aku.frq, _bu6.frq, _bf9_c.del, _co0.tii, _btv.fdx, _4od.nrm, _bur.frq,
_bxn.tis, _5zc.prx, _8z3_dx.del, _bf9.fdx, _bxo.prx, _cnz.tis, _bxs.prx,
_bu6.nrm, _bve.prx, _co0.fdt, _9pt_9w.del, _bf9.prx, _co0.fdx, _bvn.tii,
_bve.tii, _bf9.nrm, _bvn.tis, _bxp.prx, _bql.nrm, _cnz.frq, _cny.fnm,
_9pt.nrm, _bxm.prx, _bxn.frq, _bxt.nrm, _bvn.nrm, _bql.frq, _4od.frq,
_bve.fnm, _bve.nrm, _5zc.fdt, _aku.fnm, _cny.prx, _bxt.tis, _4od_8.del,
_bv2.frq, _bn1.prx, _5ex.tis, _bv2.tis, _btv.prx, _2se.fdx, _bur.fdx,
_bur.tis, _bv2.tii, _bxt.tii, _5zc.fdx, _bur.fdt, _cnz.nrm, _5zc_b.del,
_5zc.tii, _5ex.tii, _2se.fdt, _bxr.tis, _9pt.fdx, _bxv.frq, _5zc.frq,
_5zc.tis, _9pt.fdt, _btv.fnm, _bur.tii, _bxr.tii, _bvn.fnm, _2se.nrm,
_bxr.fdx, _bf9.frq, _aku.tis, _aku.tii, _bu6.tii, _bu6.fdx, _bn1.fnm,
_bxr.fdt, _bk3.tis, _bxo.fdt, _4od.tii, _bu6.tis, _bu6.fdt, _bxo.fdx,
_bxp.fdt, _bk3.tii, _aku.prx, _bxp.fdx, _4od.tis, _bxq.frq, _bxm.tis,
_bug.fdx, _bvn.fdx, _bxp_2.del, _9pt.tii, _bvn.fdt, _bxs.frq,
_aku_a.del, _2se_kw.del, _bk3.nrm, _9pt.tis, _4od.prx, _co0.nrm,
_bv2.prx, _bxo.tis, _bf9.tis, _5ex_8.del, _bxq.nrm, _bug.prx, _bxu.nrm,
_bk3.fdx, _bxo.tii, _bk3_3.del, _bk3.fdt, segments_4we, _bxq.tis,
_bxu.tis, _bxr.prx, _bk3.frq, _bug.fnm, _bxn.fnm, _2se.tis, _bxo.frq,
_cnz.fnm, _bxq.tii, _bf9.tii, _bxu.tii, _4od.fnm, _2se.tii, _bxu.frq,
_bn1.nrm, _8z3.nrm, _bxv.nrm, _9pt.frq, _bxo.nrm, _8z3.tis, _bxn.prx,
_bxv.fnm, _bvn.frq, _bv2.fdt, _btv_1.del, _co0.frq, _bv2.fdx, _bql.fdx,
_bv2.nrm, _bn1.frq, _bve.tis, _bf9.fnm, _bql.fdt, _bxm.tii, _bxm.fnm,
_bql.prx, _8z3.tii, _bug.fdt, _cny_1.del, _bxq_1.del, _bug.nrm, _5zc.nrm]
commit{dir=/opt/solr/cores/item/data/index,segFN=segments_4wf,version=1300132289521,generation=6351,filenames=[_bxr.fnm,
_bn1.fdt, _8z3.frq, _bu6.prx, _bxs.fnm, _bn1.fdx, _2se.fnm, _bxt.fnm,
_cnz_1.del, _bxs.tii, _bn1.tii, _co0.fnm, _bxs.tis, _bxo.fnm, _bql.tis,
_8z3.fdt, _bn1.tis, _8z3.fdx, _bxm.nrm, _bxs.nrm, _5ex.fnm, _bxp.fnm,
_bql.tii, _bk3.fnm, _bug.frq, _btv.nrm, _co0.prx, _cny.fdt, _cny.fdx,
_co1.fdt, _bxu.fnm, _bu6.fnm, _cny.nrm, _co1.fdx, _2se.prx, _bxv.tii,
_bur.fnm, _cny.tis, _aku.nrm, _cnz.prx, _co1.prx, _btv.tii, _bql_1.del,
_bur.prx, _5ex.prx, _cny.tii, _bxt.prx, _co1.nrm, _4od.fdx, _bxp.tii,
_aku.fdt, _4od.fdt, _bxv.tis, _9pt.fnm, _btv.tis, _aku.fdx, _bxp.tis,
_bvn.prx, _bxr.frq, _8z3.fnm, _bur.nrm, _bxu.prx, _bug.tis, _bxm.fdt,
_bf9.fdt, _5ex.fdt, _bxm.fdx, _5ex.fdx, _bxt.frq, _2se.frq, _bk3.prx,
_bxm.frq, _5ex.nrm, _bxn.tii, _9pt.prx, _bxr.nrm, _bve.frq, _5zc.fnm,
_bxv.prx, _bxv.fdx, _5ex.frq, _bv2.fnm, _bxp.frq, _bxv.fdt, _bug.tii,
_cny.frq, _cnz.fdt, _bxq.fnm, _btv.frq, _bxn.nrm, _co0.tis, _btv.fdt,
_bxn.fdt, _cnz.fdx, _bxr_1.del, _bxn.fdx, _cnz.tii, _bxq.prx, _bql.fnm,
_8z3.prx, _bxp.nrm, _bve.fdx, _bve.fdt, _aku.frq, _bu6.frq, _bf9_c.del,
_co0.tii, _btv.fdx, _4od.nrm, _bur.frq, _bxn.tis, _5zc.prx, _8z3_dx.del,
_bf9.fdx, _bxo.prx, _cnz.tis, _bxs.prx, _bu6.nrm, _bve.prx, _co0.fdt,
_9pt_9w.del, _bf9.prx, _co0.fdx, _bvn.tii, _bve.tii, _bf9.nrm, _bvn.tis,
_bxp.prx, _bql.nrm, _cnz.frq, _cny.fnm, _9pt.nrm, _bxm.prx, _bxn.frq,
_bxt.nrm, _bvn.nrm, _bql.frq, _4od.frq, _bve.fnm, _bve.nrm, _5zc.fdt,
_aku.fnm, _cny.prx, _bxt.tis, _4od_8.del, _bv2.frq, _bn1.prx, _5ex.tis,
_bv2.tis, _btv.prx, _2se.fdx, _co0_1.del, _bur.fdx, _bur.tis, _bv2.tii,
_bxt.tii, _5zc.fdx, _bur.fdt, _cnz.nrm, _5zc_b.del, _5zc.tii, _5ex.tii,
_2se.fdt, _bxr.tis, _9pt.fdx, _bxv.frq, _5zc.frq, _5zc.tis, _9pt.fdt,
_btv.fnm, _bur.tii, _bxr.tii, _bvn.fnm, _2se.nrm, _bxr.fdx, _bf9.frq,
_aku.tis, _co1.fnm, _aku.tii, _bu6.tii, _bu6.fdx, _bn1.fnm, _bxr.fdt,
_bk3.tis, _bxo.fdt, _4od.tii, _bu6.tis, _bu6.fdt, _bxo.fdx, _bxp.fdt,
_bk3.tii, _aku.prx, _bxp.fdx, _4od.tis, _bxq.frq, _bxm.tis, _bug.fdx,
_bvn.fdx, _bxp_2.del, _9pt.tii, _co1.tis, _bvn.fdt, _bxs.frq,
_aku_a.del, _2se_kw.del, _bk3.nrm, _9pt.tis, _4od.prx, _co0.nrm,
_bv2.prx, _bxo.tis, _bf9.tis, _5ex_8.del, _bxq.nrm, _bug.prx, _bxu.nrm,
_bk3.fdx, _bxo.tii, _co1.tii, _bk3_3.del, _bk3.fdt, _bxq.tis, _bxu.tis,
segments_4wf, _bxr.prx, _bk3.frq, _bug.fnm, _bxn.fnm, _2se.tis,
_bxo.frq, _cnz.fnm, _bxq.tii, _bf9.tii, _bxu.tii, _4od.fnm, _2se.tii,
_bxu.frq, _bn1.nrm, _8z3.nrm, _bxv.nrm, _9pt.frq, _bxo.nrm, _8z3.tis,
_bxn.prx, _bxv.fnm, _bvn.frq, _bv2.fdt, _btv_1.del, _co0.frq, _bv2.fdx,
_bql.fdx, _co1.frq, _bv2.nrm, _bn1.frq, _bve.tis, _bf9.fnm, _bql.fdt,
_bxm.tii, _bxm.fnm, _bql.prx, _8z3.tii, _bug.fdt, _cny_1.del,
_bxq_1.del, _bug.nrm, _5zc.nrm]|#]
[#|2011-05-01T13:42:11.973+0200|INFO|glassfish3.0.1|org.apache.solr.core.SolrCore|_ThreadID=19;_ThreadName=Thread-1;|newest
commit = 1300132289521|#]
[#|2011-05-01T13:42:11.977+0200|INFO|glassfish3.0.1|org.apache.solr.search.SolrIndexSearcher|_ThreadID=19;_ThreadName=Thread-1;|Opening
Searcher@6d307854 main|#]
[#|2011-05-01T13:42:11.977+0200|INFO|glassfish3.0.1|org.apache.solr.update.UpdateHandler|_ThreadID=19;_ThreadName=Thread-1;|end_commit_flush|#]
[#|2011-05-01T13:42:11.978+0200|INFO|glassfish3.0.1|org.apache.solr.search.SolrIndexSearcher|_ThreadID=20;_ThreadName=Thread-1;|autowarming
Searcher@6d307854 main from Searcher@26bfdb6f main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=9624,cumulative_hits=8785,cumulative_hitratio=0.91,cumulative_inserts=399,cumulative_evictions=0}|#]
[#|2011-05-01T13:42:11.978+0200|INFO|glassfish3.0.1|org.apache.solr.search.SolrIndexSearcher|_ThreadID=20;_ThreadName=Thread-1;|autowarming
result for Searcher@6d307854 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=9624,cumulative_hits=8785,cumulative_hitratio=0.91,cumulative_inserts=399,cumulative_evictions=0}|#]
[#|2011-05-01T13:42:11.978+0200|INFO|glassfish3.0.1|org.apache.solr.search.SolrIndexSearcher|_ThreadID=20;_ThreadName=Thread-1;|autowarming
Searcher@6d307854 main from Searcher@26bfdb6f main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=42828,cumulative_hits=40792,cumulative_hitratio=0.95,cumulative_inserts=2036,cumulative_evictions=0}|#]
[#|2011-05-01T13:42:11.978+0200|INFO|glassfish3.0.1|org.apache.solr.search.SolrIndexSearcher|_ThreadID=20;_ThreadName=Thread-1;|autowarming
result for Searcher@6d307854 main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=42828,cumulative_hits=40792,cumulative_hitratio=0.95,cumulative_inserts=2036,cumulative_evictions=0}|#]
[#|2011-05-01T13:42:11.978+0200|INFO|glassfish3.0.1|org.apache.solr.search.SolrIndexSearcher|_ThreadID=20;_ThreadName=Thread-1;|autowarming
Searcher@6d307854 main from Searcher@26bfdb6f main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2678,cumulative_hits=552,cumulative_hitratio=0.20,cumulative_inserts=2126,cumulative_evictions=1277}|#]
[#|2011-05-01T13:42:11.978+0200|INFO|glassfish3.0.1|org.apache.solr.search.SolrIndexSearcher|_ThreadID=20;_ThreadName=Thread-1;|autowarming
result for Searcher@6d307854 main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2678,cumulative_hits=552,cumulative_hitratio=0.20,cumulative_inserts=2126,cumulative_evictions=1277}|#]
[#|2011-05-01T13:42:11.978+0200|INFO|glassfish3.0.1|org.apache.solr.search.SolrIndexSearcher|_ThreadID=20;_ThreadName=Thread-1;|autowarming
Searcher@6d307854 main from Searcher@26bfdb6f main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=53130,cumulative_hits=16743,cumulative_hitratio=0.31,cumulative_inserts=36387,cumulative_evictions=28693}|#]
[#|2011-05-01T13:42:11.978+0200|INFO|glassfish3.0.1|org.apache.solr.search.SolrIndexSearcher|_ThreadID=20;_ThreadName=Thread-1;|autowarming
result for Searcher@6d307854 main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=53130,cumulative_hits=16743,cumulative_hitratio=0.31,cumulative_inserts=36387,cumulative_evictions=28693}|#]
[#|2011-05-01T13:42:11.979+0200|INFO|glassfish3.0.1|org.apache.solr.core.SolrCore|_ThreadID=20;_ThreadName=Thread-1;|[item]
Registered new searcher Searcher@6d307854 main|#]
[#|2011-05-01T13:42:11.979+0200|INFO|glassfish3.0.1|org.apache.solr.search.SolrIndexSearcher|_ThreadID=20;_ThreadName=Thread-1;|Closing
Searcher@26bfdb6f main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=9624,cumulative_hits=8785,cumulative_hitratio=0.91,cumulative_inserts=399,cumulative_evictions=0}
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=42828,cumulative_hits=40792,cumulative_hitratio=0.95,cumulative_inserts=2036,cumulative_evictions=0}
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=2678,cumulative_hits=552,cumulative_hitratio=0.20,cumulative_inserts=2126,cumulative_evictions=1277}
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=53130,cumulative_hits=16743,cumulative_hitratio=0.31,cumulative_inserts=36387,cumulative_evictions=28693}|#]
[#|2011-05-01T13:42:11.979+0200|INFO|glassfish3.0.1|org.apache.solr.update.processor.UpdateRequestProcessor|_ThreadID=19;_ThreadName=Thread-1;|{commit=}
0 1401|#]
[#|2011-05-01T13:42:11.979+0200|INFO|glassfish3.0.1|org.apache.solr.core.SolrCore|_ThreadID=19;_ThreadName=Thread-1;|[item]
webapp=/solr path=/update
params={commit=true&waitFlush=true&waitSearcher=true&wt=javabin&version=1}
status=0 QTime=1401 |#]
[#|2011-05-01T13:42:11.982+0200|INFO|glassfish3.0.1|dd.doubled.jobservice.PipelineImpl|_ThreadID=494;_ThreadName=Thread-1;|Job
16643 successful after 1 attempt(s).|#]
> Daniel
>
> I've been able to post documents to Solr without degrading the performance
> of search. But, I did have to make some changes to the solrconfig.xml
> (ramBufferSize, mergeFactor, autoCommit, etc).
>
> What I found to be helpful was having a look at what was the causing the OS
> to grind. If your system is swapping too much to disk, you can check if
> bumping up the ram (-Xms512m -Xmx1024m) alleviates it. Even if this isn't
> the fix, you can at least isolate if it's a memory issue, or if your issue
> is related to a disk I/O issue (e.g. running optimization on every commit).
>
>
> Also, is worth having a look in your logs to see if the server is having
> complaints about memory or issues with your schema, or some other unexpected
> issue.
>
> A resource that has been helpful for me
> http://wiki.apache.org/solr/SolrPerformanceFactors
>
>
>
>
>
>
> -----Original Message-----
> From: Daniel Huss [mailto:hussdl1985-solrus...@yahoo.de]
> Sent: Sunday, 1 May 2011 5:35 AM
> To: solr-user@lucene.apache.org
> Subject: Searching performance suffers tremendously during indexing
>
> Hi everyone,
>
> our Solr-based search is unresponsive while documents are being indexed.
> The documents to index (results of a DB query) are sent to Solr by a
> daemon in batches of varying size. The number of documents per batch may
> vary between one and several hundreds of thousands.
>
> Before investigating any further, I would like to ask if this can be
> considered an issue at all. I was expecting Solr to handle concurrent
> indexing/searching quite well, in fact this was one of the main reasons
> for chosing Solr over the searching capabilities of our RDMS.
>
> Is searching performance *supposed* to drop while documents are being
> indexed?
>
