bq: solr.autoCommit.maxTime:600000
<maxDocs>100000</maxDocs>
<openSearcher>true</openSearcher>
Every 100K documents or 10 minutes (whichever comes first) your
current searchers will be closed and a new searcher opened, all the
warmup queries etc. might happen. I suspect you're not doing much with
autwarming and/or newSearcher queries. So occasionally your search has
to wait for caches to be read, terms to be populated, etc.
Some possibilities to test this:
1> create some newSearcher queries in solrconfig.xml
2> specify a reasonable autowarm count for queryResultCache (don't go
crazy here, start with 16 or some similiar)
3> set openSearcher to false above. In this case you won't be able to
see the documents until either a hard or soft commit happens, you
could cure this with a single hard commit at the end of your indexing
run. It all depends on what latency you can tolerate in terms of
searching newly-indexed documents.
Here's a reference...
http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Best,
Erick
On Tue, Apr 8, 2014 at 12:11 PM, Joshi, Shital <[email protected]> wrote:
> We don't do any soft commit. This is our hard commit setting.
>
> <autoCommit>
> <maxTime>${solr.autoCommit.maxTime:600000}</maxTime>
> <maxDocs>100000</maxDocs>
> <openSearcher>true</openSearcher>
> </autoCommit>
>
> We use this update command:
>
> solr_command=$(cat<<EnD
> time zcat --force $file2load | /usr/bin/curl --proxy "" --silent --show-error
> --max-time 3600 \
> "http://$solr_url/solr/$solr_core/update/csv?\
> commit=false\
> &separator=|\
> &escape=\\\
> &trim=true\
> &header=false\
> &skipLines=2\
> &overwrite=true\
> &_shard_=$shardid\
> &fieldnames=$fieldnames\
> &f.cs_rep.split=true\
> &f.cs_rep.separator=%5E" --data-binary @- -H 'Content-type:text/plain;
> charset=utf-8'
> EnD)
>
>
> -----Original Message-----
> From: Erick Erickson [mailto:[email protected]]
> Sent: Tuesday, April 08, 2014 2:21 PM
> To: [email protected]
> Subject: Re: solr4 performance question
>
> What do you have for hour _softcommit_ settings in solrconfig.xml? I'm
> guessing you're using SolrJ or similar, but the solrconfig settings
> will trip a commit as well.
>
> For that matter ,what are all our commit settings in solrconfig.xml,
> both hard and soft?
>
> Best,
> Erick
>
> On Tue, Apr 8, 2014 at 10:28 AM, Joshi, Shital <[email protected]> wrote:
>> Hi,
>>
>> We have 10 node Solr Cloud (5 shards, 2 replicas) with 30 GB JVM on 60GB
>> machine and 40 GB of index.
>> We're constantly noticing that Solr queries take longer time while update
>> (with commit=false setting) is in progress. The query which usually takes .5
>> seconds, take up to 2 minutes while updates are in progress. And this is not
>> the case with all queries, it is very sporadic behavior.
>>
>> Any pointer to nail this issue would be appreciated.
>>
>> Is there a way to find how much of a query result came from cache? Can we
>> enable any log settings to start printing what came from cache vs. what was
>> queried?
>>
>> Thanks!
