Midas, I’d like further clarification as well. Are you sending commits along with each document that you’re POSTing to Solr? If so, you’re essentially either opening a new searcher or flushing to disk with each POST which could explain latency between each request.
Thanks, Esther > On Aug 11, 2016, at 12:19 PM, Erick Erickson <erickerick...@gmail.com> wrote: > > bq: we post json documents through the curl it takes the time (same time i > would like to say that we are not hard committing ). that curl takes time > i.e. 1.3 sec. > > OK, I'm really confused. _what_ is taking 1.3 seconds? When you said > commit, I was thinking of Solr's commit operation, which is totally distinct > from just adding a doc to the index. But I read the above statement > as you're saying it takes 1.3 seconds just to send a doc to Solr. > > Let's see the exact curl command you're using please? > > Best, > Erick > > > On Thu, Aug 11, 2016 at 5:32 AM, Emir Arnautovic > <emir.arnauto...@sematext.com> wrote: >> Hi Midas, >> >> 1. How many indexing threads? >> 2. Do you batch documents and what is your batch size? >> 3. How frequently do you commit? >> >> I would recommend: >> 1. Move commits to Solr (set auto soft commit to max allowed time) >> 2. Use batches (bulks) >> 3. tune bulk size and number of threads to achieve max performance. >> >> Thanks, >> Emir >> >> >> >> On 11.08.2016 08:21, Midas A wrote: >>> >>> Emir, >>> >>> other queries: >>> >>> a) Solr cloud : NO >>> b) <filterCache class="solr.FastLRUCache" >>> size="5000" initialSize="5000" autowarmCount="10"/> >>> c) <queryResultCache class="solr.LRUCache" >>> size="1000" initialSize="1000" autowarmCount="10"/> >>> d) <documentCache class="solr.LRUCache" >>> size="1000" initialSize="1000" autowarmCount="10"/> >>> e) we are using multi threaded system. >>> >>> On Thu, Aug 11, 2016 at 11:48 AM, Midas A <test.mi...@gmail.com> wrote: >>> >>>> Emir, >>>> >>>> we post json documents through the curl it takes the time (same time i >>>> would like to say that we are not hard committing ). that curl takes time >>>> i.e. 1.3 sec. >>>> >>>> On Wed, Aug 10, 2016 at 2:29 PM, Emir Arnautovic < >>>> emir.arnauto...@sematext.com> wrote: >>>> >>>>> Hi Midas, >>>>> >>>>> According to your autocommit configuration and your worry about commit >>>>> time I assume that you are doing explicit commits from client code and >>>>> that >>>>> 1.3s is client observed commit time. If that is the case, than it might >>>>> be >>>>> opening searcher that is taking time. >>>>> >>>>> How do you index data - single threaded or multithreaded? How frequently >>>>> do you commit from client? Can you let Solr do soft commits instead of >>>>> explicitly committing? Do you have warmup queries? Is this SolrCloud? >>>>> What >>>>> is number of servers (what spec), shards, docs? >>>>> >>>>> In any case monitoring can give you more info about server/Solr behavior >>>>> and help you diagnose issues more easily/precisely. One such monitoring >>>>> tool is our SPM <http://sematext.com/spm>. >>>>> >>>>> Regards, >>>>> Emir >>>>> >>>>> -- >>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management >>>>> Solr & Elasticsearch Support * http://sematext.com/ >>>>> >>>>> On 10.08.2016 05:20, Midas A wrote: >>>>> >>>>>> Thanks for replying >>>>>> >>>>>> index size:9GB >>>>>> 2000 docs/sec. >>>>>> >>>>>> Actually earlier it was taking less but suddenly it has increased . >>>>>> >>>>>> Currently we do not have any monitoring tool. >>>>>> >>>>>> On Tue, Aug 9, 2016 at 7:00 PM, Emir Arnautovic < >>>>>> emir.arnauto...@sematext.com> wrote: >>>>>> >>>>>> Hi Midas, >>>>>>> >>>>>>> Can you give us more details on your index: size, number of new docs >>>>>>> between commits. Why do you think 1.3s for commit is to much and why >>>>>>> do >>>>>>> you >>>>>>> need it to take less? Did you do any system/Solr monitoring? >>>>>>> >>>>>>> Emir >>>>>>> >>>>>>> >>>>>>> On 09.08.2016 14:10, Midas A wrote: >>>>>>> >>>>>>> please reply it is urgent. >>>>>>>> >>>>>>>> On Tue, Aug 9, 2016 at 11:17 AM, Midas A <test.mi...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi , >>>>>>>> >>>>>>>>> commit is taking more than 1300 ms . what should i check on server. >>>>>>>>> >>>>>>>>> below is my configuration . >>>>>>>>> >>>>>>>>> <autoCommit> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> < >>>>>>>>> openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit> >>>>>>>>> <maxTime> >>>>>>>>> ${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>> >>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management >>>>>>> Solr & Elasticsearch Support * http://sematext.com/ >>>>>>> >>>>>>> >>>>>>> >> >> -- >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management >> Solr & Elasticsearch Support * http://sematext.com/ >>
