400kb is size of single document and i am sending 100 documents per request. solr heap size is 16gb and running on multithread.
On Tue, Aug 16, 2016 at 5:10 PM, Emir Arnautovic < emir.arnauto...@sematext.com> wrote: > Hi, > > 400KB/doc * 100doc = 40MB. If you are running it single threaded, Solr > will be idle while accepting relatively large request. Or is 400KB 100 doc > bulk that you are sending? > > What is Solr's heap size? I would try increasing number of threads and > monitor Solr's heap/CPU/IO to see where is the bottleneck. > > How complex is fields' analysis? > > Regards, > Emir > > > On 16.08.2016 13:25, kshitij tyagi wrote: > >> hi, >> >> we are sending about 100 documents per request for indexing? we have >> autocmmit set to false and commit only when 10000 documents are >> present.solr and the machine sending request are in same pool. >> >> >> >> On Tue, Aug 16, 2016 at 4:51 PM, Emir Arnautovic < >> emir.arnauto...@sematext.com> wrote: >> >> Hi, >>> >>> Do you send one doc per request? How frequently do you commit? Where is >>> Solr running? What is network connection between your machine and Solr? >>> What are JVM settings? Is 10-30s for entire indexing or single doc? >>> >>> Regards, >>> Emir >>> >>> >>> On 16.08.2016 11:34, kshitij tyagi wrote: >>> >>> Hi alexandre, >>>> >>>> 1 document of 400kb size is taking approx 10-30 sec and this is >>>> varying. I >>>> am posting document using curl >>>> >>>> On Tue, Aug 16, 2016 at 2:11 PM, Alexandre Rafalovitch < >>>> arafa...@gmail.com> >>>> wrote: >>>> >>>> How many records is that and what is 'slow'? Also is this standalone or >>>> >>>>> cluster setup? >>>>> >>>>> On 16 Aug 2016 6:33 PM, "kshitij tyagi" <kshitij.shopcl...@gmail.com> >>>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>>> I am indexing a lot of data about 8GB, but it is taking a lot of >>>>>> time. I >>>>>> have read about maxBufferedDocs, ramBufferSizeMB, merge policy ,etc in >>>>>> solrconfig file. >>>>>> >>>>>> It would be helpful if someone could help me out tune the segtting for >>>>>> faster indexing speeds. >>>>>> >>>>>> *I have read the docs but not able to get what exactly means changing >>>>>> >>>>>> these >>>>> >>>>> configs.* >>>>>> >>>>>> >>>>>> *Regards,* >>>>>> *Kshitij* >>>>>> >>>>>> >>>>>> -- >>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management >>> Solr & Elasticsearch Support * http://sematext.com/ >>> >>> >>> > -- > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > Solr & Elasticsearch Support * http://sematext.com/ > >
