That is quite big document! You need to minitor Solr to see if you are
feeding documents fast enough or if you are saturating it with large
number of large requests. Play with batch size and number of threads to
find sweet spot. Maybe try extremes first (one doc/one thread, one doc
many threads etc.) and it might tell you more what is slowing things
down. If you are not using any Solr/JVM/OS monitoring tool, it will help
you a lot to diagnose issue. One such tool is our SPM
(http://sematext.com/spm).
Regards,
Emir
On 16.08.2016 14:49, kshitij tyagi wrote:
400kb is size of single document and i am sending 100 documents per request.
solr heap size is 16gb and running on multithread.
On Tue, Aug 16, 2016 at 5:10 PM, Emir Arnautovic <
emir.arnauto...@sematext.com> wrote:
Hi,
400KB/doc * 100doc = 40MB. If you are running it single threaded, Solr
will be idle while accepting relatively large request. Or is 400KB 100 doc
bulk that you are sending?
What is Solr's heap size? I would try increasing number of threads and
monitor Solr's heap/CPU/IO to see where is the bottleneck.
How complex is fields' analysis?
Regards,
Emir
On 16.08.2016 13:25, kshitij tyagi wrote:
hi,
we are sending about 100 documents per request for indexing? we have
autocmmit set to false and commit only when 10000 documents are
present.solr and the machine sending request are in same pool.
On Tue, Aug 16, 2016 at 4:51 PM, Emir Arnautovic <
emir.arnauto...@sematext.com> wrote:
Hi,
Do you send one doc per request? How frequently do you commit? Where is
Solr running? What is network connection between your machine and Solr?
What are JVM settings? Is 10-30s for entire indexing or single doc?
Regards,
Emir
On 16.08.2016 11:34, kshitij tyagi wrote:
Hi alexandre,
1 document of 400kb size is taking approx 10-30 sec and this is
varying. I
am posting document using curl
On Tue, Aug 16, 2016 at 2:11 PM, Alexandre Rafalovitch <
arafa...@gmail.com>
wrote:
How many records is that and what is 'slow'? Also is this standalone or
cluster setup?
On 16 Aug 2016 6:33 PM, "kshitij tyagi" <kshitij.shopcl...@gmail.com>
wrote:
Hi,
I am indexing a lot of data about 8GB, but it is taking a lot of
time. I
have read about maxBufferedDocs, ramBufferSizeMB, merge policy ,etc in
solrconfig file.
It would be helpful if someone could help me out tune the segtting for
faster indexing speeds.
*I have read the docs but not able to get what exactly means changing
these
configs.*
*Regards,*
*Kshitij*
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
