Hi,
With auto-committing disabled we can now index many millions of
documents in our test environment on a 5-node cluster with 5 shards and
a replication factor of 2. The documents are uploaded from map/reduce.
No significant changes were made to solrconfig and there are no update
processors enabled. We are using a trunk revision from this weekend.
The indexing speed is well below what we are used to see, we can easily
index 5 millions documents on a non-cloud enabled Solr 3.x instance
within an hour. What could be going on? There aren't many open TCP
connections and the number of file descriptors is stable and I/O is low
but CPU-time is high! Each node has two Solr cores both writing to their
dedicated disk.
The indexing speed is stable, it was slow at start and still is. It's
now running for well over 6 hours and only 3.5 millions documents are
indexed. Another strange detail is that the node receiving all incoming
documents (we're not yet using a client side Solr server pool) has a
much larger disk usage than all other nodes. This is peculiar as we
expected all replica's to be a about the same size.
The receiving node has slightly higher CPU than the other nodes but the
thread dump shows a very large amount of threads of type
cmdDistribExecutor-8-thread-292260 (295090) with 0-100ms CPU-time. At
the top of the list these threads all have < 20ms time but near the
bottom it rises to just over 100ms. All nodes have a couple of
http-80-30 (121994) threads with very high CPU-time each.
Is this a known issue? Did i miss something? Any ideas?
Thanks
