Hi
I build my SolrCloud using Solr 4.6.0 (java version:1.7.0_45). In my
cloud,I have a collection with 30 shard,and each shard has one replica.
each core of the shard contains nearly 50 million docs that is 15GB in
size,so does the replica.
Before applying my cloud in the real world,I do a performance test with
JMeter 2.11.
The scenario of the my test is simple:100 threads sending requests for 20
seconds ,and these requests are only sent to a specific core of a specific
shard.the request is similar to the following :
http://IP:port/solr/tv_201407/select?q=*:*&fq=BEGINTIME:["2014-06-01
00:00:00"+TO+*]+AND+(CONTACT:${user})+AND (TV_STATE:"00")&shards=tv_201407
&rows=2000&sort=BEGINTIME+desc.
I encountered the drastic query latency during performance testing and
DIH import delay after performance testing.Please help me. I have tested
several times and get the same problem and can not handle it by myself.Any
suggestion will be apprecaited.
The following steps describes what I have done .
Step 1: Before the test,the DIH import job is very fast.As the statistics
[1], the DIH importing takes only 1s for 10 docs.
[1]-----------------------------------
Indexing completed. Added/Updated: 10 documents. Deleted 0 documents.
(Duration: 01s)
Requests: 1 (1/s), Fetched: 10 (10/s), Skipped: 0, Processed: 10 (10/s)
Started: less than a minute ago
---------------------------------------
Step 2: Then ,Doing the test under the caches are cleaned. The summery
statistics data is as [2]. Although I have clean the caches,I never think
the query latency becomes so drastic that it cannot be acceptable in my
real application.
The red font describes the latency of the query performance test on the
core tv_201407 of the shard tv_201407 .
So would you experts can give some hints about the drastic query latency
?
[2]-----------------------------------
[solr@solr2 test]$ ../bin/jmeter.sh -n -t solrCoudKala20140401.jmx -l
logfile_solrCloud_20.jtl
Creating summariser <aggregate>
Created the tree successfully using solrCoudKala20140401.jmx
Starting the test @ Wed Jul 16 15:59:28 CST 2014 (1405497568104)
Waiting for possible shutdown message on port 4445
aggregate + 1 in 8.1s = 0.1/s Avg: 8070 Min: 8070 Max: 8070 Err:
0 (0.00%) Active: 100 Started: 100 Finished: 0
aggregate + 103 in 13.4s = 7.7/s Avg: 8027 Min: 4191 Max: 8434 Err:
0 (0.00%) Active: 97 Started: 100 Finished: 3
aggregate = 104 in 13.4s = 7.7/s Avg: 8027 Min: 4191 Max: 8434 Err:
0 (0.00%)
aggregate + 96 in 7s = 14.5/s Avg: 6160 Min: 5295 Max: 6625 Err:
0 (0.00%) Active: 0 Started: 100 Finished: 100
aggregate = 200 in 15s = 13.6/s Avg: 7131 Min: 4191 Max: 8434 Err:
0 (0.00%)
Tidying up ... @ Wed Jul 16 15:59:43 CST 2014 (1405497583461)
... end of run
[solr@solr2 test]$
---------------------------------------
Step 3:To be continued,after the test,I do the DIH importing job again
using the same import expresion.However the performance of the DIH becomes
so unacceptable.
to import the 10 docs takes 2 m 15 s [3]!
Having noticing that ,solr can fetched the 10 docs fast,the processing is
slow.
[3]-----------------------------------
*Indexing completed. Added/Updated: 10 documents. Deleted 0 documents.
(Duration: 2m 15s)*
Requests: 1 (0/s), Fetched: 10 (0/s), Skipped: 0, Processed: 10 (0/s)
Started: about an hour ago
---------------------------------------
By the way. jvm gc goes normal,and there is no long full gc during the
test. the load of my system(rhel 6.5) are also normal.
Regards
