I trying to migrate 100M documents from a solr index (v3.6) to a solrcloud index (v4.1, 4 shards) by using SolrEntityProcessor. My data-config.xml is like
<dataConfig> <document> <entity name="sep" processor="SolrEntityProcessor" url="http://10.64.35.117:8995/solr/"; query="*:*" rows="2000" fl= "author_class,authorlink,author_location_text,author_text,author,category,date,dimension,entity,id,language,md5_text,op_dimension,opinion_text,query_id,search_source,sentiment,source_domain_text,source_domain,text,textshingle,title,topic,topic_text,url" /> </document> </dataConfig> Initially, the data import rate is about 1K docs/second, but it eventually decrease to 20docs/second after running for tens of hours. Last time I tried data import with solorentityprocessor, the transfer rate can be as high as 3K docs/seconds. Anyone has any clues what can cause the slowdown? Thanks, Ming-
