On 1/22/2016 10:29 AM, Brian Narsi wrote:
What are the various ways DataImportHandler can be scaled?
I'm not very familiar with how DIH interacts with SolrCloud. I know you can use it with SolrCloud, but nothing else. Assuming you're not running SolrCloud, the following information will apply:
The only way I know of to scale DIH performance on a single Solr core is to define multiple DIH handlers, each of which imports a subset of your data. Then you can run multiple imports simultaneously. A single DIH handler can only run one import at a time, and it will be single-threaded.
If you've got a sharded index, each shard would be in its own core, and therefore each one would have its own instance of your dataimport handler with its own config, so you can run an import on each shard simultaneously. This is what happens on my index when I need to fully rebuild it.
Thanks, Shawn
