On 12/5/2017 10:47 AM, Sarah Weissman wrote:
I’ve recently been using the dataimport handler to import records from a
database into a Solr cloud collection with multiple shards. I have 6 dataimport
handlers configured on 6 different paths all running simultaneously against the
same DB. I’ve noticed that when I do this I often get “idle” status from the
DIH even when the import is still running. The percentage of the time I get an
“idle” response seems proportional to the number of shards. I.e., with 1 shard
it always shows me non-idle status, with 2 shards I see idle about half the
time I check the status, with 96 shards it seems to be showing idle almost all
the time. I can see the size of each shard increasing, so I’m sure the import
is still going.
I recently switched from 6.1 to 7.1 and I don’t remember this happening in 6.1.
Does anyone know why the DIH would report idle when it’s running?
e.g.:
curl http://myserver:8983/solr/collection/dataimport6
When you send a DIH request to the collection name, SolrCloud is going
to load balance that request across the cloud, just like it would with
any other request. Solr will look at the list of all responding nodes
that host part of the collection and send multiple such requests to
different cores (shards/replicas) across the cloud. If there are four
cores in the collection and the nodes hosting them are all working, then
each of those cores would only see requests to /dataimport about one
fourth of the time.
DIH imports happen at the core level, NOT the collection level, so when
you start an import on a collection with four cores in the cloud, only
one of those four cores is actually going to be doing the import, the
rest of them are idle.
This behavior should happen with any version, so I would expect it in
6.1 as well as 7.1.
To use DIH with SolrCloud, you should be sending your request directly
to a shard replica core, not the collection, so that you can be
absolutely certain that the import command and the status command are
going to the same place. You MIGHT need to also have a distrib=false
parameter on the request, but I do not know whether that is required to
prevent the load balancing on the dataimport handler.
A similar question came to this list two days ago, and I replied to that
one yesterday.
http://lucene.472066.n3.nabble.com/Dataimporter-status-tp4365602p4365879.html
Somebody did open an issue a LONG time ago about this problem:
https://issues.apache.org/jira/browse/SOLR-3666
I just commented on the issue.
Thanks,
Shawn
