Collection reload task is taking long time

[Moulay Hicham]

Hi,

We have a solr cluster of 30 nodes with a Replication Factor =3.
Each index size is about 80GB.
Solr version is 8.1
The cluster has high TPS both in read and write.

We have recently made a schema change and uploaded it using ZKCLI script.
Then we issue a collection reload async request:
admin/collections?action=RELOAD&name=<collectionname>&async=1000'

When we check on the status of this request, it shows that it's still
running:

admin/collections?action=REQUESTSTATUS&requestid=1000'

{

  "responseHeader":{

    "status":0,

    "QTime":1},

  "status":{

    "state":"running",

    "msg":"found [1000] in running tasks"}}

This task has been in a running state for *about 5 hours* so far. I am not
sure if this is expected or the status of this task failed or completed but
never reported back to zookeeper.

Also if running for that long - is it because the index is being actively
(with high TPS) updated? We have a softcommit of 10s and hadcommit of 60s.

Please help me understand what's going on.

Thanks,
Moulay