On 3/23/2018 10:32 PM, Susheel Kumar wrote:
I did schema update to Solr cloud of Source CDCR cluster and same on
target. After Collection Reload, noticed "error opening searcher" /
IndexWriter closed etc. on leader node while all replica's went into
recovery mode.
Later after restarting Solr on Leader noticed below too many file open
errors. The ulimit shows unlimited.
What could have gone wrong above/and how can we avoid. This is with 6.6.2
and there were ingestions/delete and searches happening the time I reload
the collection.
Caused by: org.apache.solr.common.SolrException:
java.io.FileNotFoundException:
/app/solr/data/COLL_shard1_replica4/data/tlog/tlog.0000000000000002214.1591528549576081408
(Too many open files)
A 'ulimit' command that you typed may have shown unlimited, but the Solr
process apparently DOES have a limit.
If you are trying to change limits with the ulimit command for Solr, I
have frequently seen this NOT work. On Linux, /etc/security/limits.conf
appears to be the only way to successfully increase the open file
limit. The default limit on Linux is normally 1024, which is pretty low.
I would recommend not doing reloads when there is a lot of indexing
activity. When indexing is heavy, it can lead to recovery operations,
if the number of documents indexed during the reload exceeds the limit
for syncing from transaction logs. This limit is normally 100 docs.
Thanks,
Shawn
