On 7/3/2018 6:55 AM, Joe Obernberger wrote: > I think the root issue is related to some weirdness with HDFS. Log > file is here: > http://lovehorsepower.com/solr.log.4 > Config is here: > http://lovehorsepower.com/solrconfig.xml > I don't see anything set to 20 seconds. > > I believe the root exception is: > > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /solr7.1.0/UNCLASS_30DAYS/core_node-1684300827/data/tlog/tlog.0000000000000008930 > could only be replicated to 0 nodes instead of minReplication (=1). > There are 41 datanode(s) running and no node(s) are excluded in this > operation.
That does look like what's causing all the errors. This is a purely hadoop/hdfs exception. There are no Solr classes in the "Caused by" part of the exception. If you have any hdfs experts in-house, you should talk to them. If not, you may need to find a hadoop mailing list. Looking up the exception, I've seen a couple of answers that say when this happens you have to format your datanode and lose all your data. Or it could be a configuration problem, a permission problem, or a disk space problem. Perhaps if I knew anything about HDFS, I could make sense of the google search results. The logs on your hadoop servers might have more information, but I do not know how to interpret them. Thanks, Shawn
