We are hitting a weird HDFS issue after a good number of Hadoop nodes simultaneously crashed.
The problem is that after all the down servers came back, the HDFS complained there was 1 missing block. But the file that that block belongs to has already been deleted after the crash. So it's an orphan block. Because the block doesn't belong to any file, there is no way to delete that block. FSCK also failed with a cause:java.io.IOException: Premature EOF exception. A side effect now is that HDFS won't free up any space even after Trash bin was emptied. The space utilization just constantly went up. Any suggestion on how to resolve this issue is highly appreciated! Thanks, -Songting
