The root cause was the aggressive logging filling up the file system. Our admins have the logs on the same file system with the data, so when the filesystem got full it couldn't write to the transaction logs which corrupted them
Thank you for the tips on recovery, I will forward them to our admins Web On Fri, Jan 6, 2017 at 12:43 PM, Shawn Heisey <apa...@elyograg.org> wrote: > On 1/6/2017 10:09 AM, Webster Homer wrote: > > This happened while testing and was not in a production system. So we > > just deleted both collections and recreated them after fixing the root > > cause. If this had been a production system that would not have been > > acceptable. What is the best way to recover from a problem like this? > > Stop cdcr and delete the tlog files? > > What was the root cause? Need to know that before anyone can tell you > whether or not you've run into a bug. > > If it was the problem you've separately described where CDCR logging > filled up your disk ... handling that gracefully in a program is very > difficult. It's possible, but there's very little incentive for anyone > to attempt it. Lucene and Solr have a general requirement of plenty of > free disk space (enough for the index to triple in size temporarily) > just for normal operation, so coding for disk space exhaustion isn't > likely to happen. Server monitoring should send an alarm when disk > space gets low so you can fix it before it causes real problems. > > Thanks, > Shawn > > -- This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, you must not copy this message or attachment or disclose the contents to any other person. If you have received this transmission in error, please notify the sender immediately and delete the message and any attachment from your system. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept liability for any omissions or errors in this message which may arise as a result of E-Mail-transmission or for damages resulting from any unauthorized changes of the content of this message and any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not guarantee that this message is free of viruses and does not accept liability for any damages caused by any virus transmitted therewith. Click http://www.emdgroup.com/disclaimer to access the German, French, Spanish and Portuguese versions of this disclaimer.
