Hello Alexandre - if you, and others, allow me to be a bit lazy right now; are there unit tests that input corrupted segments, where not the structure but the data is affected, to the codec?
Thanks, Markus -----Original message----- > From:Alexandre Rafalovitch <arafa...@gmail.com> > Sent: Thursday 12th March 2015 23:52 > To: solr-user <solr-user@lucene.apache.org> > Subject: Re: SSD endurance > > Lucene 5 has added a lot of various CRCs to catch index corruption > situations. I don't know if it is 'perfect', but there was certainly a > lot of work. > > Regards, > Alex. > ---- > Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: > http://www.solr-start.com/ > > > On 12 March 2015 at 18:39, Markus Jelsma <markus.jel...@openindex.io> wrote: > > Thanks for sharing Toke! > > > > Reliability should not be a problem for a Solr cloud environment. A > > corrupted index cannot be loaded due to exceptions so the core should not > > enter an active state. However, what would happen if parts of the data > > become corrupted but can still be processed by the codec? I don't even know > > if the data has a CRC check to guard against such madness? > > > > Markus > > > > -----Original message----- > >> From:Toke Eskildsen <t...@statsbiblioteket.dk> > >> Sent: Thursday 12th March 2015 21:33 > >> To: solr-user <solr-user@lucene.apache.org> > >> Subject: SSD endurance > >> > >> For those who have not yet taken the leap to SSD goodness because they are > >> afraid of flash wear, the burnout test from The Tech Report seems worth a > >> read. The short story is that they wrote data to the drives until they > >> wore out. All tested drives survived considerably longer than guaranteed, > >> but 4/6 failed catastrophically when they did die. > >> > >> http://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead > >> > >> I am disappointed about the catastrophic failures. One of the promises of > >> SSDs was graceful end of life by switching to read-only mode. Some of them > >> did give warnings before the end, but I wonder how those are communicated > >> in a server environment? > >> > >> > >> Regarding Lucene/Solr, the write pattern when updating an index is benign > >> to SSDs: Updates are relatively bulky, rather than the evil > >> constantly-flip-random-single-bits-and-flush pattern of databases. With > >> segments being immutable, the bird's eye view is that Lucene creates and > >> deletes large files, which makes it possible for the SSD's wear-leveler to > >> select the least-used flash sectors for new writes: The write pattern over > >> time is not too far from the one that The Tech Report tested with. > >> > >> - Toke Eskildsen > >> Whose trusty old 160GB Intel X25-M reports an accumulated 36TB of writes. > >> >
