Re: Preventing index segment corruption when windows crashes

2010-12-02 Thread Michael McCandless
On Thu, Dec 2, 2010 at 4:53 AM, Peter Sturge wrote: > As I'm not familiar with the syncing in Lucene, I couldn't say whether > there's a specific problem with regards Win7/2008 server etc. > > Windows has long had the somewhat odd behaviour of deliberately > caching file handles after an explicit

Re: Preventing index segment corruption when windows crashes

2010-12-02 Thread Peter Sturge
As I'm not familiar with the syncing in Lucene, I couldn't say whether there's a specific problem with regards Win7/2008 server etc. Windows has long had the somewhat odd behaviour of deliberately caching file handles after an explicit close(). This has been part of NTFS since NT 4 days, but there

Re: Preventing index segment corruption when windows crashes

2010-12-02 Thread Michael McCandless
On Thu, Dec 2, 2010 at 4:10 AM, Peter Sturge wrote: > The Win7 crashes aren't from disk drivers - they come from, in this > case, a Broadcom wireless adapter driver. > The corruption comes as a result of the 'hard stop' of Windows. > > I would imagine this same problem could/would occur on any OS

Re: Preventing index segment corruption when windows crashes

2010-12-02 Thread Peter Sturge
The Win7 crashes aren't from disk drivers - they come from, in this case, a Broadcom wireless adapter driver. The corruption comes as a result of the 'hard stop' of Windows. I would imagine this same problem could/would occur on any OS if the plug was pulled from the machine. Thanks, Peter On T

Re: Preventing index segment corruption when windows crashes

2010-12-01 Thread Lance Norskog
Is there any way that Windows 7 and disk drivers are not honoring the fsync() calls? That would cause files and/or blocks to get saved out of order. On Tue, Nov 30, 2010 at 3:24 PM, Peter Sturge wrote: > After a recent Windows 7 crash (:-\), upon restart, Solr starts giving > LockObtainFailedExce

Re: Preventing index segment corruption when windows crashes

2010-11-30 Thread Peter Sturge
After a recent Windows 7 crash (:-\), upon restart, Solr starts giving LockObtainFailedException errors: (excerpt) 30-Nov-2010 23:10:51 org.apache.solr.common.SolrException log SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: nativefsl...@solr\.\.\data0\index

Re: Preventing index segment corruption when windows crashes

2010-11-30 Thread Peter Sturge
The index itself isn't corrupt - just one of the segment files. This means you can read the index (less the offending segment(s)), but once this happens it's no longer possible to access the documents that were in that segment (they're gone forever), nor write/commit to the index (depending on the

Re: Preventing index segment corruption when windows crashes

2010-11-29 Thread Yonik Seeley
On Mon, Nov 29, 2010 at 10:46 AM, Peter Sturge wrote: > If a Solr index is running at the time of a system halt, this can > often corrupt a segments file, requiring the index to be -fix'ed by > rewriting the offending file. Really? That shouldn't be possible (if you mean the index is truly corru

Preventing index segment corruption when windows crashes

2010-11-29 Thread Peter Sturge
Hi, With the advent of new windows versions, there are increasing instances of system blue-screens, crashes, freezes and ad-hoc failures. If a Solr index is running at the time of a system halt, this can often corrupt a segments file, requiring the index to be -fix'ed by rewriting the offending fi