On 11/2/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:
On 11/2/06, Mike Klaas <[EMAIL PROTECTED]> wrote:
> The one thing I'm worried about is closing the writer while documents
> are being added to it. IndexWriter is nominally thread-safe, but I'm
> not sure what happens to documents that are being added at the time.
> Looking at IndexWriter.java, it seems like if addDocument() is entered
> but hasn't reached the synchronized block, then close() is called, the
> document could be lost or an exception raised.
This seems harder to address in "user code" and still maintain parallelism.
Perhaps a Lucene patch would be more appropriate?
Perhaps IndexWriter should have a close flag, and addDocument should
return a boolean indicating if the document was added or not. Then we
could move addDocument() outside the sync block, and put a big do
while(!addDocument()) loop around the whole thing.
There is still another case to consider: if a commit happens between
adding the id to the pset and adding the document to the index, and
the add succeeds, the id will no longer be in the pset so we will end
up with a duplicate after the next commit.
I think that I've come up with a new locking strategy that circumvents
all these issues... stay tuned.
> I was going to try to put in some basic autoCommit logic while I was
> mucking about here. One question: did you intend for maxCommitTime to
> trigger deterministically (regardless of any events occurring or not)?
I hadn't thought through the whole thing, but it seems like it should
only trigger if it would make a difference.
Right--I was more concerned with whether it would occur by itself, or
if this was a condition that would trigger if true when checked.
> I had in mind checking these constraints only when documents are
> added, but this could result in maxCommitTime elapsing without a
> commit.
If there is nothing to commit, that should be fine.
I think the type of guarantee we should make is that if you add a
document, it will be committed within a certain period of time
(leaving out variances for autowarming time, etc).
That's the condition I was wondering about. I may leave tha tout of
the patch for the time being.
-Mike
