On 3/14/2012 2:54 PM, Mike Austin wrote:
The odd thing is that if I optimize the index it doubles in size.. If I
then, add one more document to the index it goes back down to half size?
Is there a way to force this without needing to wait until another document
is added? Or do you have more information on what you think is going on?
I'm using a trunk version of solr4 from 9/12/2011 with a master with two
slaves setup. Everything besides this is working great!
The not-very-helpful-but-true answer: Don't run on Windows. I checked
your prior messages to the list to verify that this is your
environment. If you can control index updates so they don't happen at
the same time as your optimizes, you can also get around this problem by
doing the optimize twice. You would have to be absolutely sure that no
changes are made to the index between the two optimizes, so the second
one basically doesn't do anything except take care of the deletes.
Nuts and bolts of why this happens: Solr keeps the old files open so the
existing reader can continue to serve queries. That reader will not be
closed until the last query completes, which may not happen until well
after the time the new reader is completely online and ready. I assume
that the delete attempt occurs as soon as the new index segments are
completely online, before the old reader begins to close. I've not read
the source code to find out.
On Linux and other UNIX-like environments, you can delete files while
they are open by a process. They continue to exist as in-memory links
and take up space until those processes close them, at which point they
are truly gone. On Windows, an attempt to delete an open file will
fail, even if it's open read-only.
There are probably a number of ways that this problem could be solved
for Windows platforms. The simplest that I can think of, assuming it's
even possible, would be to wait until the old reader is closed before
attempting the segment deletion. That may not be possible - the
information may not be available to the portion of code that does the
deletion. There are a few things standing in the way of me fixing this
problem myself: 1) I'm a beginning Java programmer. 2) I'm not familiar
with the Solr code at all. 3) My interest level is low because I run on
Linux, not Windows.
Thanks,
Shawn
