Hmm OK that seems like a possible explanation then. Still it's spooky
that it's taking 5 minutes. How many files are in the index at the
time you call commit?
I wonder if you were to simply pause for say 30 seconds, before
issuing the commit, whether you'd then see the commit go faster? On
Windows at least such a silly trick does seem to improve performance,
I think because it allows the OS to move the bytes from its write
cache onto stable storage "on its own schedule" whereas when we commit
we are demanding the OS move the bytes on our [arbitrary] schedule.
I really wish OSs would add an API that would just block & return once
the file has made it to stable storage (letting the OS sync on its own
optimal schedule), rather than demanding the file be fsync'd
immediately.
I really haven't explored the performance of fsync on different
filesystems. I think I've read that ReiserFS may have issues, though
it could have been addressed by now. I *believe* ext3 is OK (at
least, it didn't show the strange "sleep to get better performance"
issue above, in my limited testing).
Mike
Uwe Klosa wrote:
Thanks Mike
The use of fsync() might be the answer to my problem, because I have
installed Solr for lack of other possibilities in a zone on Solaris
with ZFS
which slows down when many fsync() calls are made. This will be
fixed in a
upcoming release of Solaris, but I will move as soon as possible the
Solr
instances to another server with a different file system. Would the
use of a
different file system than ext3 boost the performance?
Uwe
On Fri, Oct 3, 2008 at 8:28 PM, Michael McCandless <
[EMAIL PROTECTED]> wrote:
Yonik Seeley wrote:
On Fri, Oct 3, 2008 at 1:56 PM, Uwe Klosa <[EMAIL PROTECTED]>
wrote:
I have a big problem with one of my solr instances. A commit can
take up
to
5 minutes. This time does not depend on the number of documents
which are
updated. The difference for 1 or 100 updated documents is only a
few
seconds.
Since Solr's commit logic really hasn't changed, I wonder if this
could be lucene related somehow.
Lucene's commit logic has changed: we now fsync() each file in the
index to
ensure all bytes are on stable storage, before returning.
But I can't imagine that taking 5 minutes, unless there are somehow
a great
many files added to the index?
Uwe, what filesystem are you using?
Yonik, when Solr commits what does it actually do?
Mike
