It would be good to be able to commit without opening a new reader however with Lucene 2.9 the segment readers for all available segments are already created and available via getReader which manages the reference counting internally.
Using reopen redundantly creates SRs that are already held internally in IW. On Wed, Oct 7, 2009 at 9:59 AM, Yonik Seeley <yo...@lucidimagination.com> wrote: > On Wed, Oct 7, 2009 at 12:51 PM, Phillip Farber <pfar...@umich.edu> wrote: >> >> In a separate thread, I've detailed how an optimize is taking > 2x disk >> space. We don't use solr distribution/snapshooter. We are using the default >> deletion policy = 1. We can't optimize a 192G index in 400GB of space. >> >> This thread in lucene/java-user >> >> http://www.gossamer-threads.com/lists/lucene/java-user/43475 >> >> suggests that an optimize should not take > 2x unless perhaps an IndexReader >> is holding on to segments. This could be our problem since when optimization >> runs out of space, if we stop tomcat, a number of files go away and space is >> recovered. >> >> But we are not searching the index so how could a Searcher/IndexReader have >> any segments open? >> >> I notice in the logs that as part of routine commits or as part of optimize >> a Searcher is registered and autowarmed from a previous searcher (of course >> there's nothing in the caches -- this is just a build machine). >> >> INFO: registering core: >> Oct 6, 2009 2:16:20 PM org.apache.solr.core.SolrCore registerSearcher >> INFO: [] Registered new searcher searc...@2e097617 main >> >> Does this means that there's always a lucene IndexReader holding segment >> files open so they can't be deleted during an optimize so we run out of disk >> space > 2x? > > Yes. > A feature could probably now be developed now that avoids opening a > reader until it's requested. > That wasn't really possible in the past - due to many issues such as > Lucene autocommit. > > -Yonik > http://www.lucidimagination.com >
