Do you have autocommit enabled? I tested this with 1m docs indexed by using the default example config and saw used file descriptors go up to 2400 (did not come down even after the final commit at the end). Then I disabled autocommit, reindexed and the descriptor count stayed pretty much flat at around 400-500.
-- Sami Siren On Thu, Mar 1, 2012 at 12:36 PM, Markus Jelsma <markus.jel...@openindex.io> wrote: > Hi, > > Yesterday we had an issue with too many open files, which was solved because > a username was misspelled. But there is still a problem with open files. > > We cannot succesfully index a few millions documents from MapReduce to a > 5-node Solr cloud cluster. One of the problems is that after a while > ClassNotFoundErrors and other similar weirdness begin to appear. This will > not solve itself if indexing is stopped. > > With lsof i found that Solr keeps open roughly 9k files 8 hours after > indexing failed. Out of the 9k there are roughly 7.5k deleted files that > still have a file descriptor open for the tomcat6 user, these are all > segments files: > > /opt/solr/openindex_a/data/index.20120228101550/_34s.tvd > java 10049 tomcat6 DEL REG 9,0 > 515607 /opt/solr/openindex_a/data/index.20120228101550/_34s.tvx > java 10049 tomcat6 DEL REG 9,0 > 515504 /opt/solr/openindex_a/data/index.20120228101550/_34s.fdx > java 10049 tomcat6 DEL REG 9,0 > 515735 /opt/solr/openindex_a/data/index.20120228101550/_34s_nrm.cfs > java 10049 tomcat6 DEL REG 9,0 > 515595 /opt/solr/openindex_a/data/index.20120228101550/_34v_nrm.cfs > java 10049 tomcat6 DEL REG 9,0 > 515592 /opt/solr/openindex_a/data/index.20120228101550/_34v_0.tim > java 10049 tomcat6 DEL REG 9,0 > 515591 /opt/solr/openindex_a/data/index.20120228101550/_34v_0.prx > java 10049 tomcat6 DEL REG 9,0 > 515590 /opt/solr/openindex_a/data/index.20120228101550/_34v_0.frq > .... any many more > > Did i misconfigure anything? This is a pretty standard (no changes to > IndexDefaults section) and a recent Solr trunk revision. Is there a bug > somewhere? > > Thanks, > Markus
