Not sure, but I believe I just found the problem. It has to do with the way reference counts are used in the code. Once Yonik mentioned the reference count I looked through the Solr source to see how that worked. Then I looked through the custom code I've added to my Solr install and found an instance of SolrQueryRequest that I wasn't calling close() on when done. Added the call to close() and now there doesn't seem to be any stale file handles floating around. I'm still in the middle of testing this (tried on Windows XP and a Mac so far, waiting for access to a Linux machine for the real test), but if you've added any custom code to your Solr install you may want to make sure you are doing everything possible to keep Solr's reference count to SolrIndexSearcher instances up to date.
Mark On Dec 26, 2007 10:32 AM, amamare <[EMAIL PROTECTED]> wrote: > > Yes I'm using Solr 1.2. I made a typo in the subject of the thread though, > I > believe it is SolrIndexSearcher, not SolrIndexWriter, that is holding the > reference... Did you find a solution to it? Can you give me the url to > your > thread? > > > Mark Baird wrote: > > > > Just noticed this thread. The issue you are seeing looks identical to > the > > one I am seeing. I started a thread about this same issue on Monday. > Are > > you also running Solr 1.2? > > > > On Dec 20, 2007 8:20 AM, amamare <[EMAIL PROTECTED]> wrote: > > > >> > >> I have an application consisting of three web applications running on > >> JBoss > >> 1.4.2 on a Linux Redhat server. I'm using Solr/Lucene embeddedly to > >> create > >> and maintain a frequently updated index. Once updated, the index is > >> copied > >> to another directory used for searching. Old index-files in the search > >> directory are then deleted. The streams used to copy the files are > closed > >> in > >> finally-blocks. After a few days an IOException occurs because of "too > >> many > >> open files". When I run the linux command > >> > >> ls -l /proc/26788/fd/ > >> > >> where 26788 is jboss' process id, it gives me a seemingly > ever-increasing > >> list of deleted files (1 per update since I optimize on every update > and > >> use > >> compound file format), marked with 'deleted' in parantheses. They are > all > >> located in the search directory. From what I understand this means that > >> something still holds a reference to the file, and that the file will > be > >> permanently deleted once this something loses its reference to it. > >> > >> Only SolrIndexSearcher objects are in direct contact with these files > in > >> the > >> search application. The searchers are local objects in search-methods, > >> and > >> are closed after every search operation. In theory, the garbage > collector > >> should collect these objects later (though while profiling other > >> applications I've noticed that it often doesn't garbage collect until > the > >> allocated memory starts running out). > >> > >> The other objects in contact with the files are the FileOutputStreams > >> used > >> to copy them, but as stated above, these are closed in finally-blocks > and > >> thus should hold no reference to the files. > >> > >> I need to get rid of the "too many open files"-problem. I suspect that > it > >> is > >> related to the almost-deleted files in the proc-dir, but I know too > >> little > >> of Linux to be sure. Does the problem ring a bell to anyone, or do you > >> have > >> any ideas as to how I can get rid of the problem? > >> > >> All help is greatly appreciated. > >> -- > >> View this message in context: > >> > http://www.nabble.com/SolrIndexWriter-holding-reference-to-deleted-file--tp14436326p14436326.html > >> Sent from the Solr - User mailing list archive at Nabble.com. > >> > >> > > > > > > -- > View this message in context: > http://www.nabble.com/SolrIndexWriter-holding-reference-to-deleted-file--tp14436326p14503231.html > Sent from the Solr - User mailing list archive at Nabble.com. > >
