What merge factor are you using now? The merge factor will influence the
number of files that are created as the index grows. Lower = fewer file
descriptors needed, but also slower bulk indexing.
You could up the Max Open Files settings on your OS.
You could also use
<!-- options specific to the main on-disk lucene index -->
<useCompoundFile>true</useCompoundFile>
Which writes multiple segments to one file and requires *way* less file
handles (slightly slower indexing).
It would normally be odd to hit something like that after only 50,000
documents, but a doc with 300 fields is certainly not the norm ;) Anything
else special about your setup?
--
- Mark
http://www.lucidimagination.com
On Tue, Jul 14, 2009 at 12:49 PM, Bruno Aranda <brunoara...@gmail.com>wrote:
> Hi,
>
> We are having a TooManyOpenFiles exception in our indexing process. We
> are reading data from a database and indexing this data into one of
> the two cores of our solr instance. Each of the cores has a different
> schema as they are used for a different purpose. While we index in the
> first core, we do many searches in the second core as it contains data
> to "enrich" what we index (the second core is never modifier - read
> only). After indexing about 50.000 documents (about 300 fields each)
> we get the exception. If we run the same process, but without the
> "enrichment" (not doing queries in the second core), everything goes
> all right.
> We are using spring batch, and we only commit+optimize at the very
> end, as we don't need to search anything in the data that is being
> indexed.
>
> I have seen recommendations that go from committing+optimize more
> often or lowering the merge factor? How is the merge factor affecting
> in this scenario?
>
> Thanks,
>
> Bruno
>
