i am trying to add 20 million documents to my index from another index that
contains these documents(cant help this architecture..its something that i will
have to follow) Now the problems i am facing are the following :
1) Too many files open error.. its at the code which is adding documents to my
index.
IndexWriter w = new IndexWriter();
IndexWriter w = new IndexWriter(index , analyzer,
IndexWriter.MaxFieldLength.UNLIMITED );
w.setMergeFactor(1000);
w.setMaxBufferedDocs(1000);
w.setMaxMergeDocs(600000);
for(i =0;i<reader1.numDocs();i++)
{
System.out.println(i);
addDoc(w, reader1.document(i).getField(" url ").stringValue(
),reader1.document(i).getField(" content ").stringValue().replace('.','
').replace('-',' '));
}
w.optimize();
w.close();
reader1.close();
Due to the mergefactor parameters that i have set, around 1300 .cfs files are
now opened in index, but there is only one fdt file. These files seem to be the
reason for the this error.
Are these files not closed?? do i have to do IndexWriter.commit() in the loop
to close these open files??
