Re: Indexing a lot of documents?

2007-06-01 Thread Jordan Hayes
I'm not clear exactly what you mean by batches. What I'm doing is: <-- #1 [...] <-- #100 So that's 100, in 1 HTTP POST. Lucene sometimes just requires many file descriptors (this will be somewhat alleviated with Solr 1.2). Is there a way to find out how many is "many" ...

Indexing a lot of documents?

2007-06-01 Thread Jordan Hayes
New user here, and I ran into a problem trying to load a lot of documents (~900k). I tried to load them all at once, which seemed to run for a long time and then finally crap out with "Too many open files" ... so I read in an FAQ that "about 100" might be a good number. I split my documents u