We are creating an index of about 500 - 600M records fairly small documents.
We are currently @ 252M+ records and adding documents at a rate of about 2k - 3k per second in multithreaded 1K batches sent from multiple servers. Our commits started bombing out with out of Memory Exceptions with 14G. 17G on the heap works, but begins to push memory usage above the available memory. Even when reducing the number of documents committed does not seem to mitigate the out of memory issue. When looking at the JMX charts on memory usage it doesn't look like a memory leak is present as the graph jumps up and down while maintaining a steady minimum within. However I do get the odd behavior that the memory spike maximums gradually increase during indexing, which seems odd.
