It's _very_ unlikely that optimize will help with OOMs, so that's very probably a red herring. Likely the document that's causing the issue is very large or, perhaps, you're using the extracting processor and it might be a Tika issue, consider doing the Tika processing outside Solr if so, see:
https://lucidworks.com/2012/02/14/indexing-with-solrj/ But forget optimize for curing this problem ;) Best, Erick On Thu, Mar 2, 2017 at 9:37 AM, Caruana, Matthew <mcaru...@icij.org> wrote: > Thank you, these are useful tips. > > We were previously working with a 4GB heap and getting OOMs in Solr while > updating (probably from the analysers) that would cause the index writer to > close with what’s called a “tragic” error in the writer code. Only a hard > restart of the service could bring it back. There are certain documents that > function like poison and trigger this error every time. Haven’t had time to > isolate and create a test case, so throwing RAM at it is a stopgap. > > When I do, I’ll file an issue. > >> On 2 Mar 2017, at 18:28, Walter Underwood <wun...@wunderwood.org> wrote: >> >> 6.4.0 added a lot of metrics to low-level calls. That makes many operations >> slow. Go back to 6.3.0 or wait for 6.4.2. >> >> Meanwhile, stop running optimize. You almost certainly don’t need it. >> >> 24 GB is a huge heap. Do you really need that? We run a 15 million doc index >> with an 8 GB heap (Java 8u121, G1 collector). I recommend a smaller heap so >> the OS can use that RAM to cache file buffers. >> >> wunder >> Walter Underwood >> wun...@wunderwood.org >> http://observer.wunderwood.org/ (my blog) >> >> >>> On Mar 2, 2017, at 7:04 AM, Caruana, Matthew <mcaru...@icij.org> wrote: >>> >>> I’m currently performing an optimise operation on a ~190GB index with about >>> 4 million documents. The process has been running for hours. >>> >>> This is surprising, because the machine is an EC2 r4.xlarge with four cores >>> and 30GB of RAM, 24GB of which is allocated to the JVM. >>> >>> The load average has been steady at about 1.3. Memory usage is 25% or less >>> the whole time. iostat reports ~6% util. >>> >>> What gives? >>> >>> Running Solr 6.4.1. >> >
