On 4/23/2019 6:34 AM, Brian Ecker wrote:
What I’m trying to determine is (1) How much heap does
this setup need before it stabilizes and stops crashing with OOM errors,
(2) can this requirement somehow be reduced so that we can use less
memory, and (3) from the heap histogram, what is actually using memory
(lots of primitive type arrays and data structures, but what part of
Solr is using those)?
Exactly one attachment made it through: The file named
solrconfig-anonymized.xml. Attachments can't be used to share files
because the mailing list software is going to eat them and we won't see
them. You'll need to use a file sharing website. Dropbox is often a
good choice.
We won't be able to tell anything about what's using all the memory from
a histogram. We would need an actual heap dump from Java. This file
will be huge -- if you have a 10GB heap, and that heap is full, the file
will likely be larger than 10GB.
There is no way for us to know how much heap you need. With a large
amount of information about your setup, we can make a guess, but that
guess will probably be wrong. Info we'll need to make a start:
*) How many documents is this Solr instance handling? You find this out
by looking at every core and adding up all the "maxDoc" numbers.
*) How much disk space is the index data taking? This could be found
either by getting a disk usage value for the solr home, or looking at
every core and adding up the size of each one.
*) What kind of queries are you running? Anything with facets, or
grouping? Are you using a lot of sort fields?
*) What kind of data is in each document, and how large is that data?
Your cache sizes are reasonable. So you can't reduce heap requirements
by much by reducing cache sizes.
Here's some info about what takes a lot of heap and ideas for reducing
the requirements:
https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
That page also reiterates what I said above: It's unlikely that anybody
will be able to tell you exactly how much heap you need at a minimum.
We can make guesses, but those guesses might be wrong.
Thanks,
Shawn
