On 6/4/2020 9:51 PM, Srinivas Kashyap wrote:
We are on solr 8.4.1 and In standalone server mode. We have a core with
497,767,038 Records indexed. It took around 32Hours to load data through DIH.
The disk occupancy is shown below:
82G /var/solr/data/<corename>/data/index
When I restarted solr instance and went to this core to query on solr admin GUI, it is
hanging and is showing "Connection to Solr lost. Please check the Solr
instance". But when I go back to dashboard, instance is up and I'm able to query
other cores.
Also, querying on this core is eating up JVM memory allocated(24GB)/(32GB RAM).
A query(*:*) with filterqueries is overshooting the memory with OOM.
You're going to want to have a lot more than 8GB available memory for
disk caching with an 82GB index. That's a performance thing... with so
little caching memory, Solr will be slow, but functional. That aspect
of your setup will NOT lead to out of memory.
If you are experiencing Java "OutOfMemoryError" exceptions, you will
need to figure out what resource is running out. It might be heap
memory, but it also might be that you're hitting the process/thread
limit of your operating system. And there are other possible causes for
that exception too. Do you have the text of the exception available?
It will be absolutely critical for you to determine what resource is
running out, or you might focus your efforts on the wrong thing.
If it's heap memory (something that I can't really assume), then Solr is
requiring more than the 24GB heap you've allocated.
Do you have faceting or grouping on those queries? Are any of your
filters really large or complex? These are the things that I would
imagine as requiring lots of heap memory.
What is the size of your filterCache? With about 500 million documents
in the core, each entry in the filterCache will consume nearly 60
megabytes of memory. If your filterCache has the default example size
of 512, and it actually gets that big, then that single cache will
require nearly 30 gigabytes of heap memory (on top of the other things
in Solr that require heap) ... and you only have 24GB. That could cause
OOME exceptions.
Does the server run things other than Solr?
Look here for some valuable info about performance and memory:
https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems
Thanks,
Shawn
