On 1/20/2014 3:02 AM, onetwothree wrote:
OS Windows server 2008
4 Cpu
8 GB Ram
<snip>
We're using a .Net Service (based on Solr.Net) for updating and inserting
documents on a single Solr Core instance. The size of documents sent to Solr
vary from 1 Kb up to 8Mb, we're sending the documents in batches, using one
or multiple threads. The current size of the Solr Index is about 15GB.
The indexing service is running around 4 a 5 hours per day, to complete all
inserts and updates to Solr. While the indexing process is running the
Tomcat process memory usage keeps growing up to > 7GB Ram (using Process
Explorer monitor tool) and does not reduce, even after 24 hours. After a
restart of Tomcat, or a Reload Core in the Solr Admin the memory drops back
to 1 a 2 GB Ram. While using a tool like VisualVM to monitor the Tomcat
process, the memory usage of Tomcat seems ok, memory consumption is in range
of defined jvm startup params (see image).
So it seems that filesystem buffers are consuming all the leftover memory??,
and don't release memory, even after a quite amount of time? Is there a way
handle this behaviour, in a way that not all memory is consumed? Are there
other alternatives? Best practices?
<http://lucene.472066.n3.nabble.com/file/n4112262/Capture.png>
That picture seems to be a very low-res copy of your screenshot. I
can't really make it out. I can tell you that it's completely normal
for the OS disk cache (the filesystem buffers you mention) to take up
all leftover memory. If an application requests some of that memory,
the OS will instantly give it up.
First, I'm going to explain something about memory reporting and Solr
that I've noticed, then I will give you some news you probably won't like.
The numbers reported by visualvm are a true picture of Java heap memory
usage. The actual memory usage for Solr will be just a little bit more
than those numbers. In the newest versions of Solr, there seems to be a
side effect of the Java MMAP implementation that results in incorrect
memory usage reporting at the operating system level. Here's a "top"
output on one of my Solr servers running CentOS, sorted by memory
usage. The process at the top of the list is Solr.
https://www.dropbox.com/s/y1nus7lpzlb1mp9/solr-memory-usage-2014-01-20%2010.28.28.png
Some quick numbers for you: The machine has 64GB of RAM. Solr shows a
virtual memory size of 59.2GB. My indexes take up 51293336 of disk
space, and Solr has a 6GB heap, so 59.2GB is not out of line for the
virtual memory size.
Now for where things get weird: There is 48GB of RAM taken up by the
"cached" value, which is the OS disk cache. The screenshot also shows
that Solr is using 22GB of resident RAM. If you add the 48GB in the OS
disk cache and the 22GB of resident RAM for Solr, you get 70GB ... which
is more memory than the machine even HAS, so we know something's off.
The 'shared' memory for Solr is 15GB, which when you subtract it from
the 22GB, gives you 7GB, which is much more realistic with a 6GB heap,
and also makes it fit within the total system RAM.
The news that you probably won't like:
I'm assuming that the whole reason you looked into memory usage was
because you're having performance problems. With 8GB of RAM and 3GB
given to Solr, you basically have a little bit less than 5GB of RAM for
the OS disk cache. With that much RAM, most people can effectively
cache an index up to about 10GB before performance problems show up.
Your index is 15GB. You need more total system RAM. If Solr isn't
crashing, you can probably leave the heap at 3GB with no problem.
http://wiki.apache.org/solr/SolrPerformanceProblems
Thanks,
Shawn
