On 2/21/2018 7:58 PM, Susheel Kumar wrote:
Below output for prod machine based on the steps you described. Please
take a look. The solr searches are returning fine and no issue with
performance but since last 4 months swap space started going up. After
restart, it comes down to zero and then few weeks, it utilization reaches
to 40-50% and thus requires restart of solr process.
I bet that if you run this command, it will show you a value of 60:
cat /proc/sys/vm/swappiness
This makes the OS very aggressive about using swap, even when there is
absolutely no need for it to do so.
If you type the following series of commands, it should fix the problem
and prevent it from happening again until you reboot the system:
echo "0" > /proc/sys/vm/swappiness
swapoff -a
swapon -a
Note that when the swapoff command runs, it will force the OS to read
all the swapped data back into memory. It will take several minutes for
this to occur, because it must read nearly a gigabyte of data and figure
out how to put it back in memory. Both of the command outputs you
included say that there is over 20GB of free memory. So I do not
anticipate the system having problems from running these commands. It
will slow the machine down temporarily, though -- so only do it during a
quiet time for your Solr install.
To make this setting survive a reboot, find the sysctl.conf file
somewhere in your /etc directory and add this line to it:
vm.swappiness = 0
This setting does not completely disable swap. If the system finds
itself with real memory pressure and actually does NEED to use swap, it
still will ... it just won't swap anything out before it's actually
required.
I do not think the behavior you are seeing is actually causing problems,
based on your system load and CPU usage. But what I've shared should
fix it for you.
Thanks,
Shawn
