Hi Shawn,
Below output for prod machine based on the steps you described. Please
take a look. The solr searches are returning fine and no issue with
performance but since last 4 months swap space started going up. After
restart, it comes down to zero and then few weeks, it utilization reaches
to 40-50% and thus requires restart of solr process.
top - 21:31:37 up 116 days, 20:09, 3 users, load average: 0.24, 0.24, 0.16
Tasks: 202 total, 1 running, 201 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.4 us, 0.0 sy, 0.0 ni, 98.5 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
KiB Mem: 65976996 total, 43846008 used, 22130988 free, 84 buffers
KiB Swap: 2097148 total, 942228 used, 1154920 free. 39754116 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
SWAP
20327 solr 20 0 53.791g 0.021t 0.017t S 10.60 33.61 5109:53
java
547524
481 root 20 0 267688 153312 9012 S 0.662 0.232 47:44.63
splunkd
34816
1293 root 20 0 94676 5536 5168 S 0.331 0.008 99:01.45
vmtoolsd
932
2699 root 20 0 21212 2740 2112 S 0.331 0.004 6:17.38
cvfwd
236
17706 kumarsu 20 0 46936 7600 5472 R 0.331 0.012 0:00.18 top
0
1 root 20 0 37532 5360 3896 S 0.000 0.008 6:07.28
systemd
248
2 root 20 0 0 0 0 S 0.000 0.000 0:04.66
kthreadd
0
3 root 20 0 0 0 0 S 0.000 0.000 0:38.21
ksoftirqd/0
0
5 root 0 -20 0 0 0 S 0.000 0.000 0:00.00
kworker/0:0H
0
7 root 20 0 0 0 0 S 0.000 0.000 115:56.38
rcu_sched
0
8 root 20 0 0 0 0 S 0.000 0.000 0:00.00
rcu_bh
0
9 root rt 0 0 0 0 S 0.000 0.000 0:28.61
migration/0
0
10 root rt 0 0 0 0 S 0.000 0.000 0:24.68
watchdog/0
0
11 root rt 0 0 0 0 S 0.000 0.000 0:23.78
watchdog/1
0
12 root rt 0 0 0 0 S 0.000 0.000 0:29.63
migration/1
0
13 root 20 0 0 0 0 S 0.000 0.000 1:49.79
ksoftirqd/1
0
15 root 0 -20 0 0 0 S 0.000 0.000 0:00.00
kworker/1:0H
0
16 root rt 0 0 0 0 S 0.000 0.000 0:24.00
watchdog/2
0
17 root rt 0 0 0 0 S 0.000 0.000 0:28.67
migration/2
0
18 root 20 0 0 0 0 S 0.000 0.000 0:26.13
ksoftirqd/2
0
20 root 0 -20 0 0 0 S 0.000 0.000 0:00.00
kworker/2:0H
0
21 root rt 0 0 0 0 S 0.000 0.000 0:26.61
watchdog/3
0
22 root rt 0 0 0 0 S 0.000 0.000 0:27.92
migration/3
0
23 root 20 0 0 0 0 S 0.000 0.000 0:23.08
ksoftirqd/3
0
25 root 0 -20 0 0 0 S 0.000 0.000 0:00.00
kworker/3:0H
0
26 root rt 0 0 0 0 S 0.000 0.000 0:28.02
watchdog/4
0
27 root rt 0 0 0 0 S 0.000 0.000 0:28.01
migration/4
0
28 root 20 0 0 0 0 S 0.000 0.000 0:51.07
ksoftirqd/4
0
31 root rt 0 0 0 0 S 0.000 0.000 0:25.29
watchdog/5
0
32 root rt 0 0 0 0 S 0.000 0.000 0:31.65
migration/5
0
33 root 20 0 0 0 0 S 0.000 0.000 0:22.52
ksoftirqd/5
0
35 root 0 -20 0 0 0 S 0.000 0.000 0:00.00
kworker/5:0H
0
36 root rt 0 0 0 0 S 0.000 0.000 0:25.90
watchdog/6
0
37 root rt 0 0 0 0 S 0.000 0.000 0:27.94
migration/6
0
38 root 20 0 0 0 0 S 0.000 0.000 1:07.91
ksoftirqd/6
0
40 root 0 -20 0 0 0 S 0.000 0.000 0:00.00
kworker/6:0H
0
>free -m
total used free shared buffers cached
Mem: 64430 42816 21614 132 0 38823
-/+ buffers/cache: 3992 60437
Swap: 2047 920 1127
On Wed, Feb 21, 2018 at 8:45 PM, Shawn Heisey <apa...@elyograg.org> wrote:
> On 2/21/2018 1:30 PM, Susheel Kumar wrote:
> > I did go thru your posts on swap usage http://lucene.472066.n3.
> > nabble.com/Solr-4-3-1-memory-swapping-td4126641.html and my situation is
> > also similar. Below is top output from our prod and performance test
> > machine and as you can see the swap utilization on Prod machine is 44%
> > while on test machines it is zero.
>
> Are those "top" outputs sorted by the default (CPU usage) or by memory?
> To make any useful determination, it needs to be by memory. Press
> shift-M to sort by memory if your top program supports that key. Also,
> the list only shows the first few processes. More of the list needs to
> be visible.
>
> The system load on the second "top" output is quite low, and doesn't
> show much used CPU percentage. So it looks like the system is not
> actually suffering due to the swap usage, which probably means that it
> is not actively swapping. The machine has plenty of memory available --
> even though almost all of the memory is allocated, the vast majority of
> what's allocated is in the "cached" state -- used by the OS disk cache.
> The OS will instantly give up this memory if a program requests it.
>
> I've learned how to use top to show which processes are using swap.
> What I've described below should work on recent version of gnu top. If
> the top is from another software provider, it may not support this.
>
> http://northernmost.org/blog/swap-usage-5-years-later/
>
> These steps are not precisely as described in that blog post: Run top,
> press f, press p, press space, then press the right angle bracket (>)
> key three times. If top is running with default settings, these
> keypresses should enable the SWAP column and move the sort to that
> column. The list should be sorted by swap usage. If a .toprc file
> exists in your home directory, then the program may be running with very
> different settings than default, and these keypresses might not work as
> expected.
>
> Thanks,
> Shawn
>
>
