MeihanLi commented on issue #11293:
URL: https://github.com/apache/pinot/issues/11293#issuecomment-1675205320
Thanks @Jackie-Jiang , We found also segment reloading thread can take ~1
core but it will trigger lots of G1 GC which can take ~4.5 cores. That
explained why in production, we saw increased CPU usage and Young Gen Time a
lot. Is there a way to improve the performance of reloading one segment?
[Testing] Before reloading:
```
top - 23:29:17 up 145 days, 15:49, 0 users, load average: 0.76, 1.01, 1.08
Threads: 372 total, 0 running, 372 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.9 us, 0.6 sy, 0.5 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
MiB Mem : 385616.8 total, 155857.7 free, 145778.1 used, 83981.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 233824.1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
446 udocker 20 0 353.9g 178.4g 43.2g S 6.2 47.4 0:53.20
rta_comms_log_m
480 udocker 20 0 353.9g 178.4g 43.2g S 6.2 47.4 0:52.93
rta_comms_log_m
1 udocker 20 0 353.9g 178.4g 43.2g S 0.0 47.4 0:00.04 java
46 udocker 20 0 353.9g 178.4g 43.2g S 0.0 47.4 0:08.96 java
47 udocker 20 0 353.9g 178.4g 43.2g S 0.0 47.4 0:39.76 GC
Thread#0
48 udocker 20 0 353.9g 178.4g 43.2g S 0.0 47.4 0:00.00 G1
Main Marker
49 udocker 20 0 353.9g 178.4g 43.2g S 0.0 47.4 0:00.27 G1
Conc#0
50 udocker 20 0 353.9g 178.4g 43.2g S 0.0 47.4 1:26.17 G1
Refine#0
```
[Testing] During reloading:
```
top - 23:32:35 up 145 days, 15:53, 0 users, load average: 3.19, 1.45, 1.21
Threads: 374 total, 3 running, 371 sleeping, 0 stopped, 0 zombie
%Cpu(s): 10.4 us, 0.8 sy, 0.5 ni, 88.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0
st
MiB Mem : 385616.8 total, 155590.3 free, 145843.2 used, 84183.2 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 233759.2 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4476 udocker 20 0 354.0g 177.7g 42.1g R 99.9 47.2 0:21.01
segment-refresh
50 udocker 20 0 354.0g 177.7g 42.1g R 93.8 47.2 1:35.56 G1
Refine#0
250 udocker 20 0 354.0g 177.7g 42.1g S 81.2 47.2 0:37.80 G1
Refine#1
251 udocker 20 0 354.0g 177.7g 42.1g S 56.2 47.2 0:35.32 G1
Refine#2
252 udocker 20 0 354.0g 177.7g 42.1g S 37.5 47.2 0:33.19 G1
Refine#3
253 udocker 20 0 354.0g 177.7g 42.1g S 31.2 47.2 0:32.30 G1
Refine#4
254 udocker 20 0 354.0g 177.7g 42.1g S 25.0 47.2 0:31.00 G1
Refine#5
255 udocker 20 0 354.0g 177.7g 42.1g S 12.5 47.2 0:30.51 G1
Refine#6
256 udocker 20 0 354.0g 177.7g 42.1g S 6.2 47.2 0:29.47 G1
Refine#7
257 udocker 20 0 354.0g 177.7g 42.1g S 6.2 47.2 0:28.70 G1
Refine#8
440 udocker 20 0 354.0g 177.7g 42.1g S 6.2 47.2 0:55.65
rta_comms_log_m
459 udocker 20 0 354.0g 177.7g 42.1g S 6.2 47.2 0:54.85
rta_comms_log_m
3584 udocker 20 0 354.0g 177.7g 42.1g S 6.2 47.2 0:46.66
rta_comms_log_m
```
[Testing] CPU Usage
<img width="1901" alt="image"
src="https://github.com/apache/pinot/assets/42751784/e74c715a-8129-49d7-92eb-82e52c43fb91";>
[Testing] Young Gen Time
<img width="456" alt="image"
src="https://github.com/apache/pinot/assets/42751784/0fdff319-881f-4894-8461-05a88b5dc103";>
[Production] CPU Usage
<img width="923" alt="image"
src="https://github.com/apache/pinot/assets/42751784/ac82de9f-c1d6-4e1f-bb2e-86d1f580a591";>
[Production] Young Gen Time
<img width="453" alt="image"
src="https://github.com/apache/pinot/assets/42751784/4487b5fe-0d68-478b-ad45-4f09111aa5fd";>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
