Hi List!
We are in the middle of migration from Ceph v16 to v18.
As a first step of migration we decided to change OS version on all OSD
hosts from Rocky Linux 8.10 to Rocky Linux 9.5 (by using clean
reinstall). We also upgraded OSD Block disks from 4TB to 10TB.
Now we have 6 OSD hosts running Rocky 9 with new 10TB disks (8 OSD Block
disks per Host). OSD hosts have 64GB of RAM.
At this moment Ceph cluster is still on version 16.2.15 "Pacific".
Migration to Ceph 18.2.x will be performed later after upgrading
to Rocky 9 all remaining parts (MGR+MON, MDS, RGW).
Problem:
After migration to Rocky 9 (and new version of Docker) we see what our
OSD hosts consumes 100% of SWAP space! It takes approximately one week
to fill SWAP from 0% to 100%.
Docker containers running OSD instances are restarted from time to time
(and there is a lot of ugly "Call Trace" messages in dmesg).
I was wondering if anyone else has similar issues?
Will it be gone after migration to Ceph v18?
P.S.
Configuration Details:
=====
[ceph: root@ceph-mon12 /]# ceph versions
{
"mon": {
"ceph version 16.2.15
(618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 3
},
"mgr": {
"ceph version 16.2.15
(618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 3
},
"osd": {
"ceph version 16.2.15
(618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 48
},
"mds": {
"ceph version 16.2.15
(618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 2
},
"rgw": {
"ceph version 16.2.15
(618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 2
},
"overall": {
"ceph version 16.2.15
(618f440892089921c3e944a991122ddc44e60516) pacific (stable)": 58
}
}
[ceph: root@ceph-mon12 /]#
===
Old Rocky 8 OSD Host:
[root@ceph-osd11 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED
STATUS PORTS NAMES
1a4c959eb813 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago
Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-37
e37a366347db quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago
Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-1
0fdfb95fd631 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago
Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-14
276ad8a11148 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago
Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-31
9dbd30d11124 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago
Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-19
0ad59f583e6a quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago
Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-7
e551a652c3ae quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago
Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-42
07ff632ae3e1 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 7 months ago
Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-25
605b7a1c00f7 quay.io/ceph/ceph "/usr/bin/ceph-crash…" 7 months ago
Up 7 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd11
fe2a6b8a35e8 quay.io/prometheus/node-exporter:v1.3.1
"/bin/node_exporter …" 7 months ago Up 7 months
ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd11
[root@ceph-osd11 ~]# uptime
10:52:59 up 216 days, 20:12, 1 user, load average: 0.20, 0.41, 0.53
[root@ceph-osd11 ~]# free -h
total used free shared buff/cache
available
Mem: 62Gi 30Gi 1.2Gi 2.1Gi 30Gi 29Gi
Swap: 8.0Gi 2.8Gi 5.2Gi
[root@ceph-osd11 ~]# rpm -qa | grep docker
docker-buildx-plugin-0.14.0-1.el8.x86_64
docker-ce-cli-26.1.3-1.el8.x86_64
docker-ce-26.1.3-1.el8.x86_64
docker-scan-plugin-0.23.0-3.el8.x86_64
docker-ce-rootless-extras-26.1.3-1.el8.x86_64
docker-compose-plugin-2.27.0-1.el8.x86_64
[root@ceph-osd11 ~]# cat /etc/rocky-release
Rocky Linux release 8.10 (Green Obsidian)
[root@ceph-osd11 ~]# uname -a
Linux ceph-osd11 4.18.0-553.16.1.el8_10.x86_64 #1 SMP Thu Aug 8 17:47:08
UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
===
New Rocky 9 OSD Host:
[root@ceph-osd17 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED
STATUS PORTS NAMES
5cc58e4a77b2 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 days ago
Up 4 days ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-52
3a60fecf648d quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago
Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-50
9c20407e79eb quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago
Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-49
9deadafef9dd quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago
Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-48
fcfe62a25fd9 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago
Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-55
38d2d96cc491 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago
Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-51
e29c6bbc1ae7 quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago
Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-54
40346a7a45ea quay.io/ceph/ceph "/usr/bin/ceph-osd -…" 4 months ago
Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-osd-53
43c3e3a65531 quay.io/ceph/ceph "/usr/bin/ceph-crash…" 4 months ago
Up 4 months ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-crash-ceph-osd17
d9e436f9788c quay.io/prometheus/node-exporter:v1.3.1
"/bin/node_exporter …" 4 months ago Up 4 months
ceph-7e8bff5c-2761-11ec-9bb0-000c29ebc936-node-exporter-ceph-osd17
[root@ceph-osd17 ~]# uptime
10:25:10 up 122 days, 22:37, 1 user, load average: 0.37, 0.26, 0.27
[root@ceph-osd17 ~]# free -h
total used free shared buff/cache
available
Mem: 62Gi 26Gi 1.0Gi 1.0Gi 36Gi 36Gi
Swap: 8.0Gi 8.0Gi 7.0Mi
[root@ceph-osd17 ~]# rpm -qa | grep docker
docker-compose-plugin-2.31.0-1.el9.x86_64
docker-buildx-plugin-0.19.2-1.el9.x86_64
docker-ce-cli-27.4.0-1.el9.x86_64
docker-ce-rootless-extras-27.4.0-1.el9.x86_64
docker-ce-27.4.0-1.el9.x86_64
[root@ceph-osd17 ~]# cat /etc/rocky-release
Rocky Linux release 9.5 (Blue Onyx)
[root@ceph-osd17 ~]# uname -a
Linux ceph-osd17 5.14.0-503.15.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Tue
Nov 26 17:24:29 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
=====
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
