Hi,
I would appreciate some guidance here
I am trying to reconfigure OSD to put DB/WAL on NVME
so I "out" and "ceph osd rm" them ( because they were automatically
configured
and I did not "set-unamage" them properly ) then "wipefs -af " them
Now CEPH is seeing drives available for deploying OSD
on but it fails doing it due to "device or resource busy"
Attempting to destroy and zap it fails due to "device or resource busy"
even after I "wipefs -af /dev/sdaa" it
this is what it is on it ( output of lsblk)
sdaa
18.2T disk
└─ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f0169
18.2T lvm
lvs,vgs does not find any logical disks /volume
cephadm ceph-volume inventory is seeing the disks as "available = True"
cephadm ceph-volume lvm list is NOT finding any LVM on /dev/sdaa
docker ps -a does not show any "left over" OSD docker
There is a reference ( block link) to the
/dev/ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f016
in /var/lib/cephFSID/removed/osd.NUMER_DATE
How can I 'clean up' the disk ?
Many thanks
Steven
cephadm ceph-volume lvm zap --destroy /dev/sdaa
Inferring fsid 0cfa836d-68b5-11f0-90bf-7cc2558e5ce8
Not using image
'sha256:4892a7ef541bbfe6181ff8fd5c8e03957338f7dd73de94986a5f15e185dacd51'
as it's not in list of non-dangling images with ceph=True label
Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host
--stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint
/usr/sbin/ceph-volume --privileged --group-add=disk --init -e
CONTAINER_IMAGE=quay.io/ceph/ceph:v19 -e NODE_NAME=ceph-host-1 -e
CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v
/var/run/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/run/ceph:z -v
/var/log/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8:/var/log/ceph:z -v
/var/lib/ceph/0cfa836d-68b5-11f0-90bf-7cc2558e5ce8/crash:/var/lib/ceph/crash:z
-v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v
/run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v
/tmp/ceph-tmp419ql4q7:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v19 lvm zap
--destroy /dev/sdaa
/usr/bin/docker: stderr --> Zapping: /dev/sdaa
/usr/bin/docker: stderr --> Removing all BlueStore signature on /dev/sdaa
if any...
/usr/bin/docker: stderr Running command: /usr/bin/ceph-bluestore-tool
zap-device --dev /dev/sdaa --yes-i-really-really-mean-it
/usr/bin/docker: stderr stderr: error from zap: (16) Device or resource
busy
/usr/bin/docker: stderr 2025-09-02T17:41:02.385+0000 72def157f9c0 -1
bdev(0x57b37fe52000 /dev/sdaa) open open got: (16) Device or resource busy
Some extra info
cephadm ceph-volume inventory output
Device Path Size Device nodes rotates available
Model name
/dev/nvme3n1 1.46 TB nvme3n1 False True
MTFDKCC1T6TGQ-1BK1DABYY
/dev/sdaa 18.19 TB sdaa True True
ST20000NM007D-3D
/dev/sdv 18.19 TB sdv True True
ST20000NM007D-3D
/dev/sdw 18.19 TB sdw True True
ST20000NM007D-3D
/dev/sdx 18.19 TB sdx True True
ST20000NM007D-3D
/dev/sdy 18.19 TB sdy True True
ST20000NM007D-3D
/dev/sdz 18.19 TB sdz True True
ST20000NM007D-3D
lsblk output
NAME
SIZE TYPE MOUNTPOINT
sdaa
18.2T disk
└─ceph--dfaa321c--b04d--48e8--b78b--7c39436be0e5-osd--block--36473043--6d2c--4c7a--b61c--b057e83f0169
18.2T lvm
lvs output ( greped for lsblk /dev/sdaa and couldn't find it )
osd-block-29a5b1f5-d442-4756-a61d-e230832a3577
ceph-013dc6ae-90c7-4271-b308-895bbafe8840 -wi-ao---- <6.99t
osd-block-5f7471f4-22ee-4cf9-9a16-2854a0c26496
ceph-27dfe22b-8316-40e3-aeee-1686bf0a95b7 -wi-ao---- <6.99t
osd-block-ff6ca83d-80c5-4fff-a50d-399d94ba1b1b
ceph-2de80ef9-8095-44be-9df2-91f458b1f1ea -wi-ao---- 18.19t
osd-block-fc022b7e-3d85-4435-998e-6513020bbdf3
ceph-57f7e6bf-cf30-4d81-8c97-d0cb503cb8f8 -wi-ao---- 13.97t
osd-block-347afb43-812c-4330-a795-060e76f33f89
ceph-5815ca32-056b-476a-8a16-64c06b9c3fe9 -wi-ao---- <6.99t
osd-block-c96901b7-8b7e-4a94-be0a-ab1316fd114e
ceph-58a9cfda-5090-47f5-bfe8-b993d6133218 -wi-ao---- <6.99t
osd-block-5e069648-e0ec-4490-a3a9-e758d6e3e1d4
ceph-59b5ad1b-a35b-4c46-b737-7328eee6ec45 -wi-ao---- 13.97t
osd-block-5d7dc85e-dcc8-4d87-b7ba-d61201dccd5e
ceph-6aedba8c-3c5a-476d-9e0c-9fd159c8c09e -wi-ao---- <6.99t
osd-block-d18a4abc-f723-4ac8-9d05-d02edd7e6018
ceph-6f7e544b-9ead-40c4-82ce-646272461b36 -wi-ao---- <6.99t
osd-block-33404537-6f45-4f10-af8a-8de491e8bcc9
ceph-7060eb95-cfb9-441b-9094-1803348ca3f2 -wi-ao---- 18.19t
osd-block-153d1ed5-007c-439e-ad9e-c20d1880ff4d
ceph-9729cc29-705c-42e5-bcc6-5d7cfd62a4ac -wi-ao---- 18.19t
osd-block-15f14591-43c9-42ed-8410-d7f216019b55
ceph-9d628846-1c8f-4c73-9c7f-6e34c69210a3 -wi-ao---- 18.19t
osd-block-44c310eb-2002-4817-8b2e-beacb0a70806
ceph-ab1ca6f2-30c8-45a7-bfec-4e09089ec856 -wi-ao---- <6.99t
osd-block-b55d7d9f-8e2a-4da5-b9c5-ace96975286d
ceph-ba4098bf-6808-4a65-93a6-e84f3ff7936a -wi-ao---- 13.97t
osd-block-a97d2620-d07a-4bd7-8f19-66801f5d613b
ceph-c93d909a-b944-4dc5-a9b2-ed3441226c46 -wi-ao---- 18.19t
osd-block-5c134f31-3a17-4fe9-989f-fb3a1dc2c0e9
ceph-c9e00bac-501b-4449-b23c-d914c61a6838 -wi-ao---- <6.99t
osd-block-89390a0d-6bc3-41a5-a79b-ebac922d540f
ceph-d8a23938-be11-4cee-ba13-951996ca9328 -wi-ao---- <6.99t
osd-block-cfabfe08-5e42-46ba-8412-1d264e9312d3
ceph-dcff97c0-5caa-4cd2-87f5-baa552b0b172 -wi-ao---- <6.99t
osd-block-2e86866b-b537-4b29-86e7-490e627f82be
ceph-de7ee41e-a121-4ecd-a838-cf284e811b7e -wi-ao---- 18.19t
osd-block-ab0b170e-91e0-41d4-a5ea-b5c4fef6c3d8
ceph-e6d22c3b-bfb6-494c-b09d-15a667816442 -wi-ao---- <6.99t
osd-db-012f9187-1596-4db8-96ba-809a96bce8fd
ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g
osd-db-124f39e2-f28d-47a8-acce-fb1364f7a638
ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g
osd-db-17edf308-b36b-42be-8855-55833d35f983
ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g
osd-db-55eb88b3-89a0-4931-a1be-d73e2aaf527f
ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g
osd-db-5e03ba4b-5641-49c6-a477-1b37dffcd398
ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g
osd-db-b22a1142-8dec-4035-96b1-7717c4587bcb
ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 -wi-ao---- 248.39g
osd-block-f9c81b89-a903-4a11-9c2a-9171b4a5760a
ceph-f8276850-dfb2-4689-a5b1-e54925605dc4 -wi-ao---- <6.99t
VGS output ( ( greped for lsblk /dev/sdaa result and couldn't find it )
VG #PV #LV #SN Attr VSize VFree
ceph-013dc6ae-90c7-4271-b308-895bbafe8840 1 1 0 wz--n- <6.99t 0
ceph-27dfe22b-8316-40e3-aeee-1686bf0a95b7 1 1 0 wz--n- <6.99t 0
ceph-2de80ef9-8095-44be-9df2-91f458b1f1ea 1 1 0 wz--n- 18.19t 0
ceph-57f7e6bf-cf30-4d81-8c97-d0cb503cb8f8 1 1 0 wz--n- 13.97t 0
ceph-5815ca32-056b-476a-8a16-64c06b9c3fe9 1 1 0 wz--n- <6.99t 0
ceph-58a9cfda-5090-47f5-bfe8-b993d6133218 1 1 0 wz--n- <6.99t 0
ceph-59b5ad1b-a35b-4c46-b737-7328eee6ec45 1 1 0 wz--n- 13.97t 0
ceph-6aedba8c-3c5a-476d-9e0c-9fd159c8c09e 1 1 0 wz--n- <6.99t 0
ceph-6f7e544b-9ead-40c4-82ce-646272461b36 1 1 0 wz--n- <6.99t 0
ceph-7060eb95-cfb9-441b-9094-1803348ca3f2 1 1 0 wz--n- 18.19t 0
ceph-9729cc29-705c-42e5-bcc6-5d7cfd62a4ac 1 1 0 wz--n- 18.19t 0
ceph-9d628846-1c8f-4c73-9c7f-6e34c69210a3 1 1 0 wz--n- 18.19t 0
ceph-ab1ca6f2-30c8-45a7-bfec-4e09089ec856 1 1 0 wz--n- <6.99t 0
ceph-ba4098bf-6808-4a65-93a6-e84f3ff7936a 1 1 0 wz--n- 13.97t 0
ceph-c93d909a-b944-4dc5-a9b2-ed3441226c46 1 1 0 wz--n- 18.19t 0
ceph-c9e00bac-501b-4449-b23c-d914c61a6838 1 1 0 wz--n- <6.99t 0
ceph-d8a23938-be11-4cee-ba13-951996ca9328 1 1 0 wz--n- <6.99t 0
ceph-dcff97c0-5caa-4cd2-87f5-baa552b0b172 1 1 0 wz--n- <6.99t 0
ceph-de7ee41e-a121-4ecd-a838-cf284e811b7e 1 1 0 wz--n- 18.19t 0
ceph-e6d22c3b-bfb6-494c-b09d-15a667816442 1 1 0 wz--n- <6.99t 0
ceph-f0e77cf5-9878-4a9b-bdc4-22211aadb951 1 6 0 wz--n- <1.46t 8.00m
ceph-f8276850-dfb2-4689-a5b1-e54925605dc4 1 1 0 wz--n- <6.99t 0
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
