On Wed, Sep 10, 2025 at 8:49 AM Robert Sander
<[email protected]> wrote:
>
> Hi,
>
> Am 9/9/25 um 17:34 schrieb Yuri Weinstein:
>
> > * Dashboard: Added support for NVMe/TCP (gateway groups, multiple
> > namespaces),
>
> I have just upgraded a test cluster and NVMEoF does not work any more.
>
> The previous version running was quay.ceph.io/ceph-ci/ceph:squid-nvmeof and
> NVMEoF was working there.
>
> The gateways seem to work and subsystem and namespaces are still configured
> but an initiator does not see namespaces.
>
> # ceph orch ps --daemon_type nvmeof
> NAME HOST PORTS
> STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID
> CONTAINER ID
> nvmeof.hosa.b236asbrce101.ccaujq b236asbrce101 *:5500,4420,8009,10008
> running (13h) 36s ago 13h 128M - 1.5.7 2c3e7ef350b8
> 74672e87a945
> nvmeof.hosa.b236asbrce102.gnbwsc b236asbrce102 *:5500,4420,8009,10008
> running (13h) 36s ago 13h 120M - 1.5.7 2c3e7ef350b8
> 1929979ab7e1
> nvmeof.hosa.b236asbrce103.bthewi b236asbrce103 *:5500,4420,8009,10008
> running (13h) 36s ago 13h 122M - 1.5.7 2c3e7ef350b8
> 14885171d1a5
> nvmeof.hosa.b236asbrce104.vcgzvg b236asbrce104 *:5500,4420,8009,10008
> running (13h) 36s ago 13h 122M - 1.5.7 2c3e7ef350b8
> 7b00b0f3356a
> # ceph nvmeof subsystem list
> +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+
> |Nqn |Serial Number |Model Number
> |Namespace Count|Subtype|Max Namespaces|Has Dhchap Key|Allow Any Host|Created
> Without Key|
> +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+
> |nqn.2025-08.com.example:HOSA.hosa|Ceph30844860597194|Ceph bdev Controller|2
> |NVMe |128 |False |True |False
> |
> +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+
> # ceph nvmeof namespace list nqn.2025-08.com.example:HOSA.hosa
> +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+
> |Bdev Name |Rbd Image|Rbd Pool|Lb Group|Rbd
> Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto
> Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn
> |Trash Image|Disable Auto Resize|Read Only|
> +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+
> |bdev_57f2c81a-0b07-406b-84aa-c62d66b5f10d|ns01 |nvmeof |5 |100GB
> |512B |0 |0 |0 |0 |True
> |[] |1
> |57f2c81a-0b07-406b-84aa-c62d66b5f10d|nqn.2025-08.com.example:HOSA.hosa|False
> |False |False |
> |bdev_b4bf328a-8c67-4879-bf13-a799884d6dbe|ns02 |nvmeof |5 |200GB
> |512B |0 |0 |0 |0 |True
> |[] |2
> |b4bf328a-8c67-4879-bf13-a799884d6dbe|nqn.2025-08.com.example:HOSA.hosa|False
> |False |False |
> +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+
> # ceph nvmeof listener list nqn.2025-08.com.example:HOSA.hosa
> +-------------+------+--------------+------+------+-------+
> |Host Name |Trtype|Traddr |Secure|Adrfam|Trsvcid|
> +-------------+------+--------------+------+------+-------+
> |b236asbrce101|TCP |10.105.141.195|False |ipv4 |4420 |
> |b236asbrce102|TCP |10.105.141.196|False |ipv4 |4420 |
> |b236asbrce103|TCP |10.105.141.197|False |ipv4 |4420 |
> |b236asbrce104|TCP |10.105.141.198|False |ipv4 |4420 |
> +-------------+------+--------------+------+------+-------+
>
> On the initiator host:
>
> # nvme discover -t tcp -a 10.105.141.195 -s 8009 -v
> warning: use hostid which does not match uuid in hostnqn
> nvme0: nqn.2014-08.org.nvmexpress.discovery connected
>
> Discovery Log Number of Records 4, Generation counter 6
> =====Discovery Log Entry 0======
> trtype: tcp
> adrfam: ipv4
> subtype: nvme subsystem
> treq: not required
> portid: 0
> trsvcid: 4420
> subnqn: nqn.2025-08.com.example:HOSA.hosa
> traddr: 10.105.141.195
> eflags: none
> sectype: none
> =====Discovery Log Entry 1======
> trtype: tcp
> adrfam: ipv4
> subtype: nvme subsystem
> treq: not required
> portid: 1
> trsvcid: 4420
> subnqn: nqn.2025-08.com.example:HOSA.hosa
> traddr: 10.105.141.196
> eflags: none
> sectype: none
> =====Discovery Log Entry 2======
> trtype: tcp
> adrfam: ipv4
> subtype: nvme subsystem
> treq: not required
> portid: 2
> trsvcid: 4420
> subnqn: nqn.2025-08.com.example:HOSA.hosa
> traddr: 10.105.141.197
> eflags: none
> sectype: none
> =====Discovery Log Entry 3======
> trtype: tcp
> adrfam: ipv4
> subtype: nvme subsystem
> treq: not required
> portid: 3
> trsvcid: 4420
> subnqn: nqn.2025-08.com.example:HOSA.hosa
> traddr: 10.105.141.198
> eflags: none
> sectype: none
> nvme0: nqn.2014-08.org.nvmexpress.discovery disconnected
>
> # nvme connect-all -t tcp -a 10.105.141.195 -s 8009 -v
> warning: use hostid which does not match uuid in hostnqn
> nvme0: nqn.2014-08.org.nvmexpress.discovery connected
> nvme1: nqn.2025-08.com.example:HOSA.hosa connected
> nvme2: nqn.2025-08.com.example:HOSA.hosa connected
> nvme3: nqn.2025-08.com.example:HOSA.hosa connected
> nvme4: nqn.2025-08.com.example:HOSA.hosa connected
> nvme0: nqn.2014-08.org.nvmexpress.discovery disconnected
>
> Looking good so far, but:
>
> # nvme list
> Node Generic SN Model
> Namespace Usage Format
> FW Rev
> --------------------- --------------------- --------------------
> ---------------------------------------- ----------
> -------------------------- ---------------- --------
> # nvme list -v
> Subsystem Subsystem-NQN
> Controllers
> ----------------
> --------------------------------------------------------------------------------------------
> ----------------
> nvme-subsys1 nqn.2025-08.com.example:HOSA.hosa
> nvme1, nvme2, nvme3, nvme4
>
> Device SN MN FR
> TxPort Address Slot Subsystem Namespaces
> -------- -------------------- ----------------------------------------
> -------- ------ -------------- ------ ------------ ----------------
> nvme1 Ceph30844860597194 Ceph bdev Controller 25.05
> tcp traddr=10.105.141.195,trsvcid=4420,src_addr=10.105.141.212
> nvme-subsys1
> nvme2 Ceph30844860597194 Ceph bdev Controller 25.05
> tcp traddr=10.105.141.196,trsvcid=4420,src_addr=10.105.141.212
> nvme-subsys1
> nvme3 Ceph30844860597194 Ceph bdev Controller 25.05
> tcp traddr=10.105.141.197,trsvcid=4420,src_addr=10.105.141.212
> nvme-subsys1
> nvme4 Ceph30844860597194 Ceph bdev Controller 25.05
> tcp traddr=10.105.141.198,trsvcid=4420,src_addr=10.105.141.212
> nvme-subsys1
>
> Device Generic NSID Usage Format
> Controllers
> ------------ ------------ ---------- --------------------------
> ---------------- ----------------
>
> # nvme list-ns /dev/nvme1 -v -a
> opcode : 06
> flags : 00
> rsvd1 : 0000
> nsid : 00000000
> cdw2 : 00000000
> cdw3 : 00000000
> data_len : 00001000
> metadata_len : 00000000
> addr : 616b6af40000
> metadata : 0
> cdw10 : 00000010
> cdw11 : 00000000
> cdw12 : 00000000
> cdw13 : 00000000
> cdw14 : 00000000
> cdw15 : 00000000
> timeout_ms : 00000000
> result : 00000000
> err : 2
> latency : 569 us
> NVMe status: Invalid Field in Command: A reserved coded value or an
> unsupported value in a defined field(0x2)
>
> How do I make it work again?
Hi Robert,
Aviv is going to take a look at this.
Thanks,
Ilya
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
