Hi,
a DMA issue could also explain why f3 reports overwritten sectors.
Below are nvme outputs of the Lexar SSD in 1st M.2 socket (PCIe Gen 5)
under Debian 6.1.112 kernel. (Kingston SSD is not installed at the
moment). MDTS size is 512 k (2 ^ 7 * 4096).
Is there a way to overwrite the mdts setting (to test whether the
reported value is wrong)?
Is there a way to enforce the PCie mode to Gen 4 ?
Regards Stefan
--------------
$ LANG=C /usr/sbin/nvme show-regs -H /dev/nvme0
get-property: Invalid argument
cap : 7fd6df0d4cc0
Controller Ready With Media Support (CRWMS): Not Supported
Controller Ready Independent of Media Support (CRIMS): Not Supported
Controller Memory Buffer Supported (CMBS): The Controller Memory Buffer
is Not Supported
Persistent Memory Region Supported (PMRS): The Persistent Memory Region
is Not Supported
Memory Page Size Maximum (MPSMAX): 4096 bytes
Memory Page Size Minimum (MPSMIN): 4096 bytes
Boot Partition Support (BPS): Yes
Command Sets Supported (CSS): NVM command set is Not
Supported
One or more I/O Command Sets
are Supported
Only Admin Command Set
Supported
NVM Subsystem Reset Supported (NSSRS): Yes
Doorbell Stride (DSTRD): 256 bytes
Timeout (TO): 111500 ms
Arbitration Mechanism Supported (AMS): Weighted Round Robin with
Urgent Priority Class is not supported
Contiguous Queues Required (CQR): Yes
Maximum Queue Entries Supported (MQES): 19649
version : df0d4cc0
NVMe specification 57101.76
cc : 0
Controller Ready Independent of Media Enable (CRIME): Disabled
I/O Completion Queue Entry Size (IOCQES): 1 bytes
I/O Submission Queue Entry Size (IOSQES): 1 bytes
Shutdown Notification (SHN): No notification; no effect
Arbitration Mechanism Selected (AMS): Round Robin
Memory Page Size (MPS): 4096 bytes
I/O Command Set Selected (CSS): NVM Command Set
Enable (EN): No
csts : 0
Processing Paused (PP): No
NVM Subsystem Reset Occurred (NSSRO): No
Shutdown Status (SHST): Normal operation (no shutdown has
been requested)
Controller Fatal Status (CFS): False
Ready (RDY): No
double free or corruption (!prev)
--------------
$ /usr/sbin/nvme id-ctrl /dev/nvme0
NVME Identify Controller:
vid : 0x1d97
ssvid : 0x1d97
sn : NHG456R005482P2202
mn : Lexar SSD NM790 4TB
fr : 12237
rab : 0
ieee : caf25b
cmic : 0
mdts : 7
cntlid : 0
ver : 0x20000
rtd3r : 0x7a120
rtd3e : 0x1e8480
oaes : 0x200
ctratt : 0x12
rrls : 0
cntrltype : 1
fguid : 00000000-0000-0000-0000-000000000000
crdt1 : 0
crdt2 : 0
crdt3 : 0
nvmsr : 0
vwci : 0
mec : 0
oacs : 0x17
acl : 2
aerl : 3
frmw : 0x14
lpa : 0xa
elpe : 63
npss : 4
avscc : 0x1
apsta : 0x1
wctemp : 363
cctemp : 368
mtfa : 8000
hmpre : 10240
hmmin : 10240
tnvmcap : 4096805658624
unvmcap : 0
rpmbs : 0x6000002
edstt : 30
dsto : 1
fwug : 1
kas : 0
hctma : 0x1
mntmt : 373
mxtmt : 388
sanicap : 0x40000002
hmminds : 320
hmmaxd : 32
nsetidmax : 0
endgidmax : 1
anatt : 0
anacap : 0
anagrpmax : 0
nanagrpid : 0
pels : 0
domainid : 0
megcap : 0
sqes : 0x66
cqes : 0x44
maxcmd : 0
nn : 1
oncs : 0x5f
fuses : 0
fna : 0
vwc : 0x7
awun : 0
awupf : 0
icsvscc : 0
nwpc : 0
acwu : 0
ocfs : 0
sgls : 0
mnan : 0
maxdna : 0
maxcna : 0
subnqn : nqn.2014-08.com.maxio:nvme:1602:M.2:NHG456R005482P2202
ioccsz : 0
iorcsz : 0
icdoff : 0
fcatt : 0
msdbd : 0
ofcs : 0
ps 0 : mp:6.50W operational enlat:0 exlat:0 rrt:0 rrl:0
rwt:0 rwl:0 idle_power:- active_power:-
active_power_workload:-
ps 1 : mp:5.80W operational enlat:0 exlat:0 rrt:1 rrl:1
rwt:1 rwl:1 idle_power:- active_power:-
active_power_workload:-
ps 2 : mp:3.60W operational enlat:0 exlat:0 rrt:2 rrl:2
rwt:2 rwl:2 idle_power:- active_power:-
active_power_workload:-
ps 3 : mp:0.0500W non-operational enlat:5000 exlat:10000 rrt:3 rrl:3
rwt:3 rwl:3 idle_power:- active_power:-
active_power_workload:-
ps 4 : mp:0.0025W non-operational enlat:8000 exlat:41000 rrt:4 rrl:4
rwt:4 rwl:4 idle_power:- active_power:-
active_power_workload:-
--------
Regards Stefan
Am 18.12.24 um 16:08 schrieb Tj:
Package: linux-image-6.12.1+debian+tj
Followup-For: Bug #1076372
X-Debbugs-Cc: tj.iam...@proton.me
I've done some investigative work with the logs and the lspci reports in
an attempt to narrow down potential causes and identify relevant clues.
1. Comparison of the Lexar NM790 PCIe configuration between M2 slot 1 and
slot 2 does not show any signficant difference:
$ diff -b -u "Slot "{1,2}" Lexar NM790.lspci.txt"
--- "Slot 1 Lexar NM790.lspci.txt" 2024-12-18 13:59:02.675502501 +0000
+++ "Slot 2 Lexar NM790.lspci.txt" 2024-12-18 14:08:42.770250485 +0000
@@ -1,12 +1,11 @@
-02:00.0 Non-Volatile memory controller: Shenzhen Longsys Electronics Co., Ltd.
Lexar NM790 NVME SSD (DRAM-less) (rev 01) (prog-if 02 [NVM Express])
+03:00.0 Non-Volatile memory controller: Shenzhen Longsys Electronics Co., Ltd.
Lexar NM790 NVME SSD (DRAM-less) (rev 01) (prog-if 02 [NVM Express])
Subsystem: Shenzhen Longsys Electronics Co., Ltd. Lexar NM790 NVME SSD
(DRAM-less)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping-
SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort-
>SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
- Interrupt: pin A routed to IRQ 41
- NUMA node: 0
- IOMMU group: 14
- Region 0: Memory at f6e00000 (64-bit, non-prefetchable) [size=16K]
+ Interrupt: pin A routed to IRQ 39
+ IOMMU group: 16
+ Region 0: Memory at f6c00000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
2. The common factor seems to be M2 Slot 1 that, as I noted previously,
is specified as Gen5x4 whereas slot 2 is Gen4x4.
3. Corrupted blocksize is reported as likely to be 128KiB. I examined
all 332 commits between v6.3.5..v6.3.7 and one in particular stands out
due to mentioning clamping NVME sysfs queue max_hw_sectors to this value:
Refs: v6.3-rc2-139-g3710e2b056cb9
Commit: 3710e2b056cb92ad816e4d79fa54a6a5b6ad8cbd
Title: nvme-pci: clamp max_hw_sectors based on DMA optimized limitation
There are recent follow-up discussions related to this commit:
"nvme-sysfs: display max_hw_sectors_kb without requiring namespaces"
https://lkml.org/lkml/2024/10/16/1721
The "DMA optimized limitation" refers to
Refs: v4.7-rc1~31^2~1
Commit: 9257b4a206fc0229dd5f84b78e4d1ebf3f91d270
Title: iommu/iova: introduce per-cpu caching to iova allocation
Where this is related to I/O Virtual Address space range caching and
says:
"To keep the cache size reasonable, we bound the IOVA space a CPU can
cache by 32 MiB (we cache a bounded number of IOVA ranges, and only
ranges of size <= 128 KiB). The shared global cache is bounded at
4 MiB of IOVA space."
The nvmi-pci commit mentions the NVME MDTS value (Maximum Data Transfer Size)
that is the power-of-2 multiplier of the MPSMIN (Memory Page Size
Minimum).
It is worth checking what the two devices support. It could be that
combined with the Gen5x4 slot 1 of this motherboard there is some incompatiblity
for certain sizes here.
Example:
$ sudo /usr/sbin/nvme show-regs -H /dev/nvme0 | grep -a MPSMIN
get-property: Invalid argument
Memory Page Size Minimum (MPSMIN): 4096 bytes
$ sudo /usr/sbin/nvme id-ctrl /dev/nvme0 | grep -a mdts
mdts : 5
$ echo $(( 2 ** 5 ))
32
$ echo $(( 2 ** 5 * 4096 ))
131072
So here the device uses 128KiB.
