Reproduced in a VM and used to verify the patch proposed by Viktor Pashaiev.
Reproducer and both boot logs attached. All runs on 7.0.0-30-generic, the kernel
the original crash happened on.
## 1. Mechanism
attr_data_get_block_locked() duplicates the resident attribute:
*res = kmemdup(resident_data(attr_b), data_size, GFP_KERNEL);
The pointer becomes iomap->inline_data, iomap->length becomes data_size.
iomap_write_end_inline() then asserts
iomap->length <= PAGE_SIZE - offset_in_page(iomap->inline_data)
A slab object rarely starts on a page boundary. data_size 129..192 comes from
kmalloc-192: order=1 slab, 42 objects, 192-byte stride. Object #21 starts at
4032, and 4032 + (129..192) = 4161..4224 > 4096. The other 41 objects pass.
## 2. The guest needs >= 16 vCPUs
calculate_order() in mm/slub.c:
nr_cpus = num_present_cpus();
min_objects = 4 * (fls(nr_cpus) + 1);
min_order = max(slub_min_order, get_order(min_objects * size));
size=192, 15 CPUs: fls(15)=4, min_objects=20, 20*192=3840, get_order(3840)=0,
min_order=0. calc_slab_order() at order 0: rem = 4096 % 192 = 64, threshold
slab_size/16 = 256, 64 <= 256 so it stops. order=0: one page, 21 objects,
offsets 0..3840, and 3840 + 192 = 4032 <= 4096. Nothing straddles a page
boundary; the BUG_ON is unreachable.
size=192, 16 CPUs: fls(16)=5, min_objects=24, 24*192=4608, get_order(4608)=1,
min_order=1. order=1: 42 objects, object #21 at 4032.
Threshold is exactly 16 present CPUs. Verify in the guest (mode 0400,
root):
# cat /sys/kernel/slab/kmalloc-192/order -> 1
# cat /sys/kernel/slab/kmalloc-192/objs_per_slab -> 42
Below that, a reproducer reports "not reproducible" on a broken kernel
too.
## 3. Workload
The original crash was a 137-byte write(2) to fd 1, rsync writing its log to an
ntfs3 volume. attr_set_size_res() only goes non-resident once
used + dsize > sbi->max_bytes_per_attr (~700 bytes at a 1024-byte MFT record),
so a small new file stays resident and its first write takes the resident
branch.
Loop: create file, one write() of 129..192 bytes, unlink.
Two constraints, both of which produce zero hits if violated:
* Appending (open(p,'ab')) writes past i_size, goes through ntfs_extend() ->
ntfs_set_size(), and once non-resident kmemdup() is never called again.
* Unrelated kmalloc-192 traffic is required. The buffer lives only from
iomap_begin to iomap_end, so SLUB returns the same object LIFO and object
#21 never surfaces.
Setup: qemu/KVM, 16 vCPUs, 4 GiB RAM, 4 GiB NTFS image, files seeded through
ntfs-3g, then remounted with ntfs3 rw.
## 4. Result on an unmodified kernel
Shipped ntfs3 module, BUG_ON 68.4 s after boot:
kernel BUG at fs/iomap/buffered-io.c:1061!
Oops: invalid opcode: 0000 [#1] SMP NOPTI
CPU: 10 UID: 0 PID: 1788 Comm: python3 Not tainted 7.0.0-30-generic
#30-Ubuntu
RIP: 0010:iomap_write_end+0x1e0/0x1f0
RAX: 0000000000000040 RBX: fffffb5d41097280 RCX: fffffb5d41097280
RDX: 0000000000000fc0 RSI: 00000000000000b8 RDI: ffff8dc4cd852fc0
Call Trace:
iomap_write_iter+0x171/0x340
iomap_file_buffered_write+0xa6/0x110
ntfs_file_write_iter+0x267/0x310 [ntfs3]
vfs_write+0x25b/0x490
ksys_write+0x71/0xf0
__x64_sys_write+0x19/0x30
Against the crash originally reported here:
original (rsync) reproducer
RDX = offset_in_page 0xfc0 = 4032 0xfc0 = 4032
RAX = PAGE_SIZE - off 0x40 = 64 0x40 = 64
write length 0x89 = 137 0xb8 = 184
inline_data low bits ...c55acfc0 ...cd852fc0
Same signature in bugzilla.kernel.org #221446 (Arch, 7.0.2, process `git`):
RDX 0xfc0, RAX 0x40, length 145. Three independent hits on offset 4032, all
lengths inside the kmalloc-192 range.
## 5. Patch verification
Built out of tree from linux-source-7.0.0 against 7.0.0-30-generic headers as
two modules differing only by Viktor's patch, which applied with no offsets and
no fuzz. Confirmed at machine-code level: in attr_data_get_block_locked() the
unpatched build calls kmemdup_noprof, the patched build calls alloc_pages_noprof
plus memcpy.
An unpatched out-of-tree build was run first as a control. Without it, "no panic
with the patch" would not distinguish a working fix from an out-of-tree build
that differs from the shipped module. Each run verified the live module by
comparing /sys/module/ntfs3/srcversion against the expected value.
Same kernel, same image, same workload. Only the module differs.
control patched
live srcversion 99295717... (ok) A80EE935... (ok)
kmalloc-192 geometry order=1 objs=42 order=1 objs=42
run length 60 s 840 s
inline mappings observed 56,013,951 864,253,125
offset_in_page values seen 76 distinct only 0
mappings violating the check 1 0
BUG_ON yes, at 29.4 s none in 840 s
The @DANGER counter and the oops agree on the same mapping. In the control run
the probe fired on the return from ntfs_iomap_begin() with
[DANGER] off=4032 len=160 sum=4192 -- BUG_ON next
and the oops milliseconds later reports RDX=0xfc0 (4032), RSI=0xa0 (160),
RAX=0x40 (4096 - 4032). In boot-vanilla.log that printf is interleaved
character by character with the oops text, since bpftrace and the kernel share
the console. So the counter that reads 0 across 864,253,125 mappings on the
patched build is measuring the exact condition that crashes.
The offset row is the load-bearing one. With the patch every inline mapping was
page aligned, so length <= PAGE_SIZE - 0 holds for any resident size (at most
~700 bytes against 4096). The failure mode is removed structurally, not made
rarer.
The BUG_ON itself is probabilistic, which matters for SRU verification. Four
unpatched runs here fired at 68.4 s, 25.7 s and 29.4 s after boot, and one did
not fire at all in 360 s despite 319,225,379 inline mappings and 85 distinct
offsets in that run. A violating mapping turns up on the order of once per a few
hundred million inline mappings, since it needs the allocation to land in object
#21 of a kmalloc-192 slab specifically. Absence of a BUG_ON in a single run is
therefore not evidence of a fix on its own; the offset distribution is the
reliable signal.
Instrumentation caveat: offset 4032 alone is not a fault indicator. It is also
the last object of a kmalloc-64 slab (63 * 64 = 4032), where 4032 + 64 = 4096
fits exactly. In longer runs most landings on 4032 were harmless for exactly
that reason. The condition to test is
length > PAGE_SIZE - offset_in_page(inline_data), which is what the @DANGER
counter in the attached bpftrace script evaluates.
## 6. Upstream
70d3855594cf6e8791970714b65cac3202d6160e
"ntfs3: Allocate iomap inline_data using alloc_page"
Mihai Brodschi
Fixes: 099ef9ab9203 ("fs/ntfs3: implement iomap-based file operations")
fs/ntfs3/attrib.c and fs/ntfs3/inode.c. Mainline v7.2, stable v7.1.5. Checked
both stable tags: fs/ntfs3/attrib.c in v7.1.4 still has
kmemdup(resident_data(...)), in v7.1.5 it has alloc_page().
099ef9ab9203, the iomap conversion, is what shipped in 7.0. 7.0.0-31.31 is at
"Upstream stable to v6.18.39, v7.1.4".
## Attachments
* ntfs3-2165844-reproducer.tar.gz - workload scripts, bpftrace script, runner,
module build script, README with the recipe and the prerequisites.
* boot-vanilla.log - control run, unpatched (next comment).
* boot-patched.log - patched run (next comment).
** Attachment added: "Reproducer: workload scripts, bpftrace instrumentation,
VM runner, module build script, README"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2165844/+attachment/6000954/+files/ntfs3-2165844-reproducer.tar.gz
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2165844
Title:
linux 7.0.0-30: kernel BUG at fs/iomap/buffered-io.c:1061 in
iomap_write_end() on ntfs3 buffered write
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2165844/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
