Hi All,
I thought that I would start a new thread to discuss issues arising when
mach-defpager is not used when operating with high disk usage. This
follows up the discussion at
https://lists.gnu.org/archive/html/bug-hurd/2026-03/msg00057.html
I have a 3GB RAM virtual machine running hurd-amd64 which is being use
to try and replicate the corruptions occurring on the buildd
(specifically boralus). My Linux host manages all disc storage using an
SSD. I have a simple script using sbuild, with a chroot on its own disc
partition, to build a selected 33 packages that match those that were
built by boralus from a boot-up to a point of known file corruption. I
can run this script (neglecting 4 packages that I cannot build using
snapshot.debian.org) whilst using mach-defpager even though I haven't
detected any corruption yet. The boralus buildd is however configured to
run without mach-defpager and generally runs for lengthy periods of time
continuously building packages.
When I run my build script without mach-defpager it locks the system
quite rapidly. I've tried about 10 times now and perhaps 7 times the
system locks within a few minutes whilst performing a large write
unpacking one of the ghc packages (eg. ghc or ghc-prof). The other
occasions run for longer but still result in the same system lock up.
There was one occasion involving an out of memory 'panic'. The latter I
have a fix for to rebalance pages when vm_page_alloc_pa() fails for a
privileged thread. That will be a separate thread once I can get a
system running long enough to test it actually works.
Anyway, when the system locks up, a large package is being installed
creating many dirty pages destined for my chroot ext2fs. pageout is
initiated and threads within all 3 ext2fs servers (/, /boot and the
chroot) are blocked waiting free memory. I've attached the relevant
kernel debugger output to illustrate the detail but in summary the cause
is related to the chroot ext2fs process ($task25):
1) thread 21 waits for memory in vm_fault and so has $map25 locked
2) threads 24, 25, 26 and 27 waiting memory
3) all other threads shown waiting to lock $map25
The vm_page_laundry_count is 5 so no further pageout requests are made
and $task25 seems unable to handle the pageout (presumably because of
the blocked threads). Consequently no progress is made to free memory
and the system locks.
Is it possible for pageout to identify pages which are executable ?
Perhaps it might help to continue searching for 'non-executable,
read-only' pages which can be trivially reclaimed under these circumstances.
I don't think any of the above is really a surprise but what is
surprising to me is how boralus manages to sustain lengthy operation
without this issue arising? Note that I have run the same virtual
machine on a different Linux host which has traditional hard discs which
runs more successfully but has also locked several times with the same
scenario.
Why does boralus buildd run without mach-defpager?
I'm trying to repeat precisely what boralus is doing in order to
replicate the corruption. I can only reliably build large packages
whilst using the default pager so would it be possible to enable the
default pager on boralus and see if file corruption then occurs?
All the best,
Mike.
db> print vm_page_alloc_paused
ffffffff810a4604
db> print $map25
ffffffffdf9eb938
db> show task $task4
4 ext2fs (ffffffffde3138f8): 28 threads:
0 diskfs (ffffffffdf4f28d0) .W..NF. 0xffffffff810a4604
23 sync (ffffffffdf975b18) .W..NF. 0xffffffff810a4604
db> show task $task24
24 /hurd/ext2fs(480) (ffffffffdf9f1678): 25 threads:
1 diskfs (ffffffffdfe9f458) .W..NF. 0xffffffff810a4604
24 sync (ffffffffde3ad020) .W..NF. 0xffffffff810a4604
db> show task $task25
25 /hurd/ext2fs(481) (ffffffffdf9f13e8): 28 threads:
13 paging_requests (ffffffffde389030) .W..NF. 0xffffffffdf9eb938
14 (ffffffffde3f6d28) .W..NF. 0xffffffffdf9eb938
15 (ffffffffde3f6b00) .W..NF. 0xffffffffdf9eb938
16 (ffffffffde3f68d8) .W..NF. 0xffffffffdf9eb938
17 (ffffffffde3f66b0) .W..NF. 0xffffffffdf9eb938
18 (ffffffffde3f6488) .W..NF. 0xffffffffdf9eb938
19 (ffffffffde3f6260) .W..NF. 0xffffffffdf9eb938
20 (ffffffffde3f6038) .W..NF. 0xffffffffdf9eb938
21 (ffffffffde3f3d30) .W..NF. 0xffffffff810a4604
22 (ffffffffde3f3b08) .W..NF. 0xffffffffdf9eb938
23 (ffffffffde3f38e0) .W..NF. 0xffffffffdf9eb938
24 sync (ffffffffdff5fac8) .W..NF. 0xffffffff810a4604
25 (ffffffffdf8466c0) .W..NF. 0xffffffff810a4604
26 (ffffffffdfb3d8f8) .W..NF. 0xffffffff810a4604
27 (ffffffffdfb3d6d0) .W..NF. 0xffffffff810a4604
db> trace $task25.21
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
vm_page_wait(...)+0x57
kmem_cache_alloc(...)+0x147
pmap_enter(...)+0x2cb
vm_fault(...)+0x1e6
user_trap(ffffffffde3f58a8)+0x237
>>>>> Page fault (14) for U 3c01188e at 0x1be46c <<<<<
>>>>> user space <<<<<
db> trace /t $task25.13
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
thread_sleep(...)+0x12
lock_write(...)+0x3a
vm_map_lock(...)+0x11
vm_map_find_entry_anywhere(...)+0x268
vm_map_enter(...)+0xea
vm_map_copyout(...)+0x2dd
ipc_kmsg_copyout_body(...)+0x70
ipc_kmsg_copyout(...)+0x51
mach_msg_receive_continue(...)+0x196
thread_continue(...)+0x25
t_zero_div()
db> trace /t $task25.14
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
thread_sleep(...)+0x12
lock_write(...)+0x74
vm_map_lock(...)+0x11
vm_map_remove(...)+0x1b
vm_deallocate(...)+0x2f
syscall_vm_deallocate(...)+0x2e
syscall64(...)+0xe3
db> trace /t $task25.15
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
thread_sleep(...)+0x12
lock_write(...)+0x3a
vm_map_lock(...)+0x11
vm_map_copyin_page_list(...)+0xcc
ipc_kmsg_copyin(...)+0x89c
mach_msg_trap(...)+0x150
syscall64(...)+0xe3
db> trace /t $task25.16
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
thread_sleep(...)+0x12
lock_write(...)+0x3a
vm_map_lock(...)+0x11
vm_map_copyin_page_list(...)+0xcc
ipc_kmsg_copyin(...)+0x89c
mach_msg_trap(...)+0x150
syscall64(...)+0xe3
db> trace /t $task25.17
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
thread_sleep(...)+0x12
lock_write(...)+0x3a
vm_map_lock(...)+0x11
vm_map_copyin_page_list(...)+0xcc
ipc_kmsg_copyin(...)+0x89c
mach_msg_trap(...)+0x150
syscall64(...)+0xe3
db> trace /t $task25.18
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
thread_sleep(...)+0x12
lock_write(...)+0x3a
vm_map_lock(...)+0x11
vm_map_copyin_page_list(...)+0xcc
ipc_kmsg_copyin(...)+0x89c
mach_msg_trap(...)+0x150
syscall64(...)+0xe3
db> trace /t $task25.19
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
thread_sleep(...)+0x12
lock_write(...)+0x3a
vm_map_lock(...)+0x11
vm_map_copyin_page_list(...)+0xcc
ipc_kmsg_copyin(...)+0x89c
mach_msg_trap(...)+0x150
syscall64(...)+0xe3
db> trace /t $task25.20
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
thread_sleep(...)+0x12
lock_read(...)+0x36
vm_map_lookup(...)+0x4b
vm_fault(...)+0x4fe
kernel_trap(ffffffff92e2ae78)+0xb4
>>>>> Page fault (14) for W 24007b50 at copyout+0x1c <<<<<
copyout(...)+0x1c
mach_msg_continue(...)+0x119
db> trace /t $task25.21
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
vm_page_wait(...)+0x57
kmem_cache_alloc(...)+0x147
pmap_enter(...)+0x2cb
vm_fault(...)+0x1e6
user_trap(ffffffffde3f58a8)+0x237
>>>>> Page fault (14) for U 3c01188e at 0x1be46c <<<<<
>>>>> user space <<<<<
db> trace /t $task25.22
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
thread_sleep(...)+0x12
lock_write(...)+0x3a
vm_map_lock(...)+0x11
vm_map_copyin_page_list(...)+0xcc
ipc_kmsg_copyin(...)+0x89c
mach_msg_trap(...)+0x150
syscall64(...)+0xe3
db> trace /t $task25.23
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
thread_sleep(...)+0x12
lock_write(...)+0x3a
vm_map_lock(...)+0x11
vm_map_copyin_page_list(...)+0xcc
ipc_kmsg_copyin(...)+0x89c
mach_msg_trap(...)+0x150
syscall64(...)+0xe3
db> trace /t $task25.24
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
vm_page_wait(...)+0x57
vm_pageout_setup(...)+0x107
memory_object_lock_request(...)+0x278
_Xmemory_object_lock_request(...)+0x102
ipc_kobject_server(...)+0xac
ipc_mqueue_send(...)+0x1a1
mach_msg_trap(...)+0x909
syscall64(...)+0xe3
db> trace /t $task25.25
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
vm_page_wait(...)+0x57
kmem_cache_alloc(...)+0x147
vm_fault(...)+0x477
user_trap(ffffffffde3da4d8)+0x237
>>>>> Page fault (14) for U f7cac at 0xf7cac <<<<<
>>>>> user space <<<<<
db> trace /t $task25.26
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
vm_page_wait(...)+0x57
kmem_cache_alloc(...)+0x147
vm_fault(...)+0x477
user_trap(ffffffffdfa8e4d8)+0x237
>>>>> Page fault (14) for U f7cac at 0xf7cac <<<<<
>>>>> user space <<<<<
db> trace /t $task25.27
switch_context(...)+0xb5
thread_invoke(...)+0xe3
thread_block(...)+0x4b
vm_page_wait(...)+0x57
kmem_cache_alloc(...)+0x147
vm_fault(...)+0x477
user_trap(ffffffffdfa8e2e8)+0x237
>>>>> Page fault (14) for U f7cac at 0xf7cac <<<<<
>>>>> user space <<<<<
