On 15/05/2024 1:39 pm, Jan Beulich wrote: > On 15.05.2024 13:58, Andrew Cooper wrote: >> Just so it doesn't get lost. In XenRT, we've found: >> >>> (XEN) ----[ Xen-4.19.0-1-d x86_64 debug=y Tainted: H ]---- >>> (XEN) CPU: 45 >>> (XEN) RIP: e008:[<ffff82d040244cbf>] >>> common/sched/credit.c#csched_load_balance+0x41/0x877 >>> (XEN) RFLAGS: 0000000000010092 CONTEXT: hypervisor >>> (XEN) rax: ffff82d040981618 rbx: ffff82d040981618 rcx: >>> 0000000000000000 >>> (XEN) rdx: 0000003ff68cd000 rsi: 000000000000002d rdi: >>> ffff83103723d450 >>> (XEN) rbp: ffff83207caa7d48 rsp: ffff83207caa7b98 r8: >>> 0000000000000000 >>> (XEN) r9: ffff831037253cf0 r10: ffff83103767c3f0 r11: >>> 0000000000000009 >>> (XEN) r12: ffff831037237990 r13: ffff831037237990 r14: >>> ffff831037253720 >>> (XEN) r15: 0000000000000000 cr0: 000000008005003b cr4: >>> 0000000000f526e0 >>> (XEN) cr3: 000000005bc2f000 cr2: 0000000000000010 >>> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: >>> 0000000000000000 >>> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008 >>> (XEN) Xen code around <ffff82d040244cbf> >>> (common/sched/credit.c#csched_load_balance+0x41/0x877): >>> (XEN) 48 8b 0c 10 48 8b 49 08 <48> 8b 79 10 48 89 bd b8 fe ff ff 49 >>> 8b 4e 28 48 >>> <snip> >>> (XEN) Xen call trace: >>> (XEN) [<ffff82d040244cbf>] R >>> common/sched/credit.c#csched_load_balance+0x41/0x877 > While this is of course pretty little information, I've still tried to > decipher it, first noticing it's credit1 that's being used here. Once > forcing csched_load_balance() non-inline (no idea why it is a separate > function in your build), I can see a sufficiently matching pattern at > approximately the same offset into the function. That's > > const struct cpupool *c = get_sched_res(cpu)->cpupool; > ... > const cpumask_t *online = c->res_valid; > ... > BUG_ON(get_sched_res(cpu) != snext->unit->res); > > overlapping, with the crash being on the middle of the quoted lines. > IOW the CPU pool is still NULL for this sched resource. Cc-ing > Jürgen for possible clues ...
We've seen it in 4.13, 4.17 and upstream, after Roger extended the existing CPU hotplug testing to try and reproduce the MTRR watchdog failure. We've found yet another "no irq for handler" from this too. It's always a deference at NULL+0x10, somewhere within csched_schedule(). ---8<--- #!/bin/bash set -e CPUS=$(xl info nr_cpus) while : do for C in $(seq 1 $(($CPUS-1))) do echo "Offline $C" xen-hptool cpu-offline $C echo "Online $C" xen-hptool cpu-online $C done done ---8<--- does rather well at reproducing the crash. ~Andrew
