On Thu, Mar 19, 2026 at 01:14:56AM +0000, Smita Koralahalli wrote:
> From: Dan Williams <[email protected]>
>
> Move hmem/ earlier in the dax Makefile so that hmem_init() runs before
> dax_cxl.
>
> In addition, defer registration of the dax_cxl driver to a workqueue
> instead of using module_cxl_driver(). This ensures that dax_hmem has
> an opportunity to initialize and register its deferred callback and make
> ownership decisions before dax_cxl begins probing and claiming Soft
> Reserved ranges.
>
> Mark the dax_cxl driver as PROBE_PREFER_ASYNCHRONOUS so its probe runs
> out of line from other synchronous probing avoiding ordering
> dependencies while coordinating ownership decisions with dax_hmem.
Hi Smita,
Replying to this patch, as it's my best guess as to why I may be
seeing this WARN when I modprobe cxl-test.
We are able to pass all the CXL unit tests because it is only that
first load that causes the WARN. All subsequent reloads of cxl-test
do not unload dax_cxl and dax_hmem so they chug happily along.
I can reproduce by unloading each piece before reloading cxl-test
# modprobe -r cxl-test
# modprobe -r dax_cxl
# modprobe -r dax_hmem
# modprobe cxl-test
and the WARN repeats.
Guessing you may recognize what is going on. Let me know if I can
try anything else out.
# dmesg (trimmed to just the init calls)
[ 34.229033] calling fwctl_init+0x0/0xff0 [fwctl] @ 1057
[ 34.230616] initcall fwctl_init+0x0/0xff0 [fwctl] returned 0 after 186 usecs
[ 34.257096] calling cxl_core_init+0x0/0x100 [cxl_core] @ 1057
[ 34.258395] initcall cxl_core_init+0x0/0x100 [cxl_core] returned 0 after 538
usecs
[ 34.264170] calling cxl_port_init+0x0/0xff0 [cxl_port] @ 1057
[ 34.264982] initcall cxl_port_init+0x0/0xff0 [cxl_port] returned 0 after 110
usecs
[ 34.268058] calling cxl_mem_driver_init+0x0/0xff0 [cxl_mem] @ 1057
[ 34.268743] initcall cxl_mem_driver_init+0x0/0xff0 [cxl_mem] returned 0
after 110 usecs
[ 34.274670] calling cxl_pmem_init+0x0/0xff0 [cxl_pmem] @ 1057
[ 34.277835] initcall cxl_pmem_init+0x0/0xff0 [cxl_pmem] returned 0 after
1671 usecs
[ 34.285807] calling cxl_acpi_init+0x0/0xff0 [cxl_acpi] @ 1057
[ 34.287105] initcall cxl_acpi_init+0x0/0xff0 [cxl_acpi] returned 0 after 262
usecs
[ 34.292967] calling cxl_test_init+0x0/0xff0 [cxl_test] @ 1057
[ 34.339841] initcall cxl_test_init+0x0/0xff0 [cxl_test] returned 0 after
45832 usecs
[ 34.342259] calling cxl_mock_mem_driver_init+0x0/0xff0 [cxl_mock_mem] @ 1063
[ 34.343459] initcall cxl_mock_mem_driver_init+0x0/0xff0 [cxl_mock_mem]
returned 0 after 356 usecs
[ 34.658602] calling dax_hmem_init+0x0/0xff0 [dax_hmem] @ 1059
[ 34.670106] calling cxl_pci_driver_init+0x0/0xff0 [cxl_pci] @ 1100
[ 34.671023] initcall cxl_pci_driver_init+0x0/0xff0 [cxl_pci] returned 0
after 197 usecs
[ 34.673051] initcall dax_hmem_init+0x0/0xff0 [dax_hmem] returned 0 after
2225 usecs
[ 34.676011] calling cxl_dax_region_init+0x0/0xff0 [dax_cxl] @ 1059
[ 34.676856] ------------[ cut here ]------------
[ 34.677533] WARNING: kernel/workqueue.c:4289 at __flush_work+0x4f9/0x550,
CPU#3: kworker/3:2/136
[ 34.678596] Modules linked in: dax_cxl(+) cxl_pci dax_hmem cxl_mock_mem(O)
cxl_test(O) cxl_acpi(O) cxl_pmem(O) cxl_mem(O) cxl_port(O) cxl_mock(O)
cxl_core(O) fwctl nd_pmem nd_btt dax_pmem nfit nd_e820 libnvdimm
[ 34.680632] initcall cxl_dax_region_init+0x0/0xff0 [dax_cxl] returned 0
after 3842 usecs
[ 34.680918] CPU: 3 UID: 0 PID: 136 Comm: kworker/3:2 Tainted: G O
7.0.0-rc4+ #156 PREEMPT(full)
[ 34.684368] Tainted: [O]=OOT_MODULE
[ 34.684993] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0
02/06/2015
[ 34.686098] Workqueue: events_long cxl_dax_region_driver_register [dax_cxl]
[ 34.687108] RIP: 0010:__flush_work+0x4f9/0x550
That addr is this line in flush_work()
if (WARN_ON(!work->func))
return false;
[ 34.687811] Code: ff 49 8b 45 00 49 8b 55 08 89 c7 48 c1 e8 04 83 e7 08 83
e0 0f 83 cf 02 49 0f ba 6d 00 03 e9 a1 fc ff ff 0f 0b e9 e6 fe ff ff <0f> 0b e9
df fe ff ff e8 9b 48 15 01 85 c0 0f 84 26 ff ff ff 80 3d
[ 34.690107] RSP: 0018:ffffc900020b7cf8 EFLAGS: 00010246
[ 34.690673] RAX: 0000000000000000 RBX: ffffffffa0ea2088 RCX: ffff8880088b2b78
[ 34.691388] RDX: 00000000834fb194 RSI: 0000000000000000 RDI: ffffffffa0ea2088
[ 34.692135] RBP: ffffc900020b7de0 R08: 0000000031ab93b0 R09: 00000000effb42e8
[ 34.692876] R10: 000000008effb42e R11: 0000000000000000 R12: ffff88807d9bb340
[ 34.693588] R13: ffffffffa0ea2088 R14: ffffffffa0ed2020 R15: 0000000000000001
[ 34.694358] FS: 0000000000000000(0000) GS:ffff8880fa45f000(0000)
knlGS:0000000000000000
[ 34.695179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 34.695775] CR2: 00007fe888b4e34c CR3: 00000000090ed004 CR4: 0000000000370ef0
[ 34.696494] Call Trace:
[ 34.696889] <TASK>
[ 34.697238] ? __lock_acquire+0xb08/0x2930
[ 34.697730] ? __this_cpu_preempt_check+0x13/0x20
[ 34.698277] flush_work+0x17/0x30
[ 34.698705] dax_hmem_flush_work+0x10/0x20 [dax_hmem]
[ 34.699270] cxl_dax_region_driver_register+0x9/0x30 [dax_cxl]
[ 34.699943] process_one_work+0x203/0x6c0
[ 34.700452] worker_thread+0x197/0x350
[ 34.700942] ? __pfx_worker_thread+0x10/0x10
[ 34.701455] kthread+0x108/0x140
[ 34.701915] ? __pfx_kthread+0x10/0x10
[ 34.702396] ret_from_fork+0x28a/0x310
[ 34.702880] ? __pfx_kthread+0x10/0x10
[ 34.703363] ret_from_fork_asm+0x1a/0x30
[ 34.703872] </TASK>
[ 34.704227] irq event stamp: 11015
[ 34.704656] hardirqs last enabled at (11025): [<ffffffff813486de>]
__up_console_sem+0x5e/0x80
[ 34.705493] hardirqs last disabled at (11036): [<ffffffff813486c3>]
__up_console_sem+0x43/0x80
[ 34.706354] softirqs last enabled at (10500): [<ffffffff812ab9f3>]
__irq_exit_rcu+0xc3/0x120
[ 34.707197] softirqs last disabled at (10495): [<ffffffff812ab9f3>]
__irq_exit_rcu+0xc3/0x120
[ 34.708015] ---[ end trace 0000000000000000 ]---
[ 34.752127] calling dax_init+0x0/0xff0 [device_dax] @ 1089
[ 34.754006] initcall dax_init+0x0/0xff0 [device_dax] returned 0 after 422
usecs
[ 34.759609] calling dax_kmem_init+0x0/0xff0 [kmem] @ 1089
[ 37.338377] initcall dax_kmem_init+0x0/0xff0 [kmem] returned 0 after 2577658
usecs
>
> Signed-off-by: Dan Williams <[email protected]>
> Signed-off-by: Smita Koralahalli <[email protected]>
> Reviewed-by: Dave Jiang <[email protected]>
> Reviewed-by: Jonathan Cameron <[email protected]>
> ---
> drivers/dax/Makefile | 3 +--
> drivers/dax/cxl.c | 27 ++++++++++++++++++++++++++-
> 2 files changed, 27 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/dax/Makefile b/drivers/dax/Makefile
> index 5ed5c39857c8..70e996bf1526 100644
> --- a/drivers/dax/Makefile
> +++ b/drivers/dax/Makefile
> @@ -1,4 +1,5 @@
> # SPDX-License-Identifier: GPL-2.0
> +obj-y += hmem/
> obj-$(CONFIG_DAX) += dax.o
> obj-$(CONFIG_DEV_DAX) += device_dax.o
> obj-$(CONFIG_DEV_DAX_KMEM) += kmem.o
> @@ -10,5 +11,3 @@ dax-y += bus.o
> device_dax-y := device.o
> dax_pmem-y := pmem.o
> dax_cxl-y := cxl.o
> -
> -obj-y += hmem/
> diff --git a/drivers/dax/cxl.c b/drivers/dax/cxl.c
> index 13cd94d32ff7..a2136adfa186 100644
> --- a/drivers/dax/cxl.c
> +++ b/drivers/dax/cxl.c
> @@ -38,10 +38,35 @@ static struct cxl_driver cxl_dax_region_driver = {
> .id = CXL_DEVICE_DAX_REGION,
> .drv = {
> .suppress_bind_attrs = true,
> + .probe_type = PROBE_PREFER_ASYNCHRONOUS,
> },
> };
>
> -module_cxl_driver(cxl_dax_region_driver);
> +static void cxl_dax_region_driver_register(struct work_struct *work)
> +{
> + cxl_driver_register(&cxl_dax_region_driver);
> +}
> +
> +static DECLARE_WORK(cxl_dax_region_driver_work,
> cxl_dax_region_driver_register);
> +
> +static int __init cxl_dax_region_init(void)
> +{
> + /*
> + * Need to resolve a race with dax_hmem wanting to drive regions
> + * instead of CXL
> + */
> + queue_work(system_long_wq, &cxl_dax_region_driver_work);
> + return 0;
> +}
> +module_init(cxl_dax_region_init);
> +
> +static void __exit cxl_dax_region_exit(void)
> +{
> + flush_work(&cxl_dax_region_driver_work);
> + cxl_driver_unregister(&cxl_dax_region_driver);
> +}
> +module_exit(cxl_dax_region_exit);
> +
> MODULE_ALIAS_CXL(CXL_DEVICE_DAX_REGION);
> MODULE_DESCRIPTION("CXL DAX: direct access to CXL regions");
> MODULE_LICENSE("GPL");
> --
> 2.17.1
>
>
