On 7/13/26 9:28 PM, Muchun Song wrote: > > > On 2026/7/9 19:25, Li Zhe wrote: >> memmap_init_zone_device() repeats nearly identical head-page >> initialization for each PFN. Prepare one reusable ZONE_DEVICE head-page >> template through the existing slow path, refresh the PFN-dependent >> fields in that template before each copy, and memcpy it into each >> destination page. >> >> The optimized path assigns _refcount through the copied template, so >> keep it disabled when the page_ref_set tracepoint is enabled. >> >> This patch accelerates head-page initialization. The pfns_per_compound >> == 1 case gets the full benefit here, compound tails are handled in the >> next patch. >> >> Tested in a VM with a 100 GB fsdax namespace device configured with >> map=dev on Intel Ice Lake server. This test exercises the nd_pmem rebind >> path (pfns_per_compound == 1). >> >> Test procedure: >> Rebind the nd_pmem driver 30 times and collect the memmap initialization >> time from the pr_debug() output of memmap_init_zone_device(). >> >> Base(v7.2-rc1): >> First binding: 1456 ms >> Average of subsequent rebinds: 244.28 ms >> >> With this patch and its prerequisites applied: >> First binding: 1440 ms >> Average of subsequent rebinds: 217.19 ms >> >> This reduces the average rebind time from 244.28 ms to 217.19 ms, or >> about 11%. > > An 11% reduction is definitely a solid achievement. While the > absolute gain is in the range of a few dozen milliseconds, I’d > love to better understand the compounding impact. In which > specific latency-sensitive scenarios or workloads does saving > these few dozen milliseconds translate into a major breakthrough > for the user experience? > > Muchun, > Thanks.
Patch 4 by itself is only an incremental step, not the full result of the series. The target scenario here is large fsdax/devdax pmem bind/rebind latency. This patch removes the repeated head-page initialization overhead on the pfns_per_compound == 1 path, so the gain at this stage is only a few dozen milliseconds. The later patches extend the same idea to compound tails and then switch the template-copy path over to memcpy_nt(), which is where the larger reduction reported for the full series comes from. Even this incremental reduction still helps shorten the control-path latency for large namespace bind/rebind operations. Thanks, Zhe > >> >> Signed-off-by: Li Zhe <[email protected]> >> --- >> mm/mm_init.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++- >> 1 file changed, 58 insertions(+), 1 deletion(-) >> >> diff --git a/mm/mm_init.c b/mm/mm_init.c >> index 5fccfbacf855..170021e182e0 100644 >> --- a/mm/mm_init.c >> +++ b/mm/mm_init.c >> @@ -1065,6 +1065,44 @@ static void __ref >> zone_device_page_init_slow(struct page *page, >> set_page_count(page, 0); >> } >> +static inline bool zone_device_page_init_optimization_enabled(void) >> +{ >> + /* >> + * The template fast path copies a preinitialized struct page >> image. >> + * Skip it when the page_ref_set tracepoint is enabled. >> + */ >> + return !page_ref_tracepoint_active(page_ref_set); >> +} >> + >> +/* >> + * 'template' is a reusable page prototype rather than a strictly >> immutable >> + * object. Most ZONE_DEVICE fields stay constant across the pages >> covered by >> + * the current template, but section bits and page->virtual may >> still depend >> + * on the PFN. Refresh those PFN-dependent fields in the template >> before >> + * copying it into @page. >> + */ >> +static inline void zone_device_page_update_template(struct page >> *template, >> + unsigned long pfn) >> +{ >> + set_page_section_from_pfn(template, pfn); >> +#ifdef WANT_PAGE_VIRTUAL >> + if (!is_highmem_idx(ZONE_DEVICE)) >> + set_page_address(template, __va(pfn << PAGE_SHIFT)); >> +#endif >> +} >> + >> +static void zone_device_page_init_from_template(struct page *page, >> + unsigned long pfn, struct page *template) >> +{ >> + /* >> + * 'template' carries the invariant portion of a ZONE_DEVICE struct >> + * page. Update the PFN-dependent fields in place before copying it >> + * to the destination page. >> + */ >> + zone_device_page_update_template(template, pfn); >> + memcpy(page, template, sizeof(*page)); >> +} >> + >> /* >> * With compound page geometry and when struct pages are stored in >> ram most >> * tail pages are reused. Consequently, the amount of unique struct >> pages to >> @@ -1120,6 +1158,7 @@ void __ref memmap_init_zone_device(struct zone >> *zone, >> unsigned long nr_pages, >> struct dev_pagemap *pgmap) >> { >> + bool use_template = zone_device_page_init_optimization_enabled(); >> unsigned long pfn, end_pfn = start_pfn + nr_pages; >> struct pglist_data *pgdat = zone->zone_pgdat; >> struct vmem_altmap *altmap = pgmap_altmap(pgmap); >> @@ -1127,6 +1166,7 @@ void __ref memmap_init_zone_device(struct zone >> *zone, >> unsigned long zone_idx = zone_idx(zone); >> unsigned long start = jiffies; >> int nid = pgdat->node_id; >> + struct page template; >> if (WARN_ON_ONCE(!pgmap || zone_idx != ZONE_DEVICE)) >> return; >> @@ -1144,7 +1184,24 @@ void __ref memmap_init_zone_device(struct zone >> *zone, >> for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) { >> struct page *page = pfn_to_page(pfn); >> - zone_device_page_init_slow(page, pfn, zone_idx, nid, pgmap); >> + if (!use_template) { >> + zone_device_page_init_slow(page, pfn, zone_idx, >> + nid, pgmap); >> + } else if (pfn == start_pfn) { >> + /* >> + * Seed the reusable head-page template from the >> + * first real struct page, because the existing >> + * page-init and pageblock helpers expect a real >> + * memmap entry rather than a stack object. >> + */ >> + zone_device_page_init_slow(page, pfn, zone_idx, >> + nid, pgmap); >> + /* init template page */ >> + memcpy(&template, page, sizeof(*page)); >> + } else { >> + zone_device_page_init_from_template(page, pfn, >> + &template); >> + } >> >> if (IS_ALIGNED(pfn, PAGES_PER_SECTION)) >> cond_resched(); >> -- >> 2.20.1 >
