On Thu, Nov 20, 2025 at 05:03:36PM +1100, Balbir Singh wrote: > On 11/20/25 16:53, Matthew Brost wrote: > > On Thu, Nov 20, 2025 at 02:58:58PM +1100, Balbir Singh wrote: > >> On 11/20/25 14:15, Matthew Brost wrote: > >>> On Thu, Nov 20, 2025 at 01:59:09PM +1100, Balbir Singh wrote: > >>>> On 11/20/25 13:50, Balbir Singh wrote: > >>>>> On 11/20/25 13:40, Matthew Brost wrote: > >>>>>> On Wed, Nov 12, 2025 at 10:52:43AM +1100, Balbir Singh wrote: > >>>>>>> On 11/12/25 10:43, Andrew Morton wrote: > >>>>>>>> On Thu, 9 Oct 2025 03:33:33 -0700 Matthew Brost > >>>>>>>> <[email protected]> wrote: > >>>>>>>> > >>>>>>>>>>>> This patch series introduces support for Transparent Huge Page > >>>>>>>>>>>> (THP) migration in zone device-private memory. The > >>>>>>>>>>>> implementation enables > >>>>>>>>>>>> efficient migration of large folios between system memory and > >>>>>>>>>>>> device-private memory > >>>>>>>>>>> > >>>>>>>>>>> Lots of chatter for the v6 series, but none for v7. I hope > >>>>>>>>>>> that's a > >>>>>>>>>>> good sign. > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> I hope so too, I've tried to address the comments in v6. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> Circling back to this series, we will itegrate and test this > >>>>>>>>> version. > >>>>>>>> > >>>>>>>> How'd it go? > >>>>>>>> > >>>>>> > >>>>>> My apologies for the delay—I got distracted by other tasks in Xe (my > >>>>>> driver) and was out for a bit. Unfortunately, this series breaks > >>>>>> something in the existing core MM code for the Xe SVM implementation. I > >>>>>> have an extensive test case that hammers on SVM, which fully passes > >>>>>> prior to applying this series, but fails randomly with the series > >>>>>> applied (to drm-tip-rc6) due to the below kernel lockup. > >>>>>> > >>>>>> I've tried to trace where the migration PTE gets installed but not > >>>>>> removed or isolate a test case which causes this failure but no luck so > >>>>>> far. I'll keep digging as I have time. > >>>>>> > >>>>>> Beyond that, if I enable Xe SVM + THP, it seems to mostly work (though > >>>>>> the same issue as above eventually occurs), but I do need two > >>>>>> additional > >>>>>> core MM patches—one is new code required for Xe, and the other could be > >>>>>> considered a bug fix. Those patches can included when Xe merges SVM THP > >>>>>> support but we need at least not break Xe SVM before this series > >>>>>> merges. > >>>>>> > >>>>>> Stack trace: > >>>>>> > >>>>>> INFO: task kworker/u65:2:1642 blocked for more than 30 > >>>>>> seconds. > >>>>>> [ 212.624286] Tainted: G S W 6.18.0-rc6-xe+ #1719 > >>>>>> [ 212.630561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > >>>>>> disables this message. > >>>>>> [ 212.638285] task:kworker/u65:2 state:D stack:0 pid:1642 > >>>>>> tgid:1642 ppid:2 task_flags:0x4208060 flags:0x00080000 > >>>>>> [ 212.638288] Workqueue: xe_page_fault_work_queue > >>>>>> xe_pagefault_queue_work [xe] > >>>>>> [ 212.638323] Call Trace: > >>>>>> [ 212.638324] <TASK> > >>>>>> [ 212.638325] __schedule+0x4b0/0x990 > >>>>>> [ 212.638330] schedule+0x22/0xd0 > >>>>>> [ 212.638331] io_schedule+0x41/0x60 > >>>>>> [ 212.638333] migration_entry_wait_on_locked+0x1d8/0x2d0 > >>>>>> [ 212.638336] ? __pfx_wake_page_function+0x10/0x10 > >>>>>> [ 212.638339] migration_entry_wait+0xd2/0xe0 > >>>>>> [ 212.638341] hmm_vma_walk_pmd+0x7c9/0x8d0 > >>>>>> [ 212.638343] walk_pgd_range+0x51d/0xa40 > >>>>>> [ 212.638345] __walk_page_range+0x75/0x1e0 > >>>>>> [ 212.638347] walk_page_range_mm+0x138/0x1f0 > >>>>>> [ 212.638349] hmm_range_fault+0x59/0xa0 > >>>>>> [ 212.638351] drm_gpusvm_get_pages+0x194/0x7b0 [drm_gpusvm_helper] > >>>>>> [ 212.638354] drm_gpusvm_range_get_pages+0x2d/0x40 > >>>>>> [drm_gpusvm_helper] > >>>>>> [ 212.638355] __xe_svm_handle_pagefault+0x259/0x900 [xe] > >>>>>> [ 212.638375] ? update_load_avg+0x7f/0x6c0 > >>>>>> [ 212.638377] ? update_curr+0x13d/0x170 > >>>>>> [ 212.638379] xe_svm_handle_pagefault+0x37/0x90 [xe] > >>>>>> [ 212.638396] xe_pagefault_queue_work+0x2da/0x3c0 [xe] > >>>>>> [ 212.638420] process_one_work+0x16e/0x2e0 > >>>>>> [ 212.638422] worker_thread+0x284/0x410 > >>>>>> [ 212.638423] ? __pfx_worker_thread+0x10/0x10 > >>>>>> [ 212.638425] kthread+0xec/0x210 > >>>>>> [ 212.638427] ? __pfx_kthread+0x10/0x10 > >>>>>> [ 212.638428] ? __pfx_kthread+0x10/0x10 > >>>>>> [ 212.638430] ret_from_fork+0xbd/0x100 > >>>>>> [ 212.638433] ? __pfx_kthread+0x10/0x10 > >>>>>> [ 212.638434] ret_from_fork_asm+0x1a/0x30 > >>>>>> [ 212.638436] </TASK> > >>>>>> > >>>>> > >>>>> Hi, Matt > >>>>> > >>>>> Thanks for the report, two questions > >>>>> > >>>>> 1. Are you using mm/mm-unstable, we've got some fixes in there > >>>>> (including fixes to remove_migration_pmd()) > >>> > >>> remove_migration_pmd - This is a PTE migration entry. > >>> > >> > >> I don't have your symbols, I thought we were hitting, the following > >> condition in the walk > >> > >> if (thp_migration_supported() && pmd_is_migration_entry(pmd)) { > >> > >> But sounds like you are not, PMD/THP has not been enabled in this case > >> > > > > No, migration_entry_wait rather than pmd_migration_entry_wait. > > > >> > >>>>> - Generally a left behind migration entry is a symptom of a failed > >>>>> migration that did not clean up > >>>>> after itself. > >>> > >>> I'm on drm-tip as I generally need the latest version of my driver > >>> because of the speed we move at. > >>> > >>> Yes, I agree it looks like somehow a migration PTE is not getting > >>> properly removed. > >>> > >>> I'm happy to cherry pick any patches that you think might be helpful > >>> into my tree. > >>> > >> > >> Could you try the mm/mm-new tree with the current xe driver? > >> > > > > Unfortunately, this is a tough one. We land a lot of patches in Xe/DRM, > > so bringing the driver up to date with an MM branch is difficult, and > > I’m not an expert at merging branches. It would be nice if, in the DRM > > flow, we could merge patches from outside our subsystem into a > > bleeding-edge kernel for the things we typically care about—but we’d > > need a maintainer to sign up for that. > > > >> In general, w.r.t failure, I would check for the following > >> > >> 1. Are the dst_pfns in migrate_vma_pages() setup correctly by the device > >> driver? > >> 2. Any failures in folio_migrate_mapping()? > >> 3. In migrate_vma_finalize() check to see if remove_migration_ptes() failed > >> > >> If (3) fails that will explain the left over migration entries > >> > > > > Good tips, but think I got it via biscet. > > > > Offending patch is: > > > > 'mm/migrate_device: handle partially mapped folios during collection' > > > > The failing test case involves some remap-related issue. It’s a > > parameterized test, so I honestly couldn’t tell you exactly what it’s > > doing beyond the fact that it seems nonsensical but stresses remap. I > > thought commit '66d81853fa3d selftests/mm/hmm-tests: partial unmap, > > mremap and anon_write tests' would catch this, but it looks like I need > > to make the remap HMM test cases a bit more robust—similar to my > > driver-side tests. I can take an action item to follow up on this. > > > > Good news, I can tell you how to fix this... > > > > In 'mm/migrate_device: handle partially mapped folios during collection': > > > > 109 +#if 0 > > 110 + folio = page ? page_folio(page) : NULL; > > 111 + if (folio && folio_test_large(folio)) { > > 112 + int ret; > > 113 + > > 114 + pte_unmap_unlock(ptep, ptl); > > 115 + ret = migrate_vma_split_folio(folio, > > 116 + > > migrate->fault_page); > > 117 + > > 118 + if (ret) { > > 119 + ptep = pte_offset_map_lock(mm, > > pmdp, addr, &ptl); > > 120 + goto next; > > 121 + } > > 122 + > > 123 + addr = start; > > 124 + goto again; > > 125 + } > > 126 +#endif > > > > You can probably just delete this and use my patch below, but if you > > want to try fixing it with a quick look: if migrate_vma_split_folio > > fails, you probably need to collect a hole. On success, you likely want > > to continue executing the remainder of the loop. I can try playing with > > this tomorrow, but it’s late here. > > > > I had privately sent you a version of this patch as a fix for Xe, and > > this one seems to work: > > > > [PATCH] mm/migrate: Split THP found in middle of PMD during page collection > > > > The migrate layer is not coded to handle a THP found in the middle of a > > PMD. This can occur if a user manipulates mappings with mremap(). If a > > THP is found mid-PMD during page collection, split it. > > > > Cc: Balbir Singh <[email protected]> > > Signed-off-by: Matthew Brost <[email protected]> > > --- > > mm/migrate_device.c | 37 +++++++++++++++++++++++++++++++++++-- > > 1 file changed, 35 insertions(+), 2 deletions(-) > > > > diff --git a/mm/migrate_device.c b/mm/migrate_device.c > > index abd9f6850db6..9ffc025bad50 100644 > > --- a/mm/migrate_device.c > > +++ b/mm/migrate_device.c > > @@ -65,6 +65,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, > > struct vm_area_struct *vma = walk->vma; > > struct mm_struct *mm = vma->vm_mm; > > unsigned long addr = start, unmapped = 0; > > + struct folio *split_folio = NULL; > > spinlock_t *ptl; > > pte_t *ptep; > > > > @@ -107,10 +108,11 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, > > } > > } > > > > - ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl); > > + ptep = pte_offset_map_lock(mm, pmdp, start, &ptl); > > if (!ptep) > > goto again; > > arch_enter_lazy_mmu_mode(); > > + ptep += (addr - start) / PAGE_SIZE; > > > > for (; addr < end; addr += PAGE_SIZE, ptep++) { > > struct dev_pagemap *pgmap; > > @@ -209,6 +211,11 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, > > bool anon_exclusive; > > pte_t swp_pte; > > > > + if (folio_order(folio)) { > > + split_folio = folio; > > + goto split; > > + } > > + > > flush_cache_page(vma, addr, pte_pfn(pte)); > > anon_exclusive = folio_test_anon(folio) && > > PageAnonExclusive(page); > > @@ -287,8 +294,34 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, > > if (unmapped) > > flush_tlb_range(walk->vma, start, end); > > > > +split: > > arch_leave_lazy_mmu_mode(); > > - pte_unmap_unlock(ptep - 1, ptl); > > + pte_unmap_unlock(ptep - 1 + !!split_folio, ptl); > > + > > + /* > > + * XXX: No clean way to support higher-order folios that don't > > match PMD > > + * boundaries for now — split them instead. Once mTHP support > > lands, add > > + * proper support for this case. > > + * > > + * The test, which exposed this as problematic, remapped (memremap) > > a > > + * large folio to an unaligned address, resulting in the folio being > > + * found in the middle of the PTEs. The requested number of pages > > was > > + * less than the folio size. Likely to be handled gracefully by > > upper > > + * layers eventually, but not yet. > > + */ > > + if (split_folio) { > > + int ret; > > + > > + ret = split_folio(split_folio); > > + if (fault_folio != split_folio) > > + folio_unlock(split_folio); > > + folio_put(split_folio); > > + if (ret) > > + return migrate_vma_collect_skip(addr, end, walk); > > + > > + split_folio = NULL; > > + goto again; > > + } > > > > return 0; > > } > > > > If I apply the #if 0 change along with my patch above (plus one core > > MM patch needed for Xe that adds a support function), Xe SVM fully > > passes our test cases with both THP enabled and disabled. > > > Excellent work! Since you found this, do you mind sending the fix to Andrew > as a fixup
Done. Here is a dri-devel patchworks link [1] to the patch. Matt [1] https://patchwork.freedesktop.org/series/157859/ > to the original patch. Since I don't have the test case, I have no way of > validating the > change or any change on top of it would continue to work > > FYI: The original code does something similar, I might be missing the > migrate_vma_collect_skip() bits. > > Thanks! > Balbir > >
