Hi Christian, Philip, Alex, Felix, and all,
These series introduce a new Shared Virtual Memory (SVM) implementation for
amdgpu that is built directly on top of the common DRM GPUSVM / drm_pagemap
core (drivers/gpu/drm/drm_gpusvm.c, drm_pagemap.c) rather than on the
existing KFD-private SVM code in amdkfd/kfd_svm.c.
The goal is to provide HMM-based unified memory through the same shared
infrastructure that the Xe driver already uses, so that amdgpu and xe
converge on one well-reviewed SVM/page-migration core instead of
maintaining a separate amdgpu-specific stack.
Phased plan
===========
This work is being upstreamed in phases to keep each submission reviewable:
* Phase 1 (this series): single GPU, XNACK on (retry faults).
* Phase 2 (in progress): single GPU, XNACK off (no-retry / eager mapping).
-
https://lore.kernel.org/amd-gfx/[email protected]/
-
https://lore.kernel.org/amd-gfx/[email protected]/
* Phase 3 (in progress): multiple GPUs (XGMI mapping, P2P device-to-device
migration).
-
https://lore.kernel.org/amd-gfx/[email protected]
Accordingly,
this??first??submission??targets??the??simplest??useful??configuration:
??*??Single??GPU??with??local??VRAM??migration:??transparent??RAM??<->??VRAM??page
??????migration??on??one??GPU??via??SDMA,??with??TTM-based??eviction??for??overcommit.
??*??XNACK??on??(retry??faults):??GPU??page??faults??are??retried??after??the??driver
??????lazily??populates??PTEs??on??demand.??The??XNACK-off/eager-mapping??path??is
??????deferred??to??Phase??2.
*??Compute??VMs??only??(amdgpu_vm_make_compute()), matching??the??KFD??SVM
??????use??case.
Everything is gated behind a new, default-n Kconfig option
(CONFIG_DRM_AMDGPU_SVM) and is therefore opt-in and isolated from existing
users until the feature matures.
UAPI
====
A new render-node ioctl, DRM_IOCTL_AMDGPU_GEM_SVM, lets userspace describe SVM
attributes over a CPU virtual-address interval (patch 1):
- operations: SET_ATTR / GET_ATTR / RESET_ATTR
- access modes: INACCESSIBLE / IN_PLACE / ALLOW_MIGRATE
- location hints: SYSMEM / UNDEFINED (preferred_loc, prefetch_loc)
- per-range flags: HOST_ACCESS, COHERENT, EXT_COHERENT, HIVE_LOCAL,
GPU_RO, GPU_EXEC, GPU_READ_MOSTLY, plus a granularity hint
Attributes are stored in a per-VM interval tree; the GPU page tables are
populated lazily on demand by retry faults, or eagerly on a prefetch request.
User Space Work
===============
- ROCm UMD interface adaptation for the new drm SVM API is being
developed in:
https://github.com/ROCm/rocm-systems/pull/4364
Design Overview
===============
The implementation is split into clearly layered modules:
amdgpu_svm.c Core context (struct amdgpu_svm embeds
struct drm_gpusvm), kref lifecycle, PASID lookup,
drm_gpusvm_ops, the GEM_SVM ioctl entry point, and
the GC workqueue.
amdgpu_svm_attr.c The attribute interval tree: validation, gap/overlap
split-merge, SET/GET/RESET, and change-trigger
classification that decides whether an attribute
change needs PTE invalidation or a remap.
amdgpu_svm_range.c Per-range GPU mapping: PTE-flag computation per GC IP
version, DMA-segment coalescing, MMU-notifier
begin/end handling, PTE zapping, and the garbage
collector.
amdgpu_svm_fault.c The retry-fault entry point amdgpu_svm_handle_fault()
and the fault_map_range() pipeline.
amdgpu_migrate.c drm_pagemap / ZONE_DEVICE VRAM migration backend.
amdgpu_svm_range_migrate.c Per-range RAM<->VRAM migration helpers.
Fault path (XNACK on):
amdgpu_vm_handle_fault() routes a compute-VM retry fault to
amdgpu_svm_handle_fault() when the VM has an SVM context. After PASID lookup
and a checkpoint-timestamp filter that drops stale retry faults left over from
a recent unmap, it looks up (or synthesizes a default) attribute range and runs
fault_map_range(): garbage-collect -> VMA permission check -> find/insert range
-> short-circuit if recently validated or already valid -> drm_gpusvm get_pages
(HMM) -> program GPU PTEs under the notifier lock with a notifier-sequence
re-check.
Invalidation / GC:
MMU-notifier callbacks zap the affected PTEs (batched, single heavyweight TLB
flush) and, for unmap events, queue the range to a high-priority GC workqueue
that removes it via drm_gpusvm_range_remove() outside notifier context.
VRAM migration (patches 13-18):
A drm_pagemap is registered over the GPU's VRAM as a ZONE_DEVICE region at
device-init / reset-restore time. struct amdgpu_bo_svm (a new BO subtype)
backs migrated ranges, with SDMA-based copy_to_devmem / copy_to_ram callbacks
using a GART window. TTM eviction of SVM BOs synchronously migrates pages back
to system memory, which keeps VRAM overcommit working. The fault and prefetch
paths call amdgpu_svm_range_migrate_to_vram() before mapping when migration is
requested, with a single -EBUSY retry that evicts conflicting pages first.
Test Results
============
Tested on gfx943 (MI300X) and gfx906 (MI60) with XNACK on:
- KFD test: 95%+ passed.
- ROCR test: all passed.
- HIP catch test: gfx943 (MI300X): 99% passed. gfx906 (MI60): 99% passed.
Changes from Old Version
========================
In this V8 version, we have consolidated all implementations for the
single-GPU XNACK-on mode. This approach follows Christian's suggestion to
make the code review process more straightforward.
Previously, the XNACK-on implementation was split into two parts:
- Basic V7
https://lore.kernel.org/amd-gfx/[email protected]/
- Migration V5
https://lore.kernel.org/amd-gfx/[email protected]/
Above two parts are no longer needed to be reviewed separately, please
focus on this series.
Thanks,
Ray/Honglei/Junhua
Honglei Huang (12):
drm/amdgpu: add SVM ioctl UAPI definitions
drm/amdgpu: add SVM core header and VM integration
drm/amdgpu: add SVM attribute subsystem types
drm/amdgpu: implement SVM attribute tree and helper functions
drm/amdgpu: implement SVM attribute set/get/clear operations
drm/amdgpu: add SVM range types and work queue interface
drm/amdgpu: implement SVM range GPU mapping core
drm/amdgpu: implement SVM range notifier and GC helpers
drm/amdgpu: add SVM notifier invalidate callback and checkpoint
drm/amdgpu: implement SVM initialization and lifecycle
drm/amdgpu: add SVM ioctl entry and fault handler module
drm/amdgpu: integrate SVM into build system and VM fault path
Junhua Shen (6):
drm/amdgpu: add VRAM migration infrastructure for drm_pagemap
drm/amdgpu: implement drm_pagemap SDMA migration callbacks
drm/amdgpu: implement synchronous TTM eviction for SVM BOs
drm/amdgpu: hook up ZONE_DEVICE registration in device init and reset
drm/amdgpu: add SVM range migration helpers for drm_pagemap
drm/amdgpu: integrate VRAM migration into SVM fault and prefetch paths
drivers/gpu/drm/amd/amdgpu/Kconfig | 10 +
drivers/gpu/drm/amd/amdgpu/Makefile | 11 +
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 8 +
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 2 +
drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c | 848 +++++++++++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h | 102 ++
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 4 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_reset.c | 2 +
drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c | 724 +++++++++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h | 197 ++++
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c | 972 ++++++++++++++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h | 171 +++
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c | 418 ++++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h | 39 +
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c | 809 +++++++++++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h | 167 +++
.../drm/amd/amdgpu/amdgpu_svm_range_migrate.c | 120 +++
.../drm/amd/amdgpu/amdgpu_svm_range_migrate.h | 35 +
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 +
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 23 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 4 +
include/uapi/drm/amdgpu_drm.h | 106 ++
23 files changed, 4791 insertions(+), 3 deletions(-)
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.c
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_migrate.h
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.c
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm.h
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.c
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_attr.h
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.c
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_fault.h
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.c
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range.h
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.c
create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_svm_range_migrate.h
--
2.53.0
