On Wed, 1 Jul 2026 at 06:45, Dave Airlie <[email protected]> wrote:
>
> On Tue, 30 Jun 2026 at 21:51, Jason Gunthorpe <[email protected]> wrote:
> >
> > On Tue, Jun 30, 2026 at 01:24:57PM +0200, Christian König wrote:
> >
> > > > Today I finally hit up iommu.strict=1 makes things a lot happier,
> > > > non-strict IOMMU seems to allow a race between dma_unmap_page and
> > > > free_page where the unmap goes into the IOMMU flush queue, where the
> > > > actual unmap are delayed until something triggers a flush later, and
> > > > the ATS translations stay alive past when they should, and after the
> > > > page has been allocated by some subsequent user.
> > >
> > > Try disabling ATS, that was a common source of problems during HW
> > > bringup for us in the past.
> >
> > The spark HW cannot run without ATS.
> >
> > > > 2. split ttm page unmap/free into two stages, and force an iommu
> > > > flush/ATS sync operation after the unmap before the free. (I don't
> > > > think iommu has an interface for this).
> > >
> > > Yeah that idea already came up on similar issues as well. I
> > > certainly wouldn't be opposed to that.
> >
> > Only for debugging, it is completely wrong to rely on this for
> > functionality.
> >
> > > You either have to use memory fences which use the same path as the
> > > DMA operations itself
> >
> > Yes, in a PCIe system generally something like the GPU TLB
> > invalidation command will complete and flush the fabric path to quiet
> > the DMAs. That won't be true on spark though.
> >
> > Dave could be facing a fabric issue. AFAIK the ATS invalidation that
> > the iommu=strict pushses should fence the fabric so would resolve a
> > missing fabric fence. I suppose if you don't see any evidence of a
> > non-present ATS response explosion with strict then this would be a
> > really good theory.
>
> It's definitely a fabric issue, the iommu ATS invalidation is what
> "fixes" it, but I'm not figuring out how to flush it from the GPU
> side, and I'm not seeing where the official driver does this either,
> guess I have to keep digging. Maybe Alistair knows something.
>
> >
> > However, the GPU must have another way to inject a fabric fence. The
> > production driver does not rely on ATS fencing to synchronize memory
> > operations!
>From the production driver, there definitely was some issues in the
past, however this seems possibly fixed with mmu notifiers, but for
these memory allocations I'm doing I'm not using MMU notifiers at all.
#if defined(NV_MMU_NOTIFIER_OPS_HAS_ARCH_INVALIDATE_SECONDARY_TLBS)
#define UVM_ATS_SMMU_WAR_REQUIRED() 0
#elif NVCPU_IS_AARCH64
#define UVM_ATS_SMMU_WAR_REQUIRED() 1
#else
#define UVM_ATS_SMMU_WAR_REQUIRED() 0
#endif
#if UVM_ATS_SMMU_WAR_REQUIRED()
void uvm_ats_smmu_invalidate_tlbs(uvm_gpu_va_space_t *gpu_va_space,
NvU64 addr, size_t size)
{
struct mm_struct *mm = gpu_va_space->va_space->va_space_mm.mm;
uvm_parent_gpu_t *parent_gpu = gpu_va_space->gpu->parent;
struct {
NvU64 low;
NvU64 high;
} *vcmdq;
unsigned long vcmdq_prod;
NvU64 end;
uvm_spin_loop_t spin;
NvU16 asid;
if (!parent_gpu->ats.smmu_war.smmu_cmdqv_base)
return;
asid = arm64_mm_context_get(mm);
vcmdq = kmap(parent_gpu->ats.smmu_war.smmu_cmdq);
uvm_mutex_lock(&parent_gpu->ats.smmu_war.smmu_lock);
vcmdq_prod = parent_gpu->ats.smmu_war.smmu_prod;
// Our queue management is very simple. The mutex prevents multiple
// producers writing to the queue and all our commands require waiting for
// the queue to drain so we know it's empty. If we can't fit enough commands
// in the queue we just invalidate the whole ASID.
//
// The command queue is a cirular buffer with the MSB representing a wrap
// bit that must toggle on each wrap. See the SMMU architecture
// specification for more details.
//
// SMMU_VCMDQ_CMDQ_ENTRIES - 1 because we need to leave space for the
// CMD_SYNC.
if ((size >> PAGE_SHIFT) > min(UVM_MAX_TLBI_OPS,
SMMU_VCMDQ_CMDQ_ENTRIES - 1)) {
vcmdq[vcmdq_prod % SMMU_VCMDQ_CMDQ_ENTRIES].low = CMDQ_OP_TLBI_EL2_ASID;
vcmdq[vcmdq_prod % SMMU_VCMDQ_CMDQ_ENTRIES].low |= (NvU64) asid << 48;
vcmdq[vcmdq_prod % SMMU_VCMDQ_CMDQ_ENTRIES].high = 0;
vcmdq_prod++;
}
else {
for (end = addr + size; addr < end; addr += PAGE_SIZE) {
vcmdq[vcmdq_prod % SMMU_VCMDQ_CMDQ_ENTRIES].low =
CMDQ_OP_TLBI_EL2_VA;
vcmdq[vcmdq_prod % SMMU_VCMDQ_CMDQ_ENTRIES].low |= (NvU64)
asid << 48;
vcmdq[vcmdq_prod % SMMU_VCMDQ_CMDQ_ENTRIES].high = addr &
~((1UL << 12) - 1);
vcmdq_prod++;
}
}
vcmdq[vcmdq_prod % SMMU_VCMDQ_CMDQ_ENTRIES].low = CMDQ_OP_CMD_SYNC;
vcmdq[vcmdq_prod % SMMU_VCMDQ_CMDQ_ENTRIES].high = 0x0;
vcmdq_prod++;
// MSB is the wrap bit
vcmdq_prod &= (1UL << (SMMU_VCMDQ_CMDQ_BASE_LOG2SIZE + 1)) - 1;
parent_gpu->ats.smmu_war.smmu_prod = vcmdq_prod;
smmu_vcmdq_write32(parent_gpu->ats.smmu_war.smmu_cmdqv_base,
SMMU_VCMDQ_PROD, parent_gpu->ats.smmu_war.smmu_prod);
UVM_SPIN_WHILE(
(smmu_vcmdq_read32(parent_gpu->ats.smmu_war.smmu_cmdqv_base,
SMMU_VCMDQ_CONS) & GENMASK(19, 0)) != vcmdq_prod,
&spin);
uvm_mutex_unlock(&parent_gpu->ats.smmu_war.smmu_lock);
kunmap(parent_gpu->ats.smmu_war.smmu_cmdq);
arm64_mm_context_put(mm);
}
#endif
