On Sun, 2025-11-09 at 16:19 +1000, David Airlie wrote:
> So this caused a regression, because the sysmem flush page has to be
> inside 40 bits.
>
> look in openrm:
> src/nvidia/src/kernel/gpu/mem_sys/arch/maxwell/kern_mem_sys_gm107.c:kmemsysInitFlushSysmemBuffer_G
> M107
>
> The prop driver tries to use GFP_DMA32, then use 40 bits and the code
> is all horrible. It's probably fine for use to just set the dma_bits
> to 40 here before and then the full range after.
Can you print out the actual flush page address on your GB203, as well as the
value of 'bits'?
So the 40-bit limit is because the register used on pre-Hopper for the address
is only 32 bits:
void
gf100_fb_sysmem_flush_page_init(struct nvkm_fb *fb)
{
nvkm_wr32(fb->subdev.device, 0x100c10, fb->sysmem.flush_page_addr >> 8);
}
But on Blackwell, the limit is 52 bits:
static void
gb202_fb_sysmem_flush_page_init(struct nvkm_fb *fb)
{
struct nvkm_device *device = fb->subdev.device;
const u64 addr = fb->sysmem.flush_page_addr;
nvkm_wr32(device, NV_PFB_FBHUB0_PCIE_FLUSH_SYSMEM_ADDR_HI,
upper_32_bits(addr));
nvkm_wr32(device, NV_PFB_FBHUB0_PCIE_FLUSH_SYSMEM_ADDR_LO,
lower_32_bits(addr));
}
The max value for NV_PFB_FBHUB0_PCIE_FLUSH_SYSMEM_ADDR_HI is 20 bits:
#define NV_PFB_FBHUB0_PCIE_FLUSH_SYSMEM_ADDR_HI
0x008a1d5c
#define NV_PFB_FBHUB0_PCIE_FLUSH_SYSMEM_ADDR_HI_ADR
31:0
#define NV_PFB_FBHUB0_PCIE_FLUSH_SYSMEM_ADDR_HI_ADR_INIT
0x00000000
#define NV_PFB_FBHUB0_PCIE_FLUSH_SYSMEM_ADDR_HI_ADR_MASK
0x000fffff
fffff is 20 bits.
OpenRM even says that the limit is 52 bits:
// Assert when Sysmem Flush buffer has more than 52-bit address
NV_ASSERT((alignedSysmemFlushBufferAddrHi &
(~NV_PFB_FBHUB0_PCIE_FLUSH_SYSMEM_ADDR_HI_ADR_MASK))
== 0);
So unless you've got some wacky IOMMU that pushes DMA address to above 52 bits,
I don't see you
could have hit this failure.
I'm going to add WARN_ONs in Nouveau to ensure that these addresses are in
range. But it appears
that on Hopper and later, the address range for the flush page is the same as
the device itself, so
my patch should not have broken it. It should have only broken things on
pre-Hopper.
