On Thu, 2 Jul 2026 00:56:40 -0400 David Hu <[email protected]> wrote:
> On Tue, Jun 30, 2026 at 8:42 AM Jason Gunthorpe <[email protected]> wrote: > > > > On Tue, Jun 23, 2026 at 11:53:50PM +0100, David Laight wrote: > > > > > > If we restrict incoming dmabuf transfers to fit within VFS-centric > > > > limits (2GB), we impose unnecessary overhead on the RDMA stack, forcing > > > > it to manage a significantly higher number of memory registrations. By > > > > cleanly splitting these massive contiguous device buffers into > > > > page-aligned SGL entries, we directly improve the efficiency of P2P > > > > transfers and memory registration. > > > > > > But a divide by '4G - PAGE_SIZE' is also non-trivial and (I think affects > > > a lot of io) when the quotient is always 1. > > > Splitting into 2G chunks is a lot cheaper. > > > > Doesn't matter this isn't fast path stuff. It is better to use fewer > > SGL entries, IHMO. > > > > > > Since this change doesn't seem to have a negative impact on standard > > > > file > > > > I/O or break existing VFS constraints, I'm curious why we shouldn't > > > > support splitting these >4GB P2P transfers? Am I missing something? > > > > > > I was only wondering whether it was needed... > > > It does bring up the question of why the >4GB transfers even need > > > splitting. > > > But that is another question. > > > > SGL can only store an unsigned int size, so any large physical range > > has to be split down. > > > > rdma now a days has code to process the sgl and restore back the > 4G > > sizes since mode RDMA HW can accept that. > > > > commit 486055f5e09df959ad4e3aa4ee75b5c91ddeec2e > > Author: Michael Margolin <[email protected]> > > Date: Mon Feb 17 14:16:23 2025 +0000 > > > > RDMA/core: Fix best page size finding when it can cross SG entries > > > > So whatever this produces needs to be compatible with that to undo it. > > Thank you everyone. It looks like most open issues are sorted out. > I'll wait for maintainers to weigh in before sending out v3 (which > will remove the type cast for min() per David L.'s feedback, and > revert to ALIGN_DOWN(UINT_MAX, PAGE_SIZE) per Jason's feedback). Does this code get used a lot for 'normal' transfers? I'm away from my normal systems and can't check. But if pretty much all of the fragments are small (< 4G) then it is probably worth adding a check for 'size < limit' before anything else and optimising that case. David > > Hi Jason, > > Thank you for your feedback. I took a closer look at the commit to > ensure compatibility. This patch is perfectly complementary, and > actually prevents a failure in an edge case for the latest > `ib_umem_find_best_pgsz` [1]. > > Regards, > David > > [1] For dma-buf split with `0xFFFFFFFF`, in case of a discontinguity > in later buffers, we will hit this code path in > `ib_umem_find_best_pgsz` > > ``` > if (i != 0) > mask |= va; > ``` > (*After `va` had been incremented by `0xFFFFFFFF`, due to `va += > sg_dma_len(sg) - pgoff`) > (*Which will set the lowest bit of `mask` to 1) > > Because `count_trailing_zeros(mask) returns 0`, > `ib_umem_find_best_pgsz()` will always return 0 in such cases.
