>> But I think, we cannot use64af97000 address in the swiotlb_bounce() >> directly. >> >I am not sure to understand what you mean. Can you clarify?
I thought, that the address 64af97000 can be used directly here: https://elixir.bootlin.com/linux/v5.10/source/kernel/dma/swiotlb.c#L572 and I was confused about it. After some diggings I realized that this code is not reachable because DMA_ATTR_SKIP_CPU_SYNC is active. >> Swiotlb did not fit requested slots because the maximum slot size equals >> IO_TLB_SEGSIZE=128 by default. > >> Ok. So it sounds like your problem is the have a DMA buffer that is too >> large for the default swiotlb. Did you try to bump the value from the >> command line? Yes, you are right. I checked the implementation more detailed and figured out that swiotlb tries to find a contiguous segment that is a big enough, as shown here: https://elixir.bootlin.com/linux/v5.10/source/kernel/dma/swiotlb.c#L528, But the maximum TLB segment equals 128K by default in accordance with https://elixir.bootlin.com/linux/v5.10/source/include/linux/swiotlb.h#L25 This means that the max size of TLB segment equals: 128 * 2048 = 256K After this, I tried to increase the TLB segment size like this: diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index fbdc65782195..85f32865bb6c 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -22,7 +22,7 @@ enum swiotlb_force { * must be a power of 2. What is the appropriate value ? * The complexity of {map,unmap}_single is linearly dependent on this value. */ -#define IO_TLB_SEGSIZE 128 +#define IO_TLB_SEGSIZE 2048 /* * log of the size of each IO TLB slab. The number of slabs is command line and the problem is gone. Also, I found the article https://www.xilinx.com/support/answers/72694.html, where I believe the same issue was mentioned. Thank you so much for your time and help! ________________________________ From: Julien Grall <[email protected]> Sent: Friday, August 13, 2021 1:51 PM To: Roman Skakun <[email protected]>; [email protected] <[email protected]> Cc: Bertrand Marquis <[email protected]>; Andrii Anisov <[email protected]>; Volodymyr Babchuk <[email protected]>; Oleksandr Tyshchenko <[email protected]>; Oleksandr Andrushchenko <[email protected]>; [email protected] <[email protected]>; Roman Skakun <[email protected]>; Jan Beulich <[email protected]> Subject: Re: Disable swiotlb for Dom0 On 13/08/2021 10:38, Roman Skakun wrote: > Hi Julien, Hi Roman, >> So 0xb6000000 is most likely the GFN used to mapped the grant from the domU. >> >> swiotlb-xen on Arm will convert it to the MFN because it is not aware >> whether the device is behind an IOMMU. > > If I'm understand right, it seems like that swiotlb-xen is not ready to > work properly in case > when we retrieved MFN instead of proper GFN mapped to Dom0 memory. > Maybe you know some ideas to overcome this condition? swiotlb-xen work as intended. You have a DMA buffer at an address that your device cannot deal with. So it will try to bounce it. > >> As the address is too high to be handled by the device, swiotlb will try >> to bounce it. I think it is correct to bounce the page but I am not sure >> why it can't. What the size of the DMA transaction? > > The DMA map size is 3686400 bytes. So that's a 3MB buffer. I am slightly confused because in an earlier message you wrote that the memory is coming from the guest. How did you map it? > > I've added several logs to swiotlb map_single() and see: > [ 151.298455] <SWIOTLB> swiotlb_tbl_map_single() origin_addr: > 64af97000, needed: 708, > avail: 7fc0, stride: 2, index: 4160 I am not sure how to read the logs... Are the number in hexadecimal or decimal? It might be useful if you post the diff of your changes. [...] > Swiotlb did not fit requested slots because the maximum slot size equals > IO_TLB_SEGSIZE=128 by default. Ok. So it sounds like your problem is the have a DMA buffer that is too large for the default swiotlb. Did you try to bump the value from the command line? > But I think, we cannot use64af97000 address in the swiotlb_bounce() > directly. I am not sure to understand what you mean. Can you clarify? Cheers, [...] -- Julien Grall
