Re: Question about xen and Rasp 4B

[Jukka Kaartinen]

On 29.1.2021 20.50, Julien Grall wrote:
Hi,

@Jukka, would it be possible to at least configure your client to quote 
with '>'? This would make easier to understand who wrote what 
(tabulation is not great for that).

If you are using gmail, then configuring it to send it as plain text 
should do the job.
Sorry about that. May it's now better. Yes it is gmail. Maybe 
Thunderbird works better.


On 27/01/2021 11:47, Jukka Kaartinen wrote:


On Tue, Jan 26, 2021 at 10:22 PM Stefano Stabellini 
<sstabell...@kernel.org <mailto:sstabell...@kernel.org>> wrote:

    On Tue, 26 Jan 2021, Jukka Kaartinen wrote:
     > On Tue, Jan 26, 2021 at 2:54 AM Stefano Stabellini
    <sstabell...@kernel.org <mailto:sstabell...@kernel.org>> wrote:
     >       On Sat, 23 Jan 2021, Jukka Kaartinen wrote:
     >       > Thanks for the response!
     >       >
     >       > On Sat, Jan 23, 2021 at 2:27 AM Stefano Stabellini
    <sstabell...@kernel.org <mailto:sstabell...@kernel.org>> wrote:
     >       >       + xen-devel, Roman,
     >       >
     >       >
     >       >       On Fri, 22 Jan 2021, Jukka Kaartinen wrote:
     >       >       > Hi Stefano,
     >       >       > I'm Jukka Kaartinen a SW developer working on
    enabling hypervisors on mobile platforms. One of our HW that we 
use on
     >       >       development is
     >       >       > Raspberry Pi 4B. I wonder if you could help me a
    bit :).
     >       >       >
     >       >       > I'm trying to enable the GPU with Xen + Raspberry
    Pi for
     >       >       dom0.
    
https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605
    
<https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323#p1797605>
     >       >       >
     >       >       > I got so far that GPU drivers are loaded (v3d &
    vc4) without errors. But now Xen returns error when X is starting:
     >       >       > (XEN) traps.c:1986:d0v1 HSR=0x93880045
    pc=0x00007f97b14e70 gva=0x7f7f817000 gpa=0x0000401315d000
     >       >       >  I tried to debug what causes this and looks
    like find_mmio_handler cannot find handler.
     >       >       > (See more here:
    
https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691 

    
<https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=232323&start=25#p1801691> 

    )
     >       >       >
     >       >       > Any ideas why the handler is not found?
     >       >
     >       >
     >       >       Hi Jukka,
     >       >
     >       >       I am glad to hear that you are interested in Xen on
    RaspberryPi :-)  I
     >       >       haven't tried the GPU yet, I have been using the
    serial only.
     >       >       Roman, did you ever get the GPU working?
     >       >
     >       >
     >       >       The error is a data abort error: Linux is trying to
    access an address
     >       >       which is not mapped to dom0. The address seems to
    be 0x401315d000. It is
     >       >       a pretty high address; I looked in device tree but
    couldn't spot it.
     >       >
     >       >       >From the HSR (the syndrom register) it looks like
    it is a translation
     >       >       fault at EL1 on stage1. As if the Linux address
    mapping was wrong.
     >       >       Anyone has any ideas how this could happen? Maybe a
    reserved-memory
     >       >       misconfiguration?
     >       >
     >       > I had issues with loading the driver in the first place.
    Apparently swiotlb is used, maybe it can cause this. I also tried to
     >       enable CMA.
     >       > config.txt:
     >       > dtoverlay=vc4-fkms-v3d,cma=320M@0x0-0x40000000
     >       > gpu_mem=128
     >
     >       Also looking at your other reply and the implementation of
     >       vc4_bo_create, it looks like this is a CMA problem.
     >
     >       It would be good to run a test with the swiotlb-xen 
disabled:
     >
     >       diff --git a/arch/arm/xen/mm.c b/arch/arm/xen/mm.c
     >       index 467fa225c3d0..2bdd12785d14 100644
     >       --- a/arch/arm/xen/mm.c
     >       +++ b/arch/arm/xen/mm.c
     >       @@ -138,8 +138,7 @@ void
    xen_destroy_contiguous_region(phys_addr_t pstart, unsigned int order)
     >        static int __init xen_mm_init(void)
     >        {
     >               struct gnttab_cache_flush cflush;
     >       -       if (!xen_initial_domain())
     >       -               return 0;
     >       +       return 0;
     >               xen_swiotlb_init(1, false);
     >
     >               cflush.op = 0;
     >
     > With this change the kernel is not booting up. (btw. I'm using
    USB SSD for my OS.)
     > [    0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask
     > [    0.076277] bcm2835-dma fe007b00.dma: Unable to set DMA mask
     > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=25: not implemented
     > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
     > [    0.592695] pci 0000:00:00.0: Failed to add - passthrough or
    MSI/MSI-X might fail!
     > (XEN) physdev.c:16:d0v0 PHYSDEVOP cmd=15: not implemented
     > [    0.606819] pci 0000:01:00.0: Failed to add - passthrough or
    MSI/MSI-X might fail!
     > [    1.212820] usb 1-1: device descriptor read/64, error 18
     > [    1.452815] usb 1-1: device descriptor read/64, error 18
     > [    1.820813] usb 1-1: device descriptor read/64, error 18
     > [    2.060815] usb 1-1: device descriptor read/64, error 18
     > [    2.845548] usb 1-1: device descriptor read/8, error -61
     > [    2.977603] usb 1-1: device descriptor read/8, error -61
     > [    3.237530] usb 1-1: device descriptor read/8, error -61
     > [    3.369585] usb 1-1: device descriptor read/8, error -61
     > [    3.480765] usb usb1-port1: unable to enumerate USB device
     >
     > Traces stop here. I could try with a memory card. Maybe it makes
    a difference.

    This is very surprising. Disabling swiotlb-xen should make things 
better
    not worse. The only reason I can think of why it could make things 
worse
    is if Linux runs out of low memory. Julien's patch
    437b0aa06a014ce174e24c0d3530b3e9ab19b18b for Xen should have 
addressed
    that issue though. Julien, any ideas?

I think, Stefano's small patch is not enough to disable the swiotlb as 
we will still override the DMA ops. You also likely want:

diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index 8a8949174b1c..aa43e249ecdd 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -2279,7 +2279,7 @@ void arch_setup_dma_ops(struct device *dev, u64 
dma_base, u64 size,
         set_dma_ops(dev, dma_ops);

  #ifdef CONFIG_XEN
-       if (xen_initial_domain())
+       if (0 || xen_initial_domain())
                 dev->dma_ops = &xen_swiotlb_dma_ops;
  #endif
         dev->archdata.dma_ops_setup = true;

Otherwise, you would still use the swiotlb DMA ops that would not be 
functional as we disabled the swiotlb.

This would explain the following error because it will check whether the 
mask is valid using the callback dma_supported():

[    0.071081] bcm2835-dma fe007000.dma: Unable to set DMA mask

Good catch.
GPU works now and I can start X! Thanks! I was also able to create domU 
that runs Raspian OS.

Now that swiotlb is disabled what does it mean?

And also can I pass the GPU to domU? Raspberry Pi 4 is limited HW and 
doesn't have IOMMU. I'm trying to create similar OS like QubesOS where 
GPU, Network, keyboard/mouse, ... are isolated to their own VMs.




I really don't know if this is a problem but in the 
allocate_memory_11  arch_get_dma_bitsize is called. That should return 
the platform->dma_bitsize but at the early stage of boot platform is 
not initialized so default 32 is returned. I tried changing the hard 
code from 32 -> 30 but it didn't make any difference.

    static void __init allocate_memory_11(struct domain *d,
                                           struct kernel_info *kinfo)
    {
         const unsigned int min_low_order =
             get_order_from_bytes(min_t(paddr_t, dom0_mem, MB(128)));
         const unsigned int min_order = get_order_from_bytes(MB(4));
         struct page_info *pg;
         unsigned int order = get_allocation_size(kinfo->unassigned_mem);
         int i;

         bool lowmem = true;
         unsigned int lowmem_bitsize = min(32U, 
*arch_get_dma_bitsize*());

Domain memory allocation will always happen after platform_init() is 
called. So the value returned here will be correct.
I see.



also here the place where static dma_bitsize is set is not called for 
dom0


    void __init end_boot_allocator(void)
    {
    ....
         if ( !dma_bitsize && (num_online_nodes() > 1) )
             dma_bitsize = arch_get_dma_bitsize();

and will lead alloc_domheap_pages not to use dma_bitsize since it is 
not set.
struct page_info *alloc_domheap_pages(
     struct domain *d, unsigned int order, unsigned int memflags)
{

uses static: dma_bitsize and currently is not set for raspberry pi.

Right, this is normal because we don't support NUMA on Arm so 
num_online_nodes() would always return 1.

But I don't yet see the problem with the existing code. We will still 
try to allocate at least one bank below 30 bits on RPI.

 From the log you provided:
(XEN) Allocating 1:1 mappings totalling 5120MB for dom0:
(XEN) BANK[0] 0x00000008000000-0x00000028000000 (512MB)
(XEN) BANK[1] 0x00000080000000-0x000000c0000000 (1024MB)
(XEN) BANK[2] 0x00000100000000-0x000001e0000000 (3584MB)

You have 512MB of memory allocated below the 30 bits mark.

Are you expecting more memory to be allocated below the 30-bits?
No not really and looks like it is enough since I can run X. :)