[email protected]
On Thu, 14 Apr 2022 at 14:54, Viresh Kumar <[email protected]> wrote:
>
> +xen-devel
>
> On 14-04-22, 14:45, Viresh Kumar wrote:
> > Hello,
> >
> > We verified our hypervisor-agnostic Rust based vhost-user backends with Qemu
> > based setup earlier, and there was growing concern if they were truly
> > hypervisor-agnostic.
> >
> > In order to prove that, we decided to give it a try with Xen, a type-1
> > bare-metal hypervisor.
> >
> > We are happy to announce that we were able to make progress on that front
> > and
> > have a working setup where we can test our existing Rust based backends,
> > like
> > I2C, GPIO, RNG (though only I2C is tested as of now) over Xen.
> >
> > Key components:
> > --------------
> >
> > - Xen: https://github.com/vireshk/xen
> >
> > Xen requires MMIO and device specific support in order to populate the
> > required devices at the guest. This tree contains four patches on the top
> > of
> > mainline Xen, two from Oleksandr (mmio/disk) and two from me (I2C).
> >
> > - libxen-sys: https://github.com/vireshk/libxen-sys
> >
> > We currently depend on the userspace tools/libraries provided by Xen, like
> > xendevicemodel, xenevtchn, xenforeignmemory, etc. This crates provides
> > Rust
> > wrappers over those calls, generated automatically with help of bindgen
> > utility in Rust, that allow us to use the installed Xen libraries. Though
> > we
> > plan to replace this with Rust based "oxerun" (find below) in longer run.
> >
> > - oxerun (WIP): https://gitlab.com/mathieupoirier/oxerun/-/tree/xen-ioctls
> >
> > This is Rust based implementations for Ioctl and hypercalls to Xen. This
> > is WIP
> > and should eventually replace "libxen-sys" crate entirely (which are C
> > based
> > implementation of the same).
> >
> > - vhost-device: https://github.com/vireshk/vhost-device
> >
> > These are Rust based vhost-user backends, maintained inside the rust-vmm
> > project. This already contain support for I2C and RNG, while GPIO is under
> > review. These are not required to be modified based on hypervisor and are
> > truly hypervisor-agnostic.
> >
> > Ideally the backends are hypervisor agnostic, as explained earlier, but
> > because of the way Xen maps the guest memory currently, we need a minor
> > update
> > for the backends to work. Xen maps the memory via a kernel file
> > /dev/xen/privcmd, which needs calls to mmap() followed by an ioctl() to
> > make
> > it work. For this a hack has been added to one of the rust-vmm crates,
> > vm-virtio, which is used by vhost-user.
> >
> >
> > https://github.com/vireshk/vm-memory/commit/54b56c4dd7293428edbd7731c4dbe5739a288abd
> >
> > The update to vm-memory is responsible to do ioctl() after the already
> > present
> > mmap().
> >
> > - vhost-user-master (WIP): https://github.com/vireshk/vhost-user-master
> >
> > This implements the master side interface of the vhost protocol, and is
> > like
> > the vhost-user-backend (https://github.com/rust-vmm/vhost-user-backend)
> > crate
> > maintained inside the rust-vmm project, which provides similar
> > infrastructure
> > for the backends to use. This shall be hypervisor independent and provide
> > APIs
> > for the hypervisor specific implementations. This will eventually be
> > maintained inside the rust-vmm project and used by all Rust based
> > hypervisors.
> >
> > - xen-vhost-master (WIP): https://github.com/vireshk/xen-vhost-master
> >
> > This is the Xen specific implementation and uses the APIs provided by
> > "vhost-user-master", "oxerun" and "libxen-sys" crates for its functioning.
> >
> > This is designed based on the EPAM's "virtio-disk" repository
> > (https://github.com/xen-troops/virtio-disk/) and is pretty much similar
> > to it.
> >
> > One can see the analogy as:
> >
> > Virtio-disk == "Xen-vhost-master" + "vhost-user-master" + "oxerun" +
> > "libxen-sys" + "vhost-device".
> >
> >
> >
> > Test setup:
> > ----------
> >
> > 1. Build Xen:
> >
> > $ ./configure --libdir=/usr/lib --build=x86_64-unknown-linux-gnu
> > --host=aarch64-linux-gnu --disable-docs --disable-golang
> > --disable-ocamltools
> > --with-system-qemu=/root/qemu/build/i386-softmmu/qemu-system-i386;
> > $ make -j9 debball CROSS_COMPILE=aarch64-linux-gnu- XEN_TARGET_ARCH=arm64
> >
> > 2. Run Xen via Qemu on X86 machine:
> >
> > $ qemu-system-aarch64 -machine virt,virtualization=on -cpu cortex-a57
> > -serial mon:stdio \
> > -device virtio-net-pci,netdev=net0 -netdev
> > user,id=net0,hostfwd=tcp::8022-:22 \
> > -device virtio-scsi-pci -drive
> > file=/home/vireshk/virtio/debian-bullseye-arm64.qcow2,index=0,id=hd0,if=none,format=qcow2
> > -device scsi-hd,drive=hd0 \
> > -display none -m 8192 -smp 8 -kernel /home/vireshk/virtio/xen/xen \
> > -append "dom0_mem=5G,max:5G dom0_max_vcpus=7 loglvl=all
> > guest_loglvl=all" \
> > -device
> > guest-loader,addr=0x46000000,kernel=/home/vireshk/kernel/barm64/arch/arm64/boot/Image,bootargs="root=/dev/sda2
> > console=hvc0 earlyprintk=xen" \
> > -device ds1338,address=0x20 # This is required to create a
> > virtual I2C based RTC device on Dom0.
> >
> > This should get Dom0 up and running.
> >
> > 3. Build rust crates:
> >
> > $ cd /root/
> > $ git clone https://github.com/vireshk/xen-vhost-master
> > $ cd xen-vhost-master
> > $ cargo build
> >
> > $ cd ../
> > $ git clone https://github.com/vireshk/vhost-device
> > $ cd vhost-device
> > $ cargo build
> >
> > 4. Setup I2C based RTC device
> >
> > $ echo ds1338 0x20 > /sys/bus/i2c/devices/i2c-0/new_device; echo 0-0020 >
> > /sys/bus/i2c/devices/0-0020/driver/unbind
> >
> > 5. Lets run everything now
> >
> > # Start the I2C backend in one terminal (open new terminal with "ssh
> > # root@localhost -p8022"). This tells the I2C backend to hook up to
> > # "/root/vi2c.sock0" socket and wait for the master to start transacting.
> > $ /root/vhost-device/target/debug/vhost-device-i2c -s /root/vi2c.sock -c
> > 1 -l 0:32
> >
> > # Start the xen-vhost-master in another terminal. This provides the path
> > of
> > # the socket to the master side and the device to look from Xen, which is
> > I2C
> > # here.
> > $ /root/xen-vhost-master/target/debug/xen-vhost-master --socket-path
> > /root/vi2c.sock0 --name i2c
> >
> > # Start guest in another terminal, i2c_domu.conf is attached. The guest
> > kernel
> > # should have Virtio related config options enabled, along with i2c-virtio
> > # driver.
> > $ xl create -c i2c_domu.conf
> >
> > # The guest should boot fine now. Once the guest is up, you can create
> > the I2C
> > # RTC device and use it. Following will create /dev/rtc0 in the guest,
> > which
> > # you can configure with 'hwclock' utility.
> >
> > $ echo ds1338 0x20 > /sys/bus/i2c/devices/i2c-0/new_device
> >
> >
> > Hope this helps.
> >
> > --
> > viresh
>
> i2c_domu.conf
>
> > kernel="/root/Image"
> > memory=512
> > vcpus=2
> > command="console=hvc0 earlycon=xenboot"
> > name="domu"
> > i2c = [ "virtio=true, irq=1, base=1" ]
>
> --
> viresh
