ccing bug for the record on testing On Wed, May 10, 2017 at 8:33 AM, Nate Watterson <nwatt...@qti.qualcomm.com> wrote: > Ok, what is the physical device that ./fio.virtualfs is being created > on? Is it and NVME or SATA (internal or PCI attached) disk? > > -----Original Message----- > From: Manoj Iyer [mailto:manoj.i...@canonical.com] > Sent: Wednesday, May 10, 2017 9:06 AM > To: Nate Watterson <nwatt...@qti.qualcomm.com> > Cc: Manoj Iyer <manoj.i...@canonical.com>; Timur Tabi > <ti...@qti.qualcomm.com> > Subject: RE: how to test iommu.passthough=0/1 ?? > > On Wed, 10 May 2017, Nate Watterson wrote: > >> >> Manoj, >> >> What was the backing store for the file mounted as /dev/loop0 in >> the results >> you sent? > > dd if=/dev/zero of=./fio.virtualfs bs=1M count=5120 > mkfs.ext4 ./fio.virtualfs > sudo losetup /dev/loop0 ./fio.virtualfs > > >> >> >> >> For iperf, I typically only test the target device as the iperf >> server and >> directly connect a host PC to run the client threads. I test this >> way >> because it usually has the worst performance with SMMU enabled. >> >> >> >> Target CMD: iperf -s >> >> Client CMD: iperf -c 192.168.2.222 -fg -i10 -t60 -P8 >> >> >> >> -Nate >> >> >> >> From: Manoj Iyer [mailto:manoj.i...@canonical.com] >> Sent: Tuesday, May 9, 2017 10:29 PM >> To: Nate Watterson <nwatt...@qti.qualcomm.com> >> Cc: Timur Tabi <ti...@qti.qualcomm.com> >> Subject: RE: how to test iommu.passthough=0/1 ?? >> >> >> >> Nate, >> >> >> >> Thanks for that info. I will give that a try. Did you see my iperf >> results ? >> those were kind of odd too.. I thought with the mlx card I might be >> testing >> with PCIe device .. but may I was hitting a limitation with iperf.. >> >> >> >> Also, I tested the same kernel on Cavium system, and with >> iommu.passthough=1/0 was I seeing a significant difference in the >> iops with >> a file mounted to /dev/loop0. Its attached here.. >> >> >> >> >> On Tue, May 9, 2017 at 8:52 PM, Nate Watterson >> <nwatt...@qti.qualcomm.com> >> wrote: >> >> Hi Manoj, >> >> Sorry for the late reply. I finally found some time to test >> passthrough mode and verified using the SMMU performance >> counters that the ‘iommu.passthrough’ param is working as >> I >> would expect it to (1 == Passthrough / 0 == non-Passthrough). >> I >> am not sure why you’re seeing this strange behavior but >> based on >> the logs you’ve sent, I have a few comments: >> >> >> >> Your boot log is showing that only the PCIe SMMUs are enabled >> (this is good, it means your FW is recent). Because of this, >> you >> will need to do your testing with a PCIe attached device >> instead >> of the SD card or SSD (assuming it was connected to the >> onboard >> SATA). I would recommend using an NVME drive as the impact of >> enabling passthrough mode will only really be apparent with >> devices capable of sustaining high IO throughput. >> >> >> >> I would recommend changing your FIO profile to access the >> device >> directly instead of going through the filesystem. >> Additionally, >> I would recommend using a readonly profile without >> randomization >> to preserve the disk and get consistent results. The >> following >> is the FIO command that I use: >> >> >> >> fio --name=global --readonly --group_reporting \ >> >> --direct=1 --ioengine=libaio --rw=read --eta-newline=1s \ >> >> --size=1T --blocksize=512k --iodepth=32 --numjobs=1 >> --runtime=10s \ >> >> --name=nvme_0 --filename=/dev/nvme0n1 >> >> >> >> *** iommu.passthrough=1 *** >> >> READ: io=25301MB, aggrb=2528.3MB/s, minb=2528.3MB/s, >> maxb=2528.3MB/s, mint=10007msec, maxt=10007msec >> >> >> >> *** iommu.passthrough=0 *** >> >> READ: io=22657MB, aggrb=2265.5MB/s, minb=2265.5MB/s, >> maxb=2265.5MB/s, mint=10001msec, maxt=10001msec >> >> >> >> Hopefully this helps. Let me know if you have any more >> questions. >> >> -Nate >> >> >> >> From: Manoj Iyer [mailto:manoj.i...@canonical.com] >> Sent: Thursday, May 4, 2017 4:55 PM >> To: Nate Watterson <nwatt...@qti.qualcomm.com> >> Cc: Timur Tabi <ti...@qti.qualcomm.com> >> Subject: RE: how to test iommu.passthough=0/1 ?? >> >> >> >> Nate, >> >> >> >> I am attaching fio output for the CAF kernel that is available in >> the >> centriq PPA. The results are not that much different. This time I >> ran >> fio on the loop back disk and also on the mmcblk0 SD card. >> >> >> >> Please see the attached test report. >> >> >> >> Thanks >> >> Manoj Iyer >> >> On Thu, May 4, 2017 at 2:46 PM, Manoj Iyer >> <manoj.i...@canonical.com> >> wrote: >> >> I have attached the boot log from the system with >> iommu.passthrough=1. >> >> >> >> Also I ran iperf on the mlx card interface.. and again the >> results don't make sense. I was expecting see better numbers >> with iommu.passthrough=1. Please see the attached test data for >> UDP and TCP. >> >> >> >> Figuring out what firmware I have was always a challenge. I had >> brought this up with the fw folks that there is no way to say I >> am on version X Y or Z.. anyways.. >> >> >> >> FW on this SDP: >> >> XBL.DF.2.0.R1-00406 QDF2400_REL CRM >> >> >> >> Type: FF6D3C53-2748-4017-838F-07BACCC91341 >> >> Version: 0000016C >> >> Version Name: QDF2400.FW.2.0.R1-00364 bait-qsvbuild-24-bldr-lnx >> >> >> >> Which I think translates to FW: >> >> commit 891a97c41712112ce38ba70220c9ca85c9e52869 >> >> Author: QC Publisher <qcpublis...@qti.qualcomm.com> >> >> Date: Mon Apr 17 13:02:52 2017 -0700 >> >> >> >> Commit label r00364.1 - N/A 0.0.364.1 >> >> >> >> TRACKING-ID:920c38a9-68f4-4218-870e-e9c593e321ad >> >> >> >> >> >> >> On Thu, May 4, 2017 at 2:21 PM, Nate Watterson >> <nwatt...@qti.qualcomm.com> wrote: >> >> Ok that makes a bit more sense. I still need the >> boot log though. Is the USB device connected to one >> of the onboard USB controllers or a PCI based USB >> controller? Do you happen to know what FW version >> you’re running? >> >> -Nate >> >> >> >> From: Manoj Iyer [mailto:manoj.i...@canonical.com] >> Sent: Thursday, May 4, 2017 2:56 PM >> To: Nate Watterson <nwatt...@qti.qualcomm.com> >> Cc: Timur Tabi <ti...@qti.qualcomm.com> >> Subject: RE: how to test iommu.passthough=0/1 ?? >> >> >> >> Opps my bad.. I am terribly sorry.. /dev/sdd is a >> USB device. >> >> >> >> *-usb:2 >> >> description: Mass storage device >> >> product: ADATA USB Flash Drive >> >> vendor: ADATA >> >> description: SCSI Disk >> >> physical id: 0.0.0 >> >> bus info: scsi@9:0.0.0 >> >> logical name: /dev/sdd >> >> >> >> >> On Thu, May 4, 2017 at 1:30 PM, Nate Watterson >> <nwatt...@qti.qualcomm.com> wrote: >> >> Please send the boot log. How is the SD card >> attached? I would have expected it to show up >> as an mmcblk device but it appears to be >> showing up at /dev/sdd. >> >> -Nate >> >> >> >> From: Nate Watterson >> Sent: Thursday, May 4, 2017 2:12 PM >> To: 'Manoj Iyer' <manoj.i...@canonical.com> >> Cc: Timur Tabi <ti...@qti.qualcomm.com> >> Subject: RE: how to test iommu.passthough=0/1 >> ?? >> >> >> >> Some of those results are shocking. I’ll try your >> FIO profile here a bit later and let you know what I >> get. >> >> -Nate >> >> >> >> From: Manoj Iyer [mailto:manoj.i...@canonical.com] >> Sent: Thursday, May 4, 2017 1:00 PM >> To: Nate Watterson <nwatt...@qti.qualcomm.com> >> Cc: Timur Tabi <ti...@qti.qualcomm.com> >> Subject: RE: how to test iommu.passthough=0/1 ?? >> >> >> >> I tried fio on a file on SSD mounted to loopback >> device, and on the SD card on the AW SDP. I dont >> have any additional disk on it so I am limited to >> those options. But the results are a bit puzzling.. >> I see lower iops for iommu.passthough=1. I have >> attached the test and output, may be I am looking at >> this wrong... >> >> >> >> Any thoughts? >> >> On Thu, May 4, 2017 at 10:25 AM, Nate Watterson >> <nwatt...@qti.qualcomm.com> wrote: >> >> Manoj, >> >> I don’t think there will be anything >> obvious in dmesg you can check. You’ll >> pretty much have to run an IO benchmark >> (eg: iperf/fio) in the *host* kernel on >> a device with sufficient IO throughput >> to show the effects of the change. You >> should see a significant performance >> benefit to having iommu.passthrough==1. >> Note that this will have absolutely no >> impact at all on IO performance within a >> VM. Let me know if you have any other >> questions. >> >> >> >> -Nate >> >> >> >> From: Manoj Iyer >> [mailto:manoj.i...@canonical.com] >> Sent: Thursday, May 4, 2017 1:26 AM >> To: Timur Tabi <ti...@qti.qualcomm.com>; >> Nate Watterson >> <nwatt...@qti.qualcomm.com> >> Subject: how to test >> iommu.passthough=0/1 ?? >> >> >> >> Nate/Timur, >> >> >> >> I backported the SMMU pass though using >> default domain patch series from linux-next. I >> have a public bug >> filed https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1688158 >> to SRU this patch series. But I cant figure >> out a way of testing iommu.passthough=0/1. >> Could you pls suggest a test and tell me what >> to look for dmesg etc to compare both cases? I >> understand it is supposed to improve >> performance is there a way of checking that ? >> >> >> >> I compared dmesg on both cases, and also >> launched vms and looked to see if I can spot >> anything obvious.. Really appreciate your help >> with suggestions on testing this. I need to >> post some testing data with my SRU request. >> >> >> >> Thanks >> >> Manoj Iyer. >> >> >> > > -- > ============================ > Manoj Iyer > Ubuntu/Canonical > ARM Servers - Cloud > ============================
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1688158 Title: [SRU][Zesty] Support SMMU passthrough using the default domain Status in linux package in Ubuntu: Incomplete Bug description: [Impact] Have the SMMU come up in a passthrough configuration, and then allow subsequent translation for things such as VFIO. Rather than do this in each SMMU driver, it's much cleaner to allow the default domain to be configured to be something other than DMA. This patch series implements a command-line option to configure the default domain type. Currently, it supports "dma" and "identity" which is sufficient for the passthrough use-case. [Fix] The following patch series in linux-next adds this support to the kernel. 4a8d8a14c0d0 arm64: dma-mapping: Only swizzle DMA ops for IOMMU_DOMAIN_DMA fccb4e3b8ab0 iommu: Allow default domain type to be set on the kernel command line beb3c6a066bf iommu/arm-smmu-v3: Install bypass STEs for IOMMU_DOMAIN_IDENTITY domains 67560edcd8e5 iommu/arm-smmu-v3: Make arm_smmu_install_ste_for_dev return void 61bc671179f1 iommu/arm-smmu: Install bypass S2CRs for IOMMU_DOMAIN_IDENTITY domains 0834cc28fa56 iommu/arm-smmu: Restrict domain attributes to UNMANAGED domains To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1688158/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
