On Fri, Sep 15, 2017 at 12:24:32PM +0200, Joel Carnat wrote:
> Hi,
>
> Initially comparing I/O speed between FreeBSD/ZFS/GELI and
> OpenBSD/FFS/CRYPTO, I noticed that there were a huge difference between
> plain and encrypted filesystem using OpenBSD. I ran the test on a 1
> vCore/1GB RAM Vultr VPS, running OpenBSD 6.2-beta. I had / configured in
> plain FFS and /home encrypted using bioctl(8). Then I ran a few `dd` and
> `bonnie++`
>
> According to those tests, writing FFS/CRYPTO is about 10 times slower than
> FFS/PLAIN.
> For the record, using the same `dd` on FreeBSD, ZFS with GELI is only 2
> times slower than plain ZFS.
> Furthemore, comparing FreeBSD/ZFS/PLAIN and OpenBSD/FFS/PLAIN, the speed is
> about the same.
> Finally, it seems reading OpenBSD/FFS/PLAIN and OpenBSD/FFS/CRYPTO is done
> at the same speed.
>
> Is this expected to have so much difference between FFS/PLAIN and FFS/CRYPTO
> when writing data?
>
> TIA,
> Jo
>
> PS: here's my test data.
>
> # sysctl kern.version hw.machine hw.model hw.ncpu hw.physmem
> kern.version=OpenBSD 6.2-beta (GENERIC) #91: Wed Sep 13 22:05:17 MDT 2017
> [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
>
> hw.machine=amd64
> hw.model=Virtual CPU a7769a6388d5
> hw.ncpu=1
> hw.physmem=1056817152
>
> # disklabel sd0
> # /dev/rsd0c:
> type: SCSI
> disk: SCSI disk
> label: Block Device
> duid: 69939b6a66c3879a
> flags:
> bytes/sector: 512
> sectors/track: 63
> tracks/cylinder: 255
> sectors/cylinder: 16065
> cylinders: 3263
> total sectors: 52428800
> boundstart: 64
> boundend: 52420095
> drivedata: 0
>
> 16 partitions:
> # size offset fstype [fsize bsize cpg]
> a: 16739680 35680384 4.2BSD 2048 16384 12958 # /
> b: 4208966 64 swap # none
> c: 52428800 0 unused
> d: 31471335 4209030 RAID
>
> # disklabel sd1
> # /dev/rsd1c:
> type: SCSI
> disk: SCSI disk
> label: SR CRYPTO
> duid: 4179a9e67beb3d4e
> flags:
> bytes/sector: 512
> sectors/track: 63
> tracks/cylinder: 255
> sectors/cylinder: 16065
> cylinders: 1958
> total sectors: 31470807
> boundstart: 64
> boundend: 31455270
> drivedata: 0
>
> 16 partitions:
> # size offset fstype [fsize bsize cpg]
> c: 31470807 0 unused
> e: 273024 64 4.2BSD 2048 16384 2133 # /etc
> h: 31182176 273088 4.2BSD 2048 16384 12958 # /home
>
> # mount
> /dev/sd0a on / type ffs (local, wxallowed)
> /dev/sd1e on /etc type ffs (local, softdep)
> /dev/sd1h on /home type ffs (local, nodev, nosuid)
>
> # df -h
> Filesystem Size Used Avail Capacity Mounted on
> /dev/sd0a 7.9G 915M 6.6G 12% /
> /dev/sd1e 131M 4.9M 120M 4% /etc
> /dev/sd1h 14.6G 2.0K 13.9G 0% /home
>
> # sync && time dd if=/dev/zero of=/TEST bs=512 count=3000000 && sync
> 3000000+0 records in
> 3000000+0 records out
> 1536000000 bytes transferred in 8.567 secs (179278802 bytes/sec)
> 0m08.61s real 0m00.29s user 0m07.70s system
>
> # sync && time dd if=/dev/zero of=/home/TEST bs=512 count=3000000 && sync
> 3000000+0 records in
> 3000000+0 records out
> 1536000000 bytes transferred in 20.875 secs (73580525 bytes/sec)
> 0m20.88s real 0m00.42s user 0m05.54s system
>
> # sync && time dd if=/dev/zero of=/TEST bs=4k count=300000 && sync
> 300000+0 records in
> 300000+0 records out
> 1228800000 bytes transferred in 4.151 secs (296024071 bytes/sec)
> 0m04.19s real 0m00.04s user 0m04.01s system
>
> # sync && time dd if=/dev/zero of=/home/TEST bs=4k count=300000 && sync
> 300000+0 records in
> 300000+0 records out
> 1228800000 bytes transferred in 22.872 secs (53723676 bytes/sec)
> 0m22.95s real 0m00.06s user 0m01.89s system
>
NOTE: a block size is 1024 bytes, so the counts are incorrect in the conversion.
the count should be 375000: 4096/512=8, 300000/8=375000.
my write numbers are:
run 1
+ dd if=/dev/zero of=/home/TEST bs=512 count=2400000
2400000+0 records in
2400000+0 records out
1228800000 bytes transferred in 8.616 secs (142611817 bytes/sec)
0m09.33s real 0m00.20s user 0m09.05s system
+ dd if=/dev/zero of=/home/TEST bs=4k count=300000
300000+0 records in
300000+0 records out
1228800000 bytes transferred in 5.591 secs (219749191 bytes/sec)
0m05.59s real 0m00.02s user 0m05.46s system
4k, 8k, 16k, 32k and 64k are comparable on my machine.
run 2
+ dd if=/dev/zero of=/home/TEST bs=512 count=2400000
2400000+0 records in
2400000+0 records out
1228800000 bytes transferred in 8.748 secs (140451506 bytes/sec)
0m09.24s real 0m00.26s user 0m08.87s system
+ dd if=/dev/zero of=/home/TEST bs=4k count=300000
300000+0 records in
300000+0 records out
1228800000 bytes transferred in 5.140 secs (239049708 bytes/sec)
0m05.87s real 0m00.03s user 0m05.74s system
4k, 8k, 16k, 32k and 64k are comparable on my machine.
This is on a physical machine with a Crucial BX100 SSD (softraid crypto) on
-current.
sd0 at scsibus1 targ 0 lun 0: <ATA, CT250BX100SSD1, MU02> SCSI3 0/direct fixed
naa...
sd0: 238475MB, 512 bytes/sector, 488397168 sectors, thin
For the read test a bs=64k or higher is fastest for me. This is the value of
struct stat: st.st_blksize.
I hope this helps you in some way.
> # sync && time dd if=/dev/zero of=/TEST bs=8k count=150000 && sync
> 150000+0 records in
> 150000+0 records out
> 1228800000 bytes transferred in 4.088 secs (300571699 bytes/sec)
> 0m04.12s real 0m00.05s user 0m03.93s system
>
> # sync && time dd if=/dev/zero of=/home/TEST bs=8k count=150000 && sync
> 150000+0 records in
> 150000+0 records out
> 1228800000 bytes transferred in 21.418 secs (57372236 bytes/sec)
> 0m21.48s real 0m00.05s user 0m01.72s system
>
> # time dd if=/TEST of=/dev/null
> 2400000+0 records in
> 2400000+0 records out
> 1228800000 bytes transferred in 12.327 secs (99677812 bytes/sec)
> 0m12.33s real 0m00.39s user 0m03.62s system
>
> # time dd if=/home/TEST of=/dev/null
> 2400000+0 records in
> 2400000+0 records out
> 1228800000 bytes transferred in 12.802 secs (95979204 bytes/sec)
> 0m12.80s real 0m00.29s user 0m02.87s system
>
> # time dd if=/TEST of=/dev/null bs=512
> 2400000+0 records in
> 2400000+0 records out
> 1228800000 bytes transferred in 12.888 secs (95337724 bytes/sec)
> 0m12.89s real 0m00.29s user 0m03.41s system
>
> # time dd if=/home/TEST of=/dev/null bs=512
> 2400000+0 records in
> 2400000+0 records out
> 1228800000 bytes transferred in 13.951 secs (88076531 bytes/sec)
> 0m13.95s real 0m00.24s user 0m02.61s system
>
> # time dd if=/TEST of=/dev/null bs=4k
> 300000+0 records in
> 300000+0 records out
> 1228800000 bytes transferred in 11.478 secs (107052456 bytes/sec)
> 0m11.48s real 0m00.07s user 0m01.91s system
>
> # time dd if=/home/TEST of=/dev/null bs=4k
> 300000+0 records in
> 300000+0 records out
> 1228800000 bytes transferred in 12.554 secs (97877823 bytes/sec)
> 0m12.55s real 0m00.06s user 0m01.80s system
>
> # time dd if=/TEST of=/dev/null bs=8k
> 150000+0 records in
> 150000+0 records out
> 1228800000 bytes transferred in 11.956 secs (102775667 bytes/sec)
> 0m11.95s real 0m00.02s user 0m01.87s system
>
> # time dd if=/home/TEST of=/dev/null bs=8k
> 150000+0 records in
> 150000+0 records out
> 1228800000 bytes transferred in 12.721 secs (96590791 bytes/sec)
> 0m12.72s real 0m00.06s user 0m01.66s system
>
> # bonnie++ -q -u www -d /tmp > /root/bonnie-PLAIN.csv
> Version 1.97 ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
> %CP
> cherie.tumfatig. 2G 1693 99 179217 93 60657 34 2314 98 92028 27 1390
> 118
> Latency 7876us 4947us 9914us 6272us 5084us
> 41171us
> Version 1.97 ------Sequential Create------ --------Random
> Create--------
> cherie.tumfatig.net -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
> %CP
> 16 1400 10 +++++ +++ 4976 21 1372 11 +++++ +++ 6331
> 34
> Latency 7572us 349us 4004us 18817us 86us
> 2868us
>
> # bonnie++ -q -u www -d /home/www > /root/bonnie-CRYPTO.csv
> Version 1.97 ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
> %CP
> cherie.tumfatig. 2G 1668 99 52340 13 18797 16 2420 96 85558 20 1037
> 84
> Latency 7189us 91008us 17138us 4562us 6496us
> 82148us
> Version 1.97 ------Sequential Create------ --------Random
> Create--------
> cherie.tumfatig.net -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
> %CP
> 16 425 7 +++++ +++ 455 4 423 7 +++++ +++ 1750
> 16
> Latency 10694us 859us 25427us 12505us 127us
> 10133us
>
> # dmesg
> OpenBSD 6.2-beta (GENERIC) #91: Wed Sep 13 22:05:17 MDT 2017
> [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
> real mem = 1056817152 (1007MB)
> avail mem = 1017909248 (970MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf68f0 (9 entries)
> bios0: vendor SeaBIOS version
> "rel-1.10.1-0-g8891697-prebuilt.qemu-project.org" date 04/01/2014
> bios0: QEMU Standard PC (i440FX + PIIX, 1996)
> acpi0 at bios0: rev 0
> acpi0: sleep states S3 S4 S5
> acpi0: tables DSDT FACP APIC HPET
> acpi0: wakeup devices
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Virtual CPU a7769a6388d5, 2400.42 MHz
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,RDTSCP,LONG,LAHF,ABM,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT
> cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
> 64b/line 16-way L2 cache
> cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
> cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 999MHz
> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
> acpihpet0 at acpi0: 100000000 Hz
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpicpu0 at acpi0: C1(@1 halt!)
> "ACPI0006" at acpi0 not configured
> "PNP0F13" at acpi0 not configured
> "PNP0700" at acpi0 not configured
> "PNP0A06" at acpi0 not configured
> "PNP0A06" at acpi0 not configured
> "PNP0A06" at acpi0 not configured
> "QEMU0002" at acpi0 not configured
> "ACPI0010" at acpi0 not configured
> pvbus0 at mainbus0: KVM
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
> pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
> pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel
> 0 wired to compatibility, channel 1 wired to compatibility
> pciide0: channel 0 disabled (no drives)
> atapiscsi0 at pciide0 channel 1 drive 0
> scsibus1 at atapiscsi0: 2 targets
> cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> ATAPI 5/cdrom
> removable
> cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
> uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0 int 11
> piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int
> 9
> iic0 at piixpm0
> vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
> vio0 at virtio0: address 56:00:00:8a:f3:5a
> virtio0: msix shared
> virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
> vioblk0 at virtio1
> scsibus2 at vioblk0: 2 targets
> sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
> sd0: 25600MB, 512 bytes/sector, 52428800 sectors
> virtio1: msix shared
> virtio2 at pci0 dev 5 function 0 "Qumranet Virtio Memory" rev 0x00
> viomb0 at virtio2
> virtio2: apic 0 int 10
> virtio3 at pci0 dev 6 function 0 "Qumranet Virtio RNG" rev 0x00
> viornd0 at virtio3
> virtio3: apic 0 int 10
> isa0 at pcib0
> isadma0 at isa0
> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pms0 at pckbc0 (aux slot)
> wsmouse0 at pms0 mux 0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> usb0 at uhci0: USB revision 1.0
> uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev
> 1.00/1.00 addr 1
> uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet"
> rev 2.00/0.00 addr 2
> uhidev0: iclass 3/0
> ums0 at uhidev0: 3 buttons, Z dir
> wsmouse1 at ums0 mux 0
> vscsi0 at root
> scsibus3 at vscsi0: 256 targets
> softraid0 at root
> scsibus4 at softraid0: 256 targets
> root on sd0a (69939b6a66c3879a.a) swap on sd0b dump on sd0b
> fd0 at fdc0 drive 1: density unknown
> sd1 at scsibus4 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
> sd1: 15366MB, 512 bytes/sector, 31470807 sectors
>
--
Kind regards,
Hiltjo
