On Mon, Jul 25, 2011 at 5:54 PM, Mattieu Baptiste <[email protected]> wrote: > Hi all, > > It's really weird. I see very slow writes when softdep is enabled. > Write caches seem to be enabled on disks. Utilities like bonnie++ > confirm that writes are very slow with softdep enabled (although it > helps with lots of file creations). This happens whether I use native > disks or RAID 1 volume behind mpi(4) (on which the controller's cache > is also enabled).
After digging around this problem with help from Pedro, I noticed that a 4.9-release kernel works ok, whereas a 5.0-beta does not. So I tried to identify which commit was responsible and found out this one is responsible of *very* slow writes with softdep (I only have this bug on sparc64): http://marc.info/?l=openbsd-cvs&m=130730313107059&w=2 I can add that this bug is just affecting softdep: - dd'ing to the raw device is fast, - dd'ing to the file system mounted sync is very fast, - dd'ing to the file system mounted async is very fast, - dd'ing to the file system mounted with softdep is very slow. I can also add that changing a working kernel to bufcachepercent >=42 via sysctl is ok too. > > With softdep: > # mount -o softdep /dev/sd2a /mnt > # dd if=/dev/zero of=/mnt/test > # dd if=/dev/zero of=/mnt/test.dump bs=4k count=10000 > 10000+0 records in > 10000+0 records out > 40960000 bytes transferred in 16.060 secs (2550321 bytes/sec) > # rm /mnt/test.dump > # dd if=/dev/zero of=/mnt/test.dump bs=1m count=100 > 100+0 records in > 100+0 records out > 104857600 bytes transferred in 3.483 secs (30101713 bytes/sec) > > Without softdep: > # mount /dev/sd2a /mnt > # dd if=/dev/zero of=/mnt/test.dump bs=4k count=10000 > 10000+0 records in > 10000+0 records out > 40960000 bytes transferred in 0.330 secs (123880958 bytes/sec) > # rm /mnt/test.dump > # dd if=/dev/zero of=/mnt/test.dump bs=1m count=100 > 100+0 records in > 100+0 records out > 104857600 bytes transferred in 0.932 secs (112438785 bytes/sec) > > # scsi -f /dev/rsd2c -m8 > IC: 1 > ABPF: 0 > CAP: 0 > DISC: 1 > SIZE: 0 > WCE: 1 > MF: 0 > RCD: 0 > Demand Retention Priority: 0 > Write Retention Priority: 0 > Disable Pre-fetch Transfer Length: 65535 > Minimum Pre-fetch: 0 > Maximum Pre-fetch: 65535 > Maximum Pre-fetch Ceiling: 65535 > FSW: 0 > LBCSS: 0 > DRA: 0 > Vendor-specific: 0 > NV_DIS: 0 > Number of Cache Segments: 2 > Cache Segment Size: 0 > > > dmesg: > console is /pci@1e,600000/isa@7/serial@0,3f8 > Copyright (c) 1982, 1986, 1989, 1991, 1993 > The Regents of the University of California. All rights reserved. > Copyright (c) 1995-2011 OpenBSD. All rights reserved. http://www.OpenBSD.org > > OpenBSD 5.0-beta (GENERIC.MP) #3: Tue Jul 19 18:20:17 CEST 2011 > [email protected]:/usr/src/sys/arch/sparc64/compile/GENERIC.MP > real mem = 17179869184 (16384MB) > avail mem = 16901693440 (16118MB) > mainbus0 at root: Sun Fire V440 > cpu0 at mainbus0: SUNW,UltraSPARC-IIIi (rev 2.4) @ 1281 MHz > cpu0: physical 32K instruction (32 b/l), 64K data (32 b/l), 1024K > external (64 b/l) > cpu1 at mainbus0: SUNW,UltraSPARC-IIIi (rev 2.4) @ 1281 MHz > cpu1: physical 32K instruction (32 b/l), 64K data (32 b/l), 1024K > external (64 b/l) > cpu2 at mainbus0: SUNW,UltraSPARC-IIIi (rev 2.4) @ 1281 MHz > cpu2: physical 32K instruction (32 b/l), 64K data (32 b/l), 1024K > external (64 b/l) > cpu3 at mainbus0: SUNW,UltraSPARC-IIIi (rev 2.4) @ 1281 MHz > cpu3: physical 32K instruction (32 b/l), 64K data (32 b/l), 1024K > external (64 b/l) > "memory-controller" at mainbus0 not configured > "memory-controller" at mainbus0 not configured > "memory-controller" at mainbus0 not configured > "memory-controller" at mainbus0 not configured > schizo0 at mainbus0: "Tomatillo", version 4, ign 700, bus A 0 to 0 > schizo0: dvma map c0000000-dfffffff > pci0 at schizo0 > cas0 at pci0 dev 2 function 0 "Sun Cassini" rev 0x20: ivec 0x718, > address 00:03:ba:a4:90:53 > brgphy0 at cas0 phy 1: BCM5421 10/100/1000baseT PHY, rev. 1 > "ppm" at mainbus0 not configured > schizo1 at mainbus0: "Tomatillo", version 4, ign 740, bus B 0 to 1 > schizo1: dvma map c0000000-dfffffff > pci1 at schizo1 > ppb0 at pci1 dev 2 function 0 "Intel 21154AE/BE PCI-PCI" rev 0x00 > pci2 at ppb0 bus 1 > cas1 at pci2 dev 0 function 0 "Sun Cassini" rev 0x20: ivec 0x740, > address 00:03:ba:93:1c:a1 > brgphy1 at cas1 phy 1: BCM5401 10/100/1000baseT PHY, rev. 3 > schizo2 at mainbus0: "Tomatillo", version 4, ign 780, bus A 0 to 0 > schizo2: dvma map c0000000-dfffffff > pci3 at schizo2 > ebus0 at pci3 dev 7 function 0 "Acer Labs M1533 ISA" rev 0x00 > "flashprom" at ebus0 addr 0-fffff, 290-290 not configured > rtc0 at ebus0 addr 70-71: m5819p > pcfiic0 at ebus0 addr 320-321 ivec 0x1b > iic0 at pcfiic0 > "SUNW,i2c-imax" at iic0 addr 0xb not configured > "SUNW,i2c-imax" at iic0 addr 0xc not configured > admtemp0 at iic0 addr 0x18: max1617 > "pca9555" at iic0 addr 0x21 not configured > "pca9555" at iic0 addr 0x22 not configured > "pca9555" at iic0 addr 0x23 not configured > "pca9555" at iic0 addr 0x24 not configured > "adm1026" at iic0 addr 0x2e not configured > admtemp1 at iic0 addr 0x32: max1617 > admtemp2 at iic0 addr 0x40: max1617 > admtemp3 at iic0 addr 0x48: max1617 > lmtemp0 at iic0 addr 0x4e: lm75 > spdmem0 at iic0 addr 0x5b: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem1 at iic0 addr 0x5c: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem2 at iic0 addr 0x5d: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem3 at iic0 addr 0x5e: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem4 at iic0 addr 0x63: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem5 at iic0 addr 0x64: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem6 at iic0 addr 0x65: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem7 at iic0 addr 0x66: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem8 at iic0 addr 0x6b: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem9 at iic0 addr 0x6c: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem10 at iic0 addr 0x6d: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem11 at iic0 addr 0x6e: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem12 at iic0 addr 0x73: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem13 at iic0 addr 0x74: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem14 at iic0 addr 0x75: 1GB DDR SDRAM registered ECC PC2300CL2.5 > spdmem15 at iic0 addr 0x76: 1GB DDR SDRAM registered ECC PC2300CL2.5 > "ics951601" at iic0 addr 0x69 not configured > power0 at ebus0 addr 800-82f ivec 0x1a > com0 at ebus0 addr 3f8-3ff ivec 0x22: ns16550a, 16 byte fifo > com0: console > com1 at ebus0 addr 2e8-2ef ivec 0x22: ns16550a, 16 byte fifo > "rmc-comm" at ebus0 addr 3e8-3ef ivec 0x22 not configured > alipm0 at pci3 dev 6 function 0 "Acer Labs M7101 Power" rev 0x00: 223KHz clock > iic1 at alipm0 > ohci0 at pci3 dev 10 function 0 "Acer Labs M5237 USB" rev 0x03: ivec > 0x7a1, version 1.0, legacy support > ohci1 at pci3 dev 11 function 0 "Acer Labs M5237 USB" rev 0x03: ivec > 0x7a5, version 1.0, legacy support > pciide0 at pci3 dev 13 function 0 "Acer Labs M5229 UDMA IDE" rev 0xc4: > DMA, channel 0 configured to native-PCI, channel 1 configured to > native-PCI > pciide0: using ivec 0x7a6 for native-PCI interrupt > atapiscsi0 at pciide0 channel 0 drive 0 > scsibus0 at atapiscsi0: 2 targets > cd0 at scsibus0 targ 0 lun 0: <TOSHIBA, DVD-ROM SD-C2612, 1011> ATAPI > 5/cdrom removable > cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 > pciide0: channel 1 disabled (no drives) > usb0 at ohci0: USB revision 1.0 > uhub0 at usb0 "Acer Labs OHCI root hub" rev 1.00/1.00 addr 1 > usb1 at ohci1: USB revision 1.0 > uhub1 at usb1 "Acer Labs OHCI root hub" rev 1.00/1.00 addr 1 > "ppm" at mainbus0 not configured > schizo3 at mainbus0: "Tomatillo", version 4, ign 7c0, bus B 0 to 0 > schizo3: dvma map c0000000-dfffffff > pci4 at schizo3 > cas2 at pci4 dev 1 function 0 "Sun Cassini" rev 0x20: ivec 0x7dc, > address 00:03:ba:a4:90:54 > brgphy2 at cas2 phy 1: BCM5421 10/100/1000baseT PHY, rev. 1 > mpi0 at pci4 dev 2 function 0 "Symbios Logic 53c1030" rev 0x07: ivec 0x7e3 > scsibus1 at mpi0: 16 targets, initiator 7 > sd0 at scsibus1 targ 0 lun 0: <LSILOGIC, 1030 IM IM, 1000> SCSI2 0/direct fixed > sd0: 70007MB, 512 bytes/sector, 143374336 sectors > sd1 at scsibus1 targ 2 lun 0: <SEAGATE, ST314670LSUN146G, 065A> SCSI3 > 0/direct fixed t10.SEAGATE_ST314670LSUN146G3KS73HA9 > sd1: 140009MB, 512 bytes/sector, 286739329 sectors > sd2 at scsibus1 targ 3 lun 0: <SEAGATE, ST314670LSUN146G, 065A> SCSI3 > 0/direct fixed t10.SEAGATE_ST314670LSUN146G3KS73JGX > sd2: 140009MB, 512 bytes/sector, 286739329 sectors > mpi0: target 2 Sync at 160MHz width 16bit offset 63 QAS 1 DT 1 IU 1 > mpi0: target 3 Sync at 160MHz width 16bit offset 63 QAS 1 DT 1 IU 1 > mpi0: phys disk 0 Sync at 160MHz width 16bit offset 63 QAS 1 DT 1 IU 1 > mpi0: phys disk 1 Sync at 160MHz width 16bit offset 63 QAS 1 DT 1 IU 1 > mpi1 at pci4 dev 2 function 1 "Symbios Logic 53c1030" rev 0x07: ivec 0x7e4 > scsibus2 at mpi1: 16 targets, initiator 7 > "i2c" at mainbus0 not configured > vscsi0 at root > scsibus3 at vscsi0: 256 targets > softraid0 at root > scsibus4 at softraid0: 256 targets > bootpath: /pci@1f,700000/scsi@2,0/disk@0,0 > root on sd0a swap on sd0b dump on sd0b > > > -- > Mattieu Baptiste > "/earth is 102% full ... please delete anyone you can." > -- Mattieu Baptiste "/earth is 102% full ... please delete anyone you can."
