Why is the write performance of my RAID controller so slow? I've noticed
the problem on this server and another Intel Xeon based server with the
same controller. Both are running OpenBSD 4.0 generic. I will focus on
the amd64 box in this email. If this is fixed in current, great, I'll
upgrade. If not, will it be fixed in 4.1?
Here is a basic write test:
(write test running bsd kernel)
# dd if=/dev/zero of=/data/testfile count=20000 bs=128k
20000+0 records in
20000+0 records out
2621440000 bytes transferred in 113.978 secs (22999380 bytes/sec)
(write test running bsd.mp kernel)
# dd if=/dev/zero of=/data/testfile count=20000 bs=128k
20000+0 records in
20000+0 records out
2621440000 bytes transferred in 115.765 secs (22644445 bytes/sec)
Here is some information about the system:
# sysctl hw.sensors
hw.sensors.0=ciss0, sd0, drive online, OK
# bioctl -h ciss0
Volume Status Size Device
ciss0 0 Online 1.4T sd0 RAID5
0 Online 279G 1:0.0 noencl <COMPAQ BD3008A4C6
1 Online 279G 1:1.0 noencl <COMPAQ BD30089BBA
2 Online 279G 1:2.0 noencl <COMPAQ BD30089BBA
3 Online 279G 1:3.0 noencl <COMPAQ BD30089BBA
4 Online 279G 1:4.0 noencl <COMPAQ BD30089BBA
5 Online 279G 1:5.0 noencl <COMPAQ BD30089BBA
# dmesg
OpenBSD 4.0 (GENERIC.MP) #967: Sat Sep 16 20:38:15 MDT 2006
[EMAIL PROTECTED]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4118376448 (4021852K)
avail mem = 3536994304 (3454096K)
using 22937 buffers containing 412045312 bytes (402388K) of memory
mainbus0 (root)
bios0 at mainbus0: SMBIOS rev. 2.3 @ 0xec000 (64 entries)
bios0: HP ProLiant DL385 G1
mainbus0: Intel MP Specification (Version 1.4) (HP PROLIANT )
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD Opteron(tm) Processor 285, 2606.23 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: apic clock running at 200MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: AMD Opteron(tm) Processor 285, 2605.91 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu1: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu1: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu2 at mainbus0: apid 1 (application processor)
cpu2: AMD Opteron(tm) Processor 285, 2605.91 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW
cpu2: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu2: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu2: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu3 at mainbus0: apid 3 (application processor)
cpu3: AMD Opteron(tm) Processor 285, 2605.91 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW
cpu3: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu3: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu3: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
mpbios: bus 0 is type PCI
mpbios: bus 1 is type PCI
mpbios: bus 2 is type PCI
mpbios: bus 3 is type PCI
mpbios: bus 4 is type PCI
mpbios: bus 5 is type PCI
mpbios: bus 6 is type PCI
mpbios: bus 32 is type ISA
ioapic0 at mainbus0 apid 4 pa 0xfec00000, version 11, 24 pins
ioapic1 at mainbus0 apid 5 pa 0xfec10000, version 11, 4 pins
ioapic2 at mainbus0 apid 6 pa 0xfec20000, version 11, 4 pins
ioapic3 at mainbus0 apid 7 pa 0xfdc00000, version 11, 4 pins
ioapic4 at mainbus0 apid 8 pa 0xfdc10000, version 11, 4 pins
pci0 at mainbus0 bus 0: configuration mode 1
ppb0 at pci0 dev 3 function 0 "AMD 8111 PCI-PCI" rev 0x07
pci1 at ppb0 bus 1
ohci0 at pci1 dev 0 function 0 "AMD 8111 USB" rev 0x0b: apic 4 int 19
(irq 5), version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0
uhub0: AMD OHCI root hub, rev 1.00/1.00, addr 1
uhub0: 3 ports with 3 removable, self powered
ohci1 at pci1 dev 0 function 1 "AMD 8111 USB" rev 0x0b: apic 4 int 19
(irq 5), version 1.0, legacy support
usb1 at ohci1: USB revision 1.0
uhub1 at usb1
uhub1: AMD OHCI root hub, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
"Compaq iLO" rev 0x01 at pci1 dev 2 function 0 not configured
"Compaq iLO" rev 0x01 at pci1 dev 2 function 2 not configured
vga1 at pci1 dev 3 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 4 function 0 "AMD AMD8111 LPC" rev 0x05
pciide0 at pci0 dev 4 function 1 "AMD 8111 IDE" rev 0x03: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <TSSTcorp, CD-ROM TS-L162C, N204> SCSI0
5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
amdpm0 at pci0 dev 4 function 3 "AMD 8111 Power" rev 0x05: rng active
iic0 at amdpm0
ppb1 at pci0 dev 7 function 0 "AMD 8131 PCIX" rev 0x12
pci2 at ppb1 bus 2
ciss0 at pci2 dev 4 function 0 "Compaq Smart Array 64xx" rev 0x01: apic
5 int 0 (irq 7)
ciss0: 1 LD, HW rev 1, FW 2.58/2.58
scsibus1 at ciss0: 1 targets
sd0 at scsibus1 targ 0 lun 0: <HP, LOGICAL VOLUME, 2.58> SCSI0 0/direct
fixed
sd0: 1430488MB, 1430488 cyl, 64 head, 32 sec, 512 bytes/sec, 2929640988
sec total
aapic0 at pci0 dev 7 function 1 "AMD 8131 PCIX IOAPIC" rev 0x01
ppb2 at pci0 dev 8 function 0 "AMD 8131 PCIX" rev 0x12
pci3 at ppb2 bus 3
bge0 at pci3 dev 6 function 0 "Broadcom BCM5704C" rev 0x10, BCM5704 B0
(0x2100): apic 6 int 0 (irq 7), address xx:xx:xx:xx:xx:xx
brgphy0 at bge0 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
bge1 at pci3 dev 6 function 1 "Broadcom BCM5704C" rev 0x10, BCM5704 B0
(0x2100): apic 6 int 1 (irq 10), address xx:xx:xx:xx:xx:xx
brgphy1 at bge1 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
aapic1 at pci0 dev 8 function 1 "AMD 8131 PCIX IOAPIC" rev 0x01
pchb0 at pci0 dev 24 function 0 "AMD AMD64 HyperTransport" rev 0x00
pci4 at pchb0 bus 4
ppb3 at pci4 dev 9 function 0 "AMD 8131 PCIX" rev 0x12
pci5 at ppb3 bus 5
aapic2 at pci4 dev 9 function 1 "AMD 8131 PCIX IOAPIC" rev 0x01
ppb4 at pci4 dev 10 function 0 "AMD 8131 PCIX" rev 0x12
pci6 at ppb4 bus 6
aapic3 at pci4 dev 10 function 1 "AMD 8131 PCIX IOAPIC" rev 0x01
pchb1 at pci0 dev 24 function 1 "AMD AMD64 Address Map" rev 0x00
pchb2 at pci0 dev 24 function 2 "AMD AMD64 DRAM Cfg" rev 0x00
pchb3 at pci0 dev 24 function 3 "AMD AMD64 Misc Cfg" rev 0x00
pchb4 at pci0 dev 25 function 0 "AMD AMD64 HyperTransport" rev 0x00
pchb5 at pci0 dev 25 function 1 "AMD AMD64 Address Map" rev 0x00
pchb6 at pci0 dev 25 function 2 "AMD AMD64 DRAM Cfg" rev 0x00
pchb7 at pci0 dev 25 function 3 "AMD AMD64 Misc Cfg" rev 0x00
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
dkcsum: sd0 matches BIOS drive 0x80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
