Hey folks,
I've had a helluva week - my colocated server has crashed at least
four times, and I'd like a little sanity check from people that know
a lot more than I do. Sorry for the length of this, trying to include
all the data I'm aware of that might be relevant and helpful.
For the two crashes that I've been able to capture some output
from (one from an IP KVM, one from /var/log/messages after setting
ddb.panic=0), I've seen:
uvm_fault(0xffffffff81cf2b20, 0xffff800000cef000, 0, 2) -> e
kernel: page fault trap, code=0
Stopped at memmove+0x16: repe movsq (%rsi),%es:(%rdi)
and
reboot after panic: trap type 8, code=0, pc=ffffffff81292dff
Because kernel panics are so rare in OpenBSD, I don't have much
experience debugging them. Following crash(8), I fired up gdb and
took a look at this morning's crash and auto-reboot:
gdb
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-unknown-openbsd5.4".
(gdb) file /var/crash/bsd.0
Reading symbols from /var/crash/bsd.0...(no debugging symbols found)...done.
(gdb) target kvm /var/crash/bsd.0.core
#0 0xffffffff8130a194 in dumpsys ()
(gdb) where
#0 0xffffffff8130a194 in dumpsys ()
#1 0xffffffff8130a2e5 in boot ()
#2 0xffffffff811a2d76 in panic ()
#3 0xffffffff81313d51 in trap ()
#4 0xffffffff81315766 in alltraps ()
#5 0x0000000000000000 in ?? ()
I don't *think* it was resource starvation:
vmstat -N /var/crash/bsd.0 -M /var/crash/bsd.0.core -m
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 46085 47867 10903348 1280 2417
32 3653 571 1604650 640 0
64 4215 1289 2687011 320 18492
128 5405 1411 925024 160 930
256 2066 286 629177 80 74
512 1774 338 462020 40 9397
1024 1539 685 578108 20 141600
2048 287 45 78486 10 21570
4096 83 528 144485 5 101528
8192 20 8 18105 5 7483
16384 1 0 366 5 0
32768 8 0 102 5 0
65536 2 0 1909341 5 0
524288 2 0 2 5 0
Memory usage type by bucket size
Size Type(s)
16 devbuf, pcb, routetbl, sem, dirhash, ACPI, exec, UVM amap, UVM
aobj,
USB, USB device, temp
32 devbuf, pcb, routetbl, ifaddr, sysctl, vnodes, sem, dirhash, ACPI,
in_multi, exec, UVM amap, USB, temp
64 devbuf, routetbl, ifaddr, vnodes, UFS mount, dirhash, ACPI, proc,
VFS cluster, in_multi, ether_multi, VM swap, UVM amap, USB,
USB device, NDP, temp
128 devbuf, pcb, routetbl, sysctl, UFS mount, sem, dirhash, ACPI,
NFS srvsock, ttys, pfkey data, inodedep, VM swap, UVM amap, USB,
USB device, USB HC, NDP, temp
256 devbuf, routetbl, ifaddr, ioctlops, vnodes, UFS mount, shm, VM map,
sem, dirhash, ACPI, exec, xform_data, UVM amap, USB, USB device,
temp
512 devbuf, routetbl, ifaddr, ioctlops, sem, dirhash, ACPI, file desc,
NFS daemon, ttys, xform_data, newblk, UVM amap, USB, temp
1024 devbuf, pcb, sysctl, ioctlops, mount, UFS mount, shm, dirhash,
ACPI,
file desc, proc, ttys, exec, UVM amap, crypto data, temp
2048 devbuf, ioctlops, UFS mount, sem, dirhash, ACPI, file desc, VM
swap,
UVM amap, UVM aobj, temp
4096 devbuf, ifaddr, ioctlops, UFS mount, shm, dirhash, file desc, proc,
UVM amap, memdesc, temp
8192 devbuf, file, ttys, pagedep, UVM amap, USB, temp
16384 devbuf, MSDOSFS mount, indirdep, temp
32768 devbuf, UFS quota, UFS mount, ISOFS mount, inodedep, indirdep,
NTFS hash
65536 devbuf, temp
524288 VM swap
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
devbuf 733 495K 2597K 78644K 23287 0 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
pcb 218 34K 42K 78644K 40723 0 0
16,32,128,1024
routetbl 78 9K 10K 78644K 4198 0 0
16,32,64,128,256,512
ifaddr 56 16K 16K 78644K 58 0 0
32,64,256,512,4096
sysctl 3 2K 2K 78644K 3 0 0 32,128,1024
ioctlops 0 0K 4K 78644K 4632 0 0
256,512,1024,2048,4096
mount 13 13K 13K 78644K 13 0 0 1024
vnodes 37 3K 72K 78644K 3344 0 0 32,64,256
UFS quota 1 32K 32K 78644K 1 0 0 32768
UFS mount 53 111K 111K 78644K 53 0 0
64,128,256,1024,2048,4096,32768
shm 20 74K 86K 78644K 19131 0 0 256,1024,4096
VM map 2 1K 1K 78644K 2 0 0 256
sem 14 5K 5K 78644K 16 0 0
16,32,128,256,512,2048
dirhash 1612 503K 670K 78644K 49359 0 0
16,32,64,128,256,512,1024,2048,4096
ACPI 2449 292K 305K 78644K 8406 0 0
16,32,64,128,256,512,1024,2048
file 0 0K 8K 78644K 87 0 0 8192
file desc 63 73K 112K 78644K 10748 0 0
512,1024,2048,4096
proc 22 11K 11K 78644K 22 0 0 64,1024,4096
VFS cluster 0 0K 1K 78644K 91814 0 0 64
NFS srvsock 1 1K 1K 78644K 1 0 0 128
NFS daemon 1 1K 1K 78644K 1 0 0 512
in_multi 24 2K 2K 78644K 24 0 0 32,64
ether_multi 4 1K 1K 78644K 4 0 0 64
ISOFS mount 1 32K 32K 78644K 1 0 0 32768
MSDOSFS mount 1 16K 16K 78644K 1 0 0 16384
ttys 420 308K 308K 78644K 420 0 0
128,512,1024,8192
exec 0 0K 4K 78644K 79942 0 0
16,32,256,1024
pfkey data 1 1K 1K 78644K 2 0 0 128
xform_data 0 0K 4K 78644K 744 0 0 256,512
pagedep 1 8K 8K 78644K 1 0 0 8192
inodedep 1 32K 121K 78644K 2779 0 0 128,32768
newblk 1 1K 1K 78644K 1 0 0 512
indirdep 0 0K 64K 78644K 399 0 0 16384,32768
VM swap 7 723K 723K 78644K 7 0 0
64,128,2048,524288
UVM amap 59056 3913K 4819K 78644K 15666014 0 0
16,32,64,128,256,512,1024,2048,4096,8192
UVM aobj 3 5K 5K 78644K 3 0 0 16,2048
USB 135 40K 40K 78644K 136 0 0
16,32,64,128,256,512,8192
USB device 31 3K 3K 78644K 31 0 0 16,64,128,256
USB HC 1 1K 1K 78644K 1 0 0 128
memdesc 1 4K 4K 78644K 1 0 0 4096
crypto data 1 1K 1K 78644K 1 0 0 1024
NDP 7 1K 1K 78644K 8 0 0 64,128
temp 67 58K 250K 78644K 3933805 0 0
16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
NTFS hash 1 32K 32K 78644K 1 0 0 32768
Memory Totals: In Use Free Requests
6842K 4215K 19940225
Memory resource pool statistics
Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg
Maxpg Idle
inpcbpl 360 40934 0 49 9 0 9 9 0
8 3
plimitpl 152 5302 0 47 3 0 3 3 0
8 1
synpl 248 20216 0 0 2 0 2 2 0
8 2
sackhlpl 24 5337 0 0 1 0 1 1 0
8 1
tcpqepl 32 18281 0 0 1 0 1 1 0
8 1
tcpcbpl 560 27685 0 42 13 0 13 13 0
8 5
rttmrpl 72 314 0 0 1 0 1 1 0
8 1
rtentpl 200 247 0 34 3 0 3 3 0
8 1
strpolpl 48 372 0 0 1 0 1 1 0
8 1
strprocpl 2456 744 0 0 26 18 8 10 0
8 8
pfosfp 40 840 0 420 5 0 5 5 0
8 0
pfosfpen 112 1420 0 710 34 13 21 21 0
8 0
pfrke_plain 160 48 0 48 2 0 2 2 0
8 0
pfrktable 1344 4 0 2 2 0 2 2 0
8 1
pfstateitempl 24 26514 0 52 2 0 2 2 0
8 1
pfstatekeypl 104 26514 0 52 7 0 7 7 0
8 5
pfstatepl 312 26514 0 52 21 11 10 21 0
8 6
pfsnitempl 16 445 0 2 1 0 1 1 0
8 0
pfsrctrpl 152 25 0 1 1 0 1 1 0
8 0
pfrulepl 1328 36 0 25 12 0 12 12 0
8 3
semupl 112 7095 0 2 1 0 1 1 0
8 0
semapl 112 12 0 12 1 0 1 1 0
8 0
shmpl 112 1 0 1 1 0 1 1 0
8 0
dirhash 1024 399977 0 1691 44652 44227 425 497 0
8 2
newdirblkpl 32 3633 0 0 3 0 3 3 0
8 3
dirrempl 64 196471 0 4 2432 2424 8 345 0
8 7
mkdirpl 56 11676 0 2 74 66 8 39 0
8 7
diraddpl 56 216316 0 5 1808 1800 8 388 0
8 7
freefilepl 48 14733 0 3 23 15 8 16 0
8 7
freeblkspl 192 15081 0 4 250 242 8 80 0
8 7
freefragpl 64 3195 0 1 1 0 1 1 0
8 0
allocindirpl 104 116476 0 0 2758 2750 8 346 0
8 8
indirdeppl 56 630 0 0 1 0 1 1 0
8 1
allocdirectpl 128 30713 0 5 139 131 8 78 0
8 7
bmsafemappl 64 5468 0 3 1 0 1 1 0
8 0
newblkpl 64 147189 0 0 1 0 1 1 0
8 1
inodedeppl 152 396484 0 9 9916 9908 8 1313 0
8 7
pagedeppl 128 13863 0 5 125 117 8 55 0
8 7
dino1pl 128 10011177 0 40593 1310 0 1310 1310 0
8 0
ffsino 216 10011177 0 40593 2260 4 2256 2256 0
8 0
nchpl 144 1421690 0 5920 220 0 220 220 0
8 0
vnodes 264 40610 0 40610 2708 0 2708 2708 0
8 0
namei 1024 16704896 0 0 2 0 2 2 0
8 2
scxspl 192 4523636 0 2 2 0 2 2 0
8 1
ifaddritempl 64 19 0 19 1 0 1 1 0
8 0
wqtasks 40 38688 0 11 1 0 1 1 0
8 0
sigapl 432 81820 0 108 16 0 16 16 0
8 3
knotepl 112 59123 0 224 10 0 10 10 0
8 3
kqueuepl 320 5077 0 25 4 0 4 4 0
8 1
pipepl 120 112374 0 189 8 0 8 8 0
8 2
fdescpl 440 81821 0 109 16 0 16 16 0
8 3
filepl 120 2726118 0 800 31 0 31 31 0
8 5
lockfpl 88 35919 0 6 1 0 1 1 0
8 0
pcredpl 24 81833 0 122 1 0 1 1 0
8 0
sessionpl 64 20349 0 44 1 0 1 1 0
8 0
pgrppl 40 29027 0 46 1 0 1 1 0
8 0
ucredpl 80 36657 0 122 3 0 3 3 0
8 0
zombiepl 144 81714 0 3 1 0 1 1 0
8 0
processpl 640 81833 0 122 26 0 26 26 0
8 5
procpl 616 83493 0 128 27 0 27 27 0
8 5
sockpl 472 81644 0 256 45 7 38 41 0
8 4
mcl2k 2048 1569909 0 3 134 0 134 134 4
3072 132
mbpl 256 10056711 0 117 64 0 64 64 1
384 52
bufpl 304 7009052 0 33823 51326 48719 2607 3124 0
8 0
anonpl 16 37902106 0 142101 801 15 786 786 0
253 137
amappl 72 15383835 0 58157 3565 2253 1312 1942 0
75 75
dma8192 8192 1 0 0 1 0 1 1 0
8 1
dma4096 4096 1 0 0 1 0 1 1 0
8 1
dma2048 2048 19589 0 0 1 0 1 1 0
8 1
dma1024 1024 19588 0 0 1 0 1 1 0
8 1
dma512 512 10859 0 1 1 0 1 1 0
8 0
dma256 256 9 0 0 1 0 1 1 0
8 1
dma64 64 17 0 0 1 0 1 1 0
8 1
dma32 32 8 0 0 1 0 1 1 0
8 1
dma16 16 3 0 1 1 0 1 1 0
8 0
aobjpl 64 2 0 2 1 0 1 1 0
8 0
uaddrrnd 40 81821 0 109 2 0 2 2 0
8 0
uaddrbestfit 32 2 0 2 1 0 1 1 0
8 0
uaddr 24 81821 0 109 1 0 1 1 0
8 0
vmmpekpl 168 891340 0 86 6 0 6 6 0
8 1
vmmpepl 168 34504561 0 97037 10173 5441 4732 6331 0
357 357
vmsppl 224 81820 0 108 9 0 9 9 0
8 2
pdppl 4096 81820 0 108 3616 3503 113 141 0
8 5
pvpl 32 96993649 0 309758 3570 675 2895 2912 0
265 261
pmappl 152 81820 0 108 6 0 6 6 0
8 1
extentpl 40 114 0 51 1 0 1 1 0
8 0
phpool 104 102773 0 6288 181 0 181 181 0
8 0
In use 70500K, total allocated 80816K; utilization 87.2%
The reason I am suspecting hardware is because I have run -CURRENT
snapshots on this system forever, and this is suddenly rearing its
ugly head. I was running a snapshot from February when it crashed the
first time a week from yesterday, so I went ahead and updated to
"OpenBSD 5.4-current (GENERIC.MP) #54: Tue Sep 10 17:21:35 MDT 2013".
It has since crashed another three times, which is why I doubt it's a
software issue.
Could someone that has some more knowledge about kernel panics take
a look and make a suggestion if possible? Would you suspect RAM? CPU?
Some other hardware bit? I didn't really see anything obvious to a
kernel novice other than 'memmove', so I would appreciate any and all
help you could give me to help track this down (and perhaps point the
hosting company towards a failing component if that's what it turns out
to be).
And the dmesg:
OpenBSD 5.4-current (GENERIC.MP) #54: Tue Sep 10 17:21:35 MDT 2013
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4242931712 (4046MB)
avail mem = 4121878528 (3930MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xcfedf000 (39 entries)
bios0: vendor Phoenix Technologies LTD version "1.2a" date 12/19/2008
bios0: Supermicro X7SBL
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP _MAR TCPA MCFG HPET APIC BOOT SPCR ERST HEST BERT
EINJ SLIC SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT
acpi0: wakeup devices PEG_(S5) PEX_(S5) LAN_(S5) USB4(S5) USB5(S5)
USB7(S5) ESB2(S5) EXP1(S5) EXP5(S5) EXP6(S5) USB1(S5) USB2(S5) USB3(S5)
USB6(S5) ESB1(S5) PCIB(S5) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-16
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Pentium(R) Dual-Core CPU E5400 @ 2.70GHz, 2693.65 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,XSAVE,NXE,LONG,LAHF,PERF
cpu0: 2MB 64b/line 8-way L2 cache
cpu0: apic clock running at 199MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Pentium(R) Dual-Core CPU E5400 @ 2.70GHz, 2693.25 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,XSAVE,NXE,LONG,LAHF,PERF
cpu1: 2MB 64b/line 8-way L2 cache
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEG_)
acpiprt2 at acpi0: bus -1 (PEX_)
acpiprt3 at acpi0: bus 5 (EXP1)
acpiprt4 at acpi0: bus 13 (EXP5)
acpiprt5 at acpi0: bus 15 (EXP6)
acpiprt6 at acpi0: bus 17 (PCIB)
acpicpu0 at acpi0: C1, PSS
acpicpu1 at acpi0: C1, PSS
acpibtn0 at acpi0: PWRB
acpivideo0 at acpi0: IGD0
ipmi at mainbus0 not configured
cpu0: Enhanced SpeedStep 2693 MHz: speeds: 2700, 2400, 2000, 1600, 1200 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 3200/3210 Host" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel 3200/3210 PCIE" rev 0x01: msi
pci1 at ppb0 bus 1
mfi0 at pci1 dev 0 function 0 "Symbios Logic MegaRAID SAS2008" rev 0x03:
apic 2 int 16
mfi0: "LSI MegaRAID SAS 9240-4i", firmware 20.10.1-0107
scsibus0 at mfi0: 16 targets
sd0 at scsibus0 targ 0 lun 0: <LSI, MR9240-4i, 2.13> SCSI3 0/direct fixed
naa.600605b004be9ab017e570f461b13ae3
sd0: 475883MB, 512 bytes/sector, 974608384 sectors
scsibus1 at mfi0: 256 targets
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 2 int 16
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 2 int 17
uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x02: apic 2 int 18
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 2 int 18
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb1 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02: msi
pci2 at ppb1 bus 5
ppb2 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02: msi
pci3 at ppb2 bus 13
em0 at pci3 dev 0 function 0 "Intel 82573E" rev 0x03: msi, address
00:30:48:fb:cf:9e
ppb3 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02: msi
pci4 at ppb3 bus 15
em1 at pci4 dev 0 function 0 "Intel 82573L" rev 0x00: msi, address
00:30:48:fb:cf:9f
uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 2 int 23
uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 2 int 22
uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 2 int 18
ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 2 int 23
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb4 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci5 at ppb4 bus 17
vga1 at pci5 dev 4 function 0 "XGI Technology Volari Z9s/Z9m" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.2
scsibus2 at ahci0: 32 targets
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 2
int 17
iic0 at ichiic0
lm1 at iic0 addr 0x2d: W83627HF
wbng0 at iic0 addr 0x2f: w83793g
spdmem0 at iic0 addr 0x50: 2GB DDR2 SDRAM non-parity PC2-5300CL5
spdmem1 at iic0 addr 0x52: 2GB DDR2 SDRAM non-parity PC2-5300CL5
"Intel 82801I Thermal" rev 0x02 at pci0 dev 31 function 6 not configured
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb6 at uhci4: USB revision 1.0
uhub6 at usb6 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb7 at uhci5: USB revision 1.0
uhub7 at usb7 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
Thanks much!
Benny
--
"No matter how tempted I am with the prospect of unlimited power, I
will not consume any energy field bigger than my head."
-- #22 on Peter Anspach's Evil
Overlord list
