Now, the -current SP kernel, while triggering, has a better speed and dies a
bit later with:
kernel: type -1342060160 trap, code 0
Stopped at 0x3deba01b052507f:
at this point I expected it to drop to ddb, but it didn't.
my sysctl.conf says:
ddb.panic=1 # 0=Do not drop into ddb on a kernel panic
ddb.console=1 # 1=Permit entry of ddb from the console
dmesg:
OpenBSD 5.2-current (GENERIC) #0: Thu Jan 3 22:06:40 CET 2013
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 4284039168 (4085MB)
avail mem = 4147527680 (3955MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xeadc0 (105 entries)
bios0: vendor American Megatrends Inc. version "4.6.4" date 06/30/2011
bios0: Supermicro X9SCL/X9SCM
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC SSDT MCFG HPET SPMI EINJ ERST HEST BERT
acpi0: wakeup devices PS2K(S1) PS2M(S1) UAR1(S4) UAR2(S4) BR20(S1) EUSB(S4)
USBE(S4) PEX0(S4) PEX4(S4) PEX6(S4) GBE_(S4) P0P1(S4) P0P2(S4) P0P3(S4)
P0P4(S4) SLPB(S0) PWRB(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E31220L @ 2.20GHz, 2195.33 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: apic clock running at 99MHz
cpu at mainbus0: not configured
cpu at mainbus0: not configured
cpu at mainbus0: not configured
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 7 (BR20)
acpiprt2 at acpi0: bus 5 (PEX0)
acpiprt3 at acpi0: bus 6 (PEX4)
acpiprt4 at acpi0: bus -1 (PEX6)
acpiprt5 at acpi0: bus 1 (P0P1)
acpiprt6 at acpi0: bus 2 (P0P2)
acpiprt7 at acpi0: bus -1 (P0P3)
acpiprt8 at acpi0: bus -1 (P0P4)
acpicpu0 at acpi0: C3, C1, PSS
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
ipmi at mainbus0 not configured
cpu0: Enhanced SpeedStep 2195 MHz: speeds: 2201, 2200, 2100, 2000, 1900, 1800,
1700, 1600 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 Host" rev 0x09
ppb0 at pci0 dev 1 function 0 "Intel Core 2G PCIE" rev 0x09: msi
pci1 at ppb0 bus 1
ppb1 at pci0 dev 1 function 1 "Intel Core 2G PCIE" rev 0x09: msi
pci2 at ppb1 bus 2
ppb2 at pci2 dev 0 function 0 "Intel 41210 PCIE-PCIX" rev 0x09
pci3 at ppb2 bus 3
bnx0 at pci3 dev 4 function 0 "Broadcom BCM5706" rev 0x02: apic 0 int 17
ppb3 at pci2 dev 0 function 2 "Intel 41210 PCIE-PCIX" rev 0x09
pci4 at ppb3 bus 4
bnx1 at pci4 dev 5 function 0 "Broadcom BCM5706" rev 0x02: apic 0 int 18
em0 at pci0 dev 25 function 0 "Intel 82579LM" rev 0x05: msi, address
00:25:90:27:da:85
ehci0 at pci0 dev 26 function 0 "Intel 6 Series USB" rev 0x05: apic 0 int 16
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb4 at pci0 dev 28 function 0 "Intel 6 Series PCIE" rev 0xb5: msi
pci5 at ppb4 bus 5
ppb5 at pci0 dev 28 function 4 "Intel 6 Series PCIE" rev 0xb5: msi
pci6 at ppb5 bus 6
em1 at pci6 dev 0 function 0 "Intel PRO/1000 MT (82574L)" rev 0x00: msi,
address 00:25:90:27:da:84
ehci1 at pci0 dev 29 function 0 "Intel 6 Series USB" rev 0x05: apic 0 int 23
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb6 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa5
pci7 at ppb6 bus 7
vga1 at pci7 dev 3 function 0 "Matrox MGA G200eW" rev 0x0a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel C204 LPC" rev 0x05
ahci0 at pci0 dev 31 function 2 "Intel 6 Series AHCI" rev 0x05: msi, AHCI 1.3
scsibus0 at ahci0: 32 targets
sd0 at scsibus0 targ 0 lun 0: <ATA, INTEL SSDSA2CT04, 4PC1> SCSI3 0/direct
fixed naa.500151795956a9d6
sd0: 38166MB, 512 bytes/sector, 78165360 sectors, thin
ichiic0 at pci0 dev 31 function 3 "Intel 6 Series SMBus" rev 0x05: apic 0 int 18
iic0 at ichiic0
sdtemp0 at iic0 addr 0x19: se97
sdtemp1 at iic0 addr 0x1b: se97
spdmem0 at iic0 addr 0x51: 2GB DDR3 SDRAM ECC PC3-10600 with thermal sensor
spdmem1 at iic0 addr 0x53: 2GB DDR3 SDRAM ECC PC3-10600 with thermal sensor
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
wbsio0 at isa0 port 0x2e/2: NCT6776F rev 0x33
lm1 at wbsio0 port 0xa30/8: NCT6776F
mtrr: Pentium Pro MTRR support
uhub2 at uhub0 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
uhidev0 at uhub2 port 2 configuration 1 interface 0 "Winbond Electronics Corp
Hermon USB hidmouse Device" rev 1.10/0.01 addr 3
uhidev0: iclass 3/1
ums0 at uhidev0: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
uhidev1 at uhub2 port 2 configuration 1 interface 1 "Winbond Electronics Corp
Hermon USB hidmouse Device" rev 1.10/0.01 addr 3
uhidev1: iclass 3/1
ukbd0 at uhidev1: 8 variable keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhub3 at uhub1 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on sd0a (48824925dcb6ebdf.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
bnx0: address 00:14:c2:5d:31:a8
brgphy0 at bnx0 phy 1: BCM5706 10/100/1000baseT/SX PHY, rev. 2
bnx1: address 00:14:c2:5d:31:aa
brgphy1 at bnx1 phy 1: BCM5706 10/100/1000baseT/SX PHY, rev. 2
carp: carp1 demoted group carp by 1 to 129 (carpdev)
carp: carp2 demoted group carp by 1 to 130 (carpdev)
carp: carp3 demoted group carp by 1 to 131 (carpdev)
carp: carp4 demoted group carp by 1 to 132 (carpdev)
carp: pfsync0 demoted group carp by 32 to 164 (pfsync init)
carp: pfsync0 demoted group pfsync by 32 to 32 (pfsync init)
carp: pfsync0 demoted group carp by 1 to 165 (pfsync bulk start)
carp: pfsync0 demoted group pfsync by 1 to 33 (pfsync bulk start)
carp: carp2 demoted group carp by -1 to 164 (carpdev)
carp: carp3 demoted group carp by -1 to 163 (carpdev)
carp: carp4 demoted group carp by -1 to 162 (carpdev)
carp: carp1 demoted group carp by -1 to 161 (carpdev)
carp2: state transition: BACKUP -> MASTER
carp3: state transition: BACKUP -> MASTER
carp4: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp2: state transition: MASTER -> BACKUP
carp3: state transition: MASTER -> BACKUP
carp4: state transition: MASTER -> BACKUP
carp1: state transition: MASTER -> BACKUP
carp: pfsync0 demoted group carp by -1 to 33 (pfsync bulk done)
carp: pfsync0 demoted group pfsync by -1 to 32 (pfsync bulk done)
carp: pfsync0 demoted group carp by -32 to 1 (pfsync init)
carp: pfsync0 demoted group pfsync by -32 to 0 (pfsync init)
carp2: state transition: BACKUP -> MASTER
carp3: state transition: BACKUP -> MASTER
carp4: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
On 3 jan 2013, at 20:48, mxb <[email protected]> wrote:
>
> Here is an older kernel which seems to die the same way.
> I actually can not see mtx_enter-loop, but I trigger crash the same way.
> Remote console via IPMI just blinks like a X-mass tree.
>
> OpenBSD 5.2-current (GENERIC.MP) #0: Tue Nov 6 17:03:43 CET 2012
> [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
>
> On 3 jan 2013, at 20:15, mxb <[email protected]> wrote:
>
>>
>> Now, after several tests I can state that problem is there.
>> I made sure that /usr/src is clean and up to date (I had Hennings diffs on
>> test).
>>
>> The stock -current kernel crashes with this behavior, eg. panic and loop
>> with mtx_enter on console.
>> This bug is triggered then I transfer /bsd over IPSec. While scp from the
>> same machine, but on public side goes fine.
>>
>> client (network1)<-----> fw1 ---> [---- ospf over gre over ipsec] <----- fw2
>> <-------->network2
>>
>> client does 'scp fw2.network2_ip:/bsd .' - results in panic.
>> client does 'scp fw2.public_ip:/bsd .' - all fine.
>>
>> Of cause I have CARP both on external and internal sides of fw2.
>> CARP for internal network on fw1.
>>
>> Side note is that I noticed drastic speed drop just before system goes in
>> panic.
>> Normally I have decent speed between two networks, e.g. transfers from
>> clients on network2 to client on network1.
>>
>> Any ideas?
>>
>> //mxb
>>
>> On 3 jan 2013, at 13:39, Stuart Henderson <[email protected]> wrote:
>>
>>> On 2013-01-03, mxb <[email protected]> wrote:
>>>> Sorry for the noise. I think I'v found the problem.
>>>
>>> Any more information on what it was??
>>> Even if it was something you've done, that should't happen..
>>>
>>>
>>>
>>>>> On 1 jan 2013, at 19:11, mxb <[email protected]> wrote:
>>>>>>
>>>>>> I'v got yet another panic.
>>>>>> This time, after applying Martin Pelikans' diff, catched a pointer.
>>>>>> However, machine never drops to ddb, even sysctl.conf says it should.
>>>>>>
>>>>>> panic: mxt_enter: locking against myself, 0xffffffff80a2d540
>>>>>> kernel: privileged instruction fault trap, code=0
>>>>>> kernel: double fault trap, code=0
>>>>>>
>>>>>> I was able to reproduce this several times and whenever I wanted it.
>>>>>>
>>>>>> Here is scenario(or setup) then I was able to trigger:
>>>>>> Two networks with OSPF-routing on top of GRE on top of IPSec.
>>>>>> Client on network1 starts scp-download from a firewall(fw1) on network2.
>>>>>> fw1 acts as a VPN/OSPF/GRE end-point for network2.
>>>>>> After some time(1min or so) fw1 goes down with panic above.
>>>>>> Basically I tried to scp down kernel from this machine.
>>>>>>
>>>>>> However, fw1 never goes down then I "scp up" from an Internal network
>>>>>> behind fw1.
>>>>>> I my case this was a patched kernel with Martins' diff from a VM-machine
>>>>>> sitting behind fw1.
