I have a pair of freshly installed OpenBSD 4.7/amd64-RELEASE systems.
They're running redundant failover pairs with fw1 being the master.
It seems I've got a problem getting pfsync to properly pass a full
bulk update over, so longer term sessions time out when the MASTER
takes over for the BACKUP system. In both failover cases (MASTER fails
or BACKUP fails) the pf state rules seem to miss transferring over
the full ruleset to the recently brought up system. Usually, if both
have been up for 30 minutes and older rules have timed out, there's no
issue. It's just during the recovery of the "failed" firewall where
problems seem to occur.
Both systems are identical in hardware, installed OS, and (mostly)
configuration (hostname.if differences do exist).
Any tips on where to start looking would be appreciated.
em0: external interface - fw0: 192.168.10.217; fw1: 192.168.10.218
em1: internal interface - fw0: 10.254.0.2; fw1: 10.254.0.3
em2: pfsync interface - fw0: 172.253.0.1; fw1: 172.253.02
carp0: external CARP - 192.168.10.216
carp1: internal CARP - 10.254.0.1
pfsync0: syncdev em2 maxupd: 128 defer: off (cat hostname.pfsync0: up
syncdev em2)
fw0 hostname.carp0: inet 192.168.10.216 255.255.255.0 192.168.10.255
vhid 1 carpdev em0 pass pass0 advbase 1 advskew 0
fw1 hostname.carp0: inet 192.168.10.216 255.255.255.0 192.168.10.255
vhid 1 carpdev em0 pass pass0 advbase 1 advskew 100
fw0 hostname.carp1: inet 10.254.0.1 255.255.255.0 10.254.0.255 vhid 2
pass pass1 carpdev em1 advbase 1 advskew 0
fw1 hostname.carp1: inet 10.254.0.1 255.255.255.0 10.254.0.255 vhid 2
pass pass1 carpdev em1 advbase 1 advskew 100
Some pfsync messages:
fw0:
messages.0.gz:May 21 19:56:43 fw0 /bsd: pfsync: received bulk update request
messages.0.gz:May 21 19:59:13 fw0 /bsd: pfsync: received bulk update request
messages.1.gz:May 21 17:35:10 fw0 /bsd: pfsync: failed to receive bulk update
messages.1.gz:May 21 18:25:59 fw0 /bsd: pfsync: failed to receive bulk update
fw1:
messages.0.gz:May 21 18:41:38 fw1 /bsd: pfsync: failed to receive bulk update
messages.0.gz:May 21 19:17:12 fw1 /bsd: pfsync: failed to receive bulk update
messages.0.gz:May 21 19:56:43 fw1 /bsd: pfsync: requesting bulk update
messages.0.gz:May 21 19:56:43 fw1 /bsd: pfsync: received bulk update start
messages.0.gz:May 21 19:56:43 fw1 /bsd: pfsync: received valid bulk update
end
pf.conf (identical on both systems):
#;777777$Id$#
# See pf.conf(5) for syntax and examples.
# Remember to set net.inet.ip.forwarding=1 and/or net.inet6.ip6.forwarding=1
# in /etc/sysctl.conf if packets are to be forwarded between interfaces.
# firewall settings
set block-policy return
set skip on lo
set skip on carp
set skip on em2 # internal sync interface between the firewalls
altq on em0 priq bandwidth 1.9Mb queue {std, ssh_im, dns, tcp_ack}
queue std priq(default)
queue ssh_im priority 4 priq(red)
queue dns priority 5
queue tcp_ack priority 6
## tables and macros
# firewall self identification
table <firewall> const {self}
# trusted IPstable <sf-office> const { 192.168.10.216, 192.168.10.217,
192.168.10.218,\
192.168.10.219, 192.168.10.220, 192.168.10.221, 192.168.10.222,
192.168.10.223
, \
10.122.19.56/29 }
table <mv-office> const { 192.168.24.109, 192.168.36.168/29 }
table <sj-colo> const { 10.124.148.0/24, 172.123.152.0/21 }
table <fwsync> const { 172.253.0.0/29 }
ext_if="em0"
ext_vip="192.168.10.216"
int_if="em1"
int_net="10.254.0.0/24"
int_vip="10.254.0.1"
im_port = "{ 706 1863 5190 5222 6667 6668 }"
## base filter rules
# block everything by default
block in log all
block in log quick from urpf-failed
pass out quick from <firewall> to any
pass quick on em2 proto pfsync keep state (no-sync)
pass quick on egress proto carp keep state (no-sync)
pass quick on ingress proto carp keep state (no-sync)
## NAT
match out on egress from (self) to any tag NAT nat-to (egress)
match out on egress from $int_net to !$int_net received-on ingress \
tagged OUT \
tag NAT \
nat-to $ext_vip static-port
## external interface
pass in on egress inet proto icmp from any to (egress)
pass in on egress inet proto tcp from { <mv-office>, <sfo-office> } to
(egress) port 22
pass out on egress inet proto tcp from any to any \
tagged NAT \
queue (std, tcp_ack)
pass out quick on egress inet proto { udp, tcp } from any to any \
port 53 \
tagged NAT \
queue dns7
pass out quick on egress inet proto tcp from any to any \
port 22 \
tagged NAT \
queue (std, ssh_im)
pass out quick on egress inet proto {udp, tcp} from any to any \
port $im_port \
tagged NAT \
queue (ssh_im, tcp_ack)
pass out quick on egress inet proto icmp from any to any \
tagged NAT \
queue std
## internal interface
pass in on ingress inet proto tcp from $int_if:network to self port 22
pass in on ingress inet from $int_if:network to any tag OUT
dmesg:
OpenBSD 4.7 (GENERIC.MP) #130: Wed Mar 17 20:48:50 MDT 2010
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 1013747712 (966MB)
avail mem = 975212544 (930MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe8620 (104 entries)
bios0: vendor Intel Corp. version "LDB4310H.86A.0035.2009.1125.1944" date
11/25/
2009
bios0: Intel Corporation DB43LD
acpi0 at bios0: rev 2
acpi0: tables DSDT FACP APIC MCFG ASF! SPCR
acpi0: wakeup devices P0P1(S3) PEGP(S4) PS2K(S3) PS2M(S3) UAR1(S4) P0P2(S3)
PCI1
(S4) PCI2(S4) USB0(S3) USB1(S3) USB2(S3) EUSB(S3) USB3(S3) USB4(S3) USBE(S3)
PEX
0(S4) PCE1(S4) PEX1(S4) PEX2(S4) PEX3(S4) PEX4(S4) USB5(S3) GBE_(S4) SLPB(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Pentium(R) Dual-Core CPU E5300 @ 2.60GHz, 2600.41 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CF
LUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX
16,
xTPR,NXE,LONG
cpu0: 2MB 64b/line 8-way L2 cache
cpu0: apic clock running at 200MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Pentium(R) Dual-Core CPU E5300 @ 2.60GHz, 2600.09 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CF
LUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX
16,
xTPR,NXE,LONG
cpu1: 2MB 64b/line 8-way L2 cache
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 4 (P0P2)
acpiprt2 at acpi0: bus 2 (PEX0)
acpiprt3 at acpi0: bus 3 (PEX2)
acpicpu0 at acpi0:, C3, C2, C1, PSS
acpicpu1 at acpi0:, C3, C2, C1, PSS
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
cpu0: Enhanced SpeedStep 2600 MHz: speeds: 2600, 1200 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 vendor "Intel", unknown product 0x2e40 rev
0x03
ppb0 at pci0 dev 1 function 0 vendor "Intel", unknown product 0x2e41 rev 0x03:
a
pic 0 int 16 (irq 11)
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic 0
int
16 (irq 11), address 00:15:17:d7:7e:76
em1 at pci1 dev 0 function 1 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic 0
int
17 (irq 11), address 00:15:17:d7:7e:77
vga1 at pci0 dev 2 function 0 vendor "Intel", unknown product 0x2e42 rev 0x03
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
intagp at vga1 not configured
vendor "Intel", unknown product 0x2e44 (class communications subclass
miscellane
ous, rev 0x03) at pci0 dev 3 function 0 not configured
em2 at pci0 dev 25 function 0 "Intel ICH10 D BM LM" rev 0x02: apic 0 int 20
(irq
10), address 00:1c:c0:fc:76:5e
uhci0 at pci0 dev 26 function 0 "Intel 82801JD USB" rev 0x02: apic 0 int 16
(irq
11)
uhci1 at pci0 dev 26 function 1 "Intel 82801JD USB" rev 0x02: apic 0 int 21
(irq
3)
uhci2 at pci0 dev 26 function 2 "Intel 82801JD USB" rev 0x02: apic 0 int 18
(irq
11)
ehci0 at pci0 dev 26 function 7 "Intel 82801JD USB" rev 0x02: apic 0 int 18
(irq
11)
ehci0: timed out waiting for BIOS
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
azalia0 at pci0 dev 27 function 0 "Intel 82801JD HD Audio" rev 0x02: apic 0
int
22 (irq 10)
azalia0: codecs: Realtek ALC662
audio0 at azalia0
ppb1 at pci0 dev 28 function 0 "Intel 82801JD PCIE" rev 0x02: apic 0 int 17
(irq
3)
pci2 at ppb1 bus 2
ppb2 at pci0 dev 28 function 2 "Intel 82801JD PCIE" rev 0x02
pci3 at ppb2 bus 3
uhci3 at pci0 dev 29 function 0 "Intel 82801JD USB" rev 0x02: apic 0 int 23
(irq
7)
uhci4 at pci0 dev 29 function 1 "Intel 82801JD USB" rev 0x02: apic 0 int 19
(irq
11)
uhci5 at pci0 dev 29 function 2 "Intel 82801JD USB" rev 0x02: apic 0 int 18
(irq
11)
ehci1 at pci0 dev 29 function 7 "Intel 82801JD USB" rev 0x02: apic 0 int 23
(irq
7)
ehci1: timed out waiting for BIOS
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb3 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa2
pci4 at ppb3 bus 4
pcib0 at pci0 dev 31 function 0 "Intel 82801JD LPC" rev 0x02
pciide0 at pci0 dev 31 function 2 "Intel 82801JD SATA" rev 0x02: DMA, channel
0
configured to native-PCI, channel 1 configured to native-PCI
pciide0: using apic 0 int 19 (irq 11) for native-PCI interrupt
wd0 at pciide0 channel 0 drive 0: <WDC WD3200AAKS-00L9A0>
wd0: 16-sector PIO, LBA48, 305245MB, 625142448 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 6
atapiscsi0 at pciide0 channel 1 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <Optiarc, DVD RW AD-7260S, 1.00> ATAPI 5/cdrom
rem
ovable
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 5
ichiic0 at pci0 dev 31 function 3 "Intel 82801JD SMBus" rev 0x02: apic 0 int
18
(irq 11)
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 1GB DDR2 SDRAM non-parity PC2-5300CL5
pciide1 at pci0 dev 31 function 5 "Intel 82801JD SATA" rev 0x02: DMA, channel
0
wired to native-PCI, channel 1 wired to native-PCI
pciide1: using apic 0 int 19 (irq 11) for native-PCI interrupt
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
pciide1 at pci0 dev 31 function 5 "Intel 82801JD SATA" rev 0x02: DMA, channel
0
wired to native-PCI, channel 1 wired to native-PCI
pciide1: using apic 0 int 19 (irq 11) for native-PCI interrupt
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb6 at uhci4: USB revision 1.0
uhub6 at usb6 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb7 at uhci5: USB revision 1.0
uhub7 at usb7 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
mtrr: Pentium Pro MTRR support
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
root on wd0a swap on wd0b dump on wd0b
