-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
OpenBSD 4.0 i386 on dual Nexcom 1563 firewall boxes using carp and pfsync.
In my setup, there are two carp interfaces bound to the "external"
physical interface fxp0, each in turn bound to a different internal
machine using nat and rdr. This worked fine for about six months.
Since upgrading from 3.9 to 4.0, the carp0 and carp1 interfaces have
been flapping between MASTER and BACKUP state. This is true even if I
down all the carp and pfsync interfaces on the backup firewall. (Or vice
versa; powering down either of the firewalls doesn't make the problem go
away.)
The physical links to both machines are stable. I don't see any evidence
of links going up or down, or anything like CRC or other errors.
Thanks in advance for any clues on debugging and fixing this. Below is
some relevant tcpdump and config info.
dn
pflog0 output on fw1:
# tcpdump -n -e -ttt -i pflog0 proto carp or proto pfsync
(no output)
pfsync0 output on fw1:
# tcpdump -nevv -i pfsync0
tcpdump: WARNING: pfsync0: no IPv4 address assigned
tcpdump: listening on pfsync0, link-type PFSYNC
12:39:02.188189 PFSYNCv3 count 6: UPD ST:
12:39:02.634228 PFSYNCv3 count 6: UPD ST:
12:39:03.614361 PFSYNCv3 count 6: UPD ST:
12:39:04.624496 PFSYNCv3 count 6: UPD ST:
12:39:05.634636 PFSYNCv3 count 6: UPD ST:
12:39:06.228438 PFSYNCv3 count 6: UPD ST:
12:39:06.674776 PFSYNCv3 count 6: UPD ST:
12:39:07.674914 PFSYNCv3 count 6: UPD ST:
12:39:08.685120 PFSYNCv3 count 5: UPD ST:
12:39:09.245144 PFSYNCv3 count 6: UPD ST:
12:39:09.734795 PFSYNCv3 count 5: UPD ST:
12:39:09.735217 PFSYNCv3 count 2: INS ST:
12:39:10.636297 PFSYNCv3 count 6: UPD ST:
12:39:10.795337 PFSYNCv3 count 6: UPD ST:
12:39:11.775472 PFSYNCv3 count 6: UPD ST:
12:39:12.775605 PFSYNCv3 count 6: UPD ST:
12:39:13.736501 PFSYNCv3 count 5: UPD ST:
12:39:13.785762 PFSYNCv3 count 2: DEL ST:
12:39:14.315835 PFSYNCv3 count 6: UPD ST:
12:39:14.875889 PFSYNCv3 count 6: UPD ST:
12:39:15.736744 PFSYNCv3 count 3: UPD ST:
12:39:15.826041 PFSYNCv3 count 2: DEL ST:
12:39:16.356138 PFSYNCv3 count 6: UPD ST:
12:39:16.670451 PFSYNCv3 count 2: UPD ST:
12:39:16.716224 PFSYNCv3 count 2: INS ST:
12:39:16.736912 PFSYNCv3 count 1: UPD ST:
12:39:16.846173 PFSYNCv3 count 1: DEL ST:
12:39:16.952763 PFSYNCv3 count 5: UPD ST:
12:39:16.952800 PFSYNCv3 count 1: INS ST:
12:39:17.359779 PFSYNCv3 count 1: UPD ST:
12:39:17.359840 PFSYNCv3 count 1: INS ST:
12:39:17.359945 PFSYNCv3 count 1: UPD ST:
12:39:17.359965 PFSYNCv3 count 1: UPD REQ:
id: 45b1b92b00004cd9 creatorid: e778ffb2
12:39:17.360061 PFSYNCv3 count 1: UPD REQ:
id: 45b1b92b00004cda creatorid: e778ffb2
12:39:17.360096 PFSYNCv3 count 1: UPD ST:
12:39:17.360221 PFSYNCv3 count 1: UPD ST:
12:39:17.936304 PFSYNCv3 count 6: UPD ST:
12:39:18.906434 PFSYNCv3 count 6: UPD ST:
12:39:19.637473 PFSYNCv3 count 6: UPD ST:
12:39:19.976578 PFSYNCv3 count 6: UPD ST:
12:39:20.057401 PFSYNCv3 count 2: INS ST:
12:39:20.086116 PFSYNCv3 count 2: UPD ST:
12:39:20.087355 PFSYNCv3 count 2: INS ST:
12:39:20.926744 PFSYNCv3 count 6: UPD ST:
12:39:21.637721 PFSYNCv3 count 6: UPD ST:
12:39:22.016860 PFSYNCv3 count 6: UPD ST:
12:39:22.986991 PFSYNCv3 count 6: UPD ST:
12:39:23.347764 PFSYNCv3 count 2: UPD ST:
12:39:23.347802 PFSYNCv3 count 1: INS ST:
12:39:23.737821 PFSYNCv3 count 3: UPD ST:
12:39:23.737936 PFSYNCv3 count 6: DEL ST:
12:39:23.738174 PFSYNCv3 count 1: UPD REQ:
id: 45b1b92b00004cdc creatorid: e778ffb2
12:39:23.738230 PFSYNCv3 count 1: UPD ST:
12:39:24.437214 PFSYNCv3 count 6: UPD ST:
12:39:24.737952 PFSYNCv3 count 5: UPD ST:
12:39:25.007288 PFSYNCv3 count 4: DEL ST:
12:39:25.232689 PFSYNCv3 count 5: UPD ST:
12:39:25.232725 PFSYNCv3 count 1: INS ST:
12:39:25.638268 PFSYNCv3 count 6: UPD ST:
12:39:25.638733 PFSYNCv3 count 1: UPD ST:
12:39:25.638763 PFSYNCv3 count 1: UPD REQ:
id: 45b1b92b00004cde creatorid: e778ffb2
12:39:25.638831 PFSYNCv3 count 1: UPD ST:
12:39:26.097416 PFSYNCv3 count 6: UPD ST:
12:39:27.117552 PFSYNCv3 count 6: UPD ST:
^C
fw1 settings:
hostname.fxp0
inet 207.181.8.188 255.255.255.192 NONE media autoselect
hostname.carp0
inet 207.181.8.190 255.255.255.192 207.181.8.191 vhid 1 carpdev fxp0
advbase 1 advskew 1 pass password
hostname.carp1
inet 207.181.8.130 255.255.255.192 207.181.8.191 vhid 2 carpdev fxp0
advbase 1 advskew 1 pass password
$ sysctl net.inet.carp
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=0
net.inet.carp.arpbalance=0
$ sysctl net.inet,ip.forwarding
net.inet.ip.forwarding=1
from pf.conf:
ExtIf="fxp0"
CarpIf0 = "carp0"
CarpIf1 = "carp1"
pfsyncIf = "fxp1"
# ICMP types
icmpTypes = "{ echoreq }"
# Default
block log all
# carp and pfsync
pass quick on { $pfsyncIf } proto pfsync
pass on { $ExtIf $IntIf } proto carp keep state
pass in on $ExtIf inet proto icmp from any to { $ExtIf, $CarpIf0, $CarpIf1 }
pass inet proto icmp all icmp-type $icmpTypes keep state
fw2 settings:
hostname.fxp0
inet 207.181.8.189 255.255.255.192 NONE media autoselect
hostname.carp0
inet 207.181.8.190 255.255.255.192 207.181.8.191 vhid 1 carpdev fxp0
advbase 1 advskew 128 pass password
hostname.carp1
inet 207.181.8.130 255.255.255.192 207.181.8.191 vhid 2 carpdev fxp0
advbase 1 advskew 128 pass password
$ sysctl net.ip.carp
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=0
net.inet.carp.arpbalance=0
$ sysctl net.inet.ip.forwarding
net.inet.ip.forwarding=1
from pf.conf:
ExtIf="fxp0"
CarpIf0 = "carp0"
CarpIf1 = "carp1"
pfsyncIf = "fxp1"
# ICMP types
icmpTypes = "{ echoreq }"
# Default
block log all
# carp and pfsync
pass quick on { $pfsyncIf } proto pfsync
pass on { $ExtIf $IntIf } proto carp keep state
pass in on $ExtIf inet proto icmp from any to { $ExtIf, $CarpIf0, $CarpIf1 }
pass inet proto icmp all icmp-type $icmpTypes keep state
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFFsqxIyPxGVjntI4IRAm3kAJ9Kcfwz7a6/LaqVpfBO0hjgingcRACg2/Gs
zSEDieaeeosrQCuPAoSxrcA=
=h5Dy
-----END PGP SIGNATURE-----
