On 21:31, Mon 23 Feb 09, Stuart Henderson wrote: > I suspect you might want /32 on the carp interfaces (255.255.255.255 > rather than your 255.255.255.224).
I'll try that in the next week. Thanks for the pointer. > > What are the exact symptoms of not being able to reach .197 when HostB > is in backup state? It may be stating the obvious but check there's no > PF rule that might be blocking it. There's no pf rule blocking it. I know this because if I 'unplug' HostA I can reach HostB without problem. In the info I gave in the mail you can see both hosts decided the default route is over the carp0 interface. Your suggestion to change the subnet to /32 on the carp interface ip addresses might be where the problem is now I reread all the info etc. The exact symptoms are that the host that's in BACKUP mode cannot route any traffic out to the internet. This must be because the default route is going over the carp0 interface instead of the em0 interface. > > You don't mention the OS version (this is one of the reasons dmesg is > helpful to include even when it seems irrelevant), but there have been > various routing-related changes "recently" which may change things. Both firewalls are running OpenBSD 4.4. both firewalls are exactly the same when it comes to hardware and software setup. only the /etc/hostname.* files differ because of the ip addresses and the advskew on the carp interfaces. dmesg at the bottom of this mail... I tried but running a not-released version is not accepted by the company :( > On 2009-02-21, Michiel van Baak <[email protected]> wrote: > > Hi all, > > > > I'm having some trouble with a two-node CARP setup. > > > > Configuration: > > > > HostA > > /etc/hostname.em0 > > inet XXX.XXX.XXX.196 255.255.255.244 XXX.XXX.XXX.223 \ > > media 100baseTX mediaopt full-duplex description External > > > > /etc/hostname.em1 > > inet 192.168.10.2 255.255.255.0 192.168.10.255 \ > > media 100baseTX mediaopt full-duplex description Internal > > > > /etc/hostname.em2 > > inet 10.10.10.1 255.255.255.0 10.10.10.255 \ > > media 100baseTX mediaopt full-duplex description pfsync > > > > /etc/hostname.pfsync0 > > up syncdev em2 > > > > /etc/hostname.carp0 > > inet XXX.XXX.XXX.198 255.255.255.224 XXX.XXX.XXX.223 vhid 1 pass foo > > inet alias XXX.XXX.XXX.199 255.255.255.224 NONE > > inet alias XXX.XXX.XXX.200 255.255.255.224 NONE > > inet alias XXX.XXX.XXX.201 255.255.255.224 NONE > > inet alias XXX.XXX.XXX.202 255.255.255.224 NONE > > inet alias XXX.XXX.XXX.203 255.255.255.224 NONE > > > > /etc/hostname.carp1 > > inet 192.168.10.1 255.255.255.0 192.168.10.255 vhid 2 pass bar > > > > $ cat /etc/sysctl.conf | grep -v '^#' > > > > > > net.inet.ip.forwarding=1 # 1=Permit forwarding (routing) of IPv4 packets > > net.inet.carp.preempt=1 # 1=Enable carp(4) preemption > > > > HostB > > Almost the same, but using XXX.XXX.XXX.197 on em0 and 192.168.10.3 on > > em1 and 10.10.10.2 on em2 and the carp interfaces have advskew 100 > > configured so the box is BACKUP > > > > Now the problem: > > I can reach XXX.XXX.XXX.196 and all configured aliases without trouble. > > I can ssh in, relayd relays are working fine and all. If the box goes > > down or looses connection the second box takes over and everyone is > > happy. > > BUT, I cannot reach XXX.XXX.XXX.197 when HostB is in backup state. > > My suspicion is that this is a routing issue. Looking at the output of > > route -n show: > > > > HostA: > > $ route -n show -inet > > Routing tables > > > > Internet: > > Destination Gateway Flags Refs Use Mtu Prio > > Iface > > default XXX.XXX.XXX.193 UGS 9 53475499 - 48 > > carp0 > > 10.10.10/24 link#3 UC 1 0 - 48 > > em2 > > 10.10.10.2 00:15:17:95:c4:43 UHLc 0 1207 - 48 > > em2 > > XXX.XXX.XXX.192/27 link#6 UC 21 0 - 48 > > carp0 > > XXX.XXX.XXX.193 00:00:5e:00:01:0c UHLc 1 0 - 48 > > carp0 > > XXX.XXX.XXX.194 00:17:cb:ab:81:fe UHLc 0 0 - 48 > > carp0 > > XXX.XXX.XXX.195 00:19:e2:0c:31:fe UHLc 0 0 - 48 > > carp0 > > XXX.XXX.XXX.196 00:15:17:9f:3d:88 UHLc 0 3 - 48 > > lo0 > > XXX.XXX.XXX.196/30 link#1 UC 1 0 - 48 > > em0 > > XXX.XXX.XXX.198 XXX.XXX.XXX.198 UH 0 5 - 48 > > carp0 > > XXX.XXX.XXX.199 XXX.XXX.XXX.199 UH 0 3 - 48 > > carp0 > > XXX.XXX.XXX.200 00:00:5e:00:01:01 UHLc 0 6 - 48 > > lo0 > > XXX.XXX.XXX.201 00:00:5e:00:01:01 UHLc 0 5 - 48 > > lo0 > > XXX.XXX.XXX.202 00:00:5e:00:01:01 UHLc 0 8 - 48 > > lo0 > > > > HostB: > > $ route -n show -inet > > Routing tables > > > > Internet: > > Destination Gateway Flags Refs Use Mtu Prio > > Iface > > default XXX.XXX.XXX.193 UGS 0 190387 - 48 > > carp0 > > 10.10.10/24 link#3 UC 1 0 - 48 > > em2 > > 10.10.10.1 00:15:17:95:c2:b6 UHLc 0 565 - 48 > > em2 > > XXX.XXX.XXX.192/27 link#6 UC 1 0 - 48 > > carp0 > > XXX.XXX.XXX.193 link#6 UHLc 1 0 - 48 > > carp0 > > XXX.XXX.XXX.196/30 link#1 UC 0 0 - 48 > > em0 > > > > > > Any pointers to get this setup correctly so I can reach the addresses on > > the physical interfaces of both boxen, no matter in what CARP state they > > are ? > OpenBSD 4.4 (GENERIC.MP) #1812: Tue Aug 12 17:22:53 MDT 2008 [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 2128158720 (2029MB) avail mem = 2066436096 (1970MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.5 @ 0x7fa32000 (64 entries) bios0: vendor Intel Corporation version "S5000.86B.10.00.0094.101320081858" date 10/13/2008 bios0: Intel S5000PAL acpi0 at bios0: rev 0 acpi0: tables DSDT SLIC FACP APIC SPCR HPET MCFG SSDT SSDT SSDT acpi0: wakeup devices PEX0(S4) PEX1(S4) PEX2(S4) PEX3(S4) PCE4(S4) PCE5(S4) EXPC(S4) PXHA(S4) PXHB(S4) PCE7(S4) PXHA(S4) PXHB(S4) LPC_(S1) PS2M(S1) PS2K(S1) UHC1(S1) UHC2(S1) UHC3(S1) UHC4(S1) EHCI(S1) PCIE(S4) PCIO(S4) PCIP(S4) PCIQ(S4) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5410 @ 2.33GHz, 2327.78 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG cpu0: 6MB 64b/line 16-way L2 cache cpu0: apic clock running at 332MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E5410 @ 2.33GHz, 2327.50 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG cpu1: 6MB 64b/line 16-way L2 cache cpu2 at mainbus0: apid 1 (application processor) cpu2: Intel(R) Xeon(R) CPU E5410 @ 2.33GHz, 2327.50 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG cpu2: 6MB 64b/line 16-way L2 cache cpu3 at mainbus0: apid 3 (application processor) cpu3: Intel(R) Xeon(R) CPU E5410 @ 2.33GHz, 2327.50 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG cpu3: 6MB 64b/line 16-way L2 cache ioapic0 at mainbus0 apid 8 pa 0xfec00000, version 20, 24 pins ioapic1 at mainbus0 apid 9 pa 0xfec80000, version 20, 24 pins acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 12 (P32_) acpiprt2 at acpi0: bus -1 (PEX0) acpiprt3 at acpi0: bus -1 (PEX1) acpiprt4 at acpi0: bus -1 (PEX2) acpiprt5 at acpi0: bus -1 (PEX3) acpiprt6 at acpi0: bus 8 (PCE4) acpiprt7 at acpi0: bus 9 (PCE5) acpiprt8 at acpi0: bus 10 (EXPC) acpiprt9 at acpi0: bus -1 (PXHA) acpiprt10 at acpi0: bus -1 (PXHB) acpiprt11 at acpi0: bus 11 (PCE7) acpiprt12 at acpi0: bus -1 (PXHA) acpiprt13 at acpi0: bus -1 (PXHB) acpiprt14 at acpi0: bus 1 (PCIE) acpiprt15 at acpi0: bus 6 (PCIE) acpiprt16 at acpi0: bus 2 (PCIW) acpiprt17 at acpi0: bus 3 (PCIO) acpiprt18 at acpi0: bus -1 (PCIA) acpiprt19 at acpi0: bus 4 (PCIP) acpiprt20 at acpi0: bus 5 (PCIQ) acpicpu0 at acpi0: C2, C1, PSS acpicpu1 at acpi0: C2, C1, PSS acpicpu2 at acpi0: C2, C1, PSS acpicpu3 at acpi0: C2, C1, PSS acpibtn0 at acpi0: PWRB ipmi at mainbus0 not configured cpu0: unknown i686 model 7, can't get bus clockcpu0: EST: unknown system bus clock pci0 at mainbus0 bus 0: configuration mode 1 pchb0 at pci0 dev 0 function 0 "Intel 5000P Host" rev 0xb1 ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE x8" rev 0xb1 pci1 at ppb0 bus 1 ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01 pci2 at ppb1 bus 2 ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01: apic 8 int 16 (irq 10) pci3 at ppb2 bus 3 ppb3 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01 pci4 at ppb3 bus 4 ppb4 at pci2 dev 2 function 0 "Intel 6321ESB PCIE" rev 0x01 pci5 at ppb4 bus 5 em0 at pci5 dev 0 function 0 "Intel PRO/1000 PT (80003ES2)" rev 0x01: apic 8 int 18 (irq 5), address 00:15:17:9f:3d:88 em1 at pci5 dev 0 function 1 "Intel PRO/1000 PT (80003ES2)" rev 0x01: apic 8 int 19 (irq 11), address 00:15:17:9f:3d:89 ppb5 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01 pci6 at ppb5 bus 6 ppb6 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0xb1 pci7 at ppb6 bus 7 ppb7 at pci0 dev 4 function 0 "Intel 5000 PCIE x8" rev 0xb1: apic 8 int 16 (irq 0) pci8 at ppb7 bus 8 em2 at pci8 dev 0 function 0 "Intel PRO/1000 PT (82572EI)" rev 0x06: apic 8 int 16 (irq 10), address 00:15:17:95:c2:b6 ppb8 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev 0xb1 pci9 at ppb8 bus 9 ppb9 at pci0 dev 6 function 0 "Intel 5000 PCIE x8" rev 0xb1: apic 8 int 16 (irq 0) pci10 at ppb9 bus 10 ppb10 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0xb1 pci11 at ppb10 bus 11 pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0xb1 pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0xb1 pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0xb1 pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0xb1 pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0xb1 pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0xb1 pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0xb1 uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: apic 8 int 23 (irq 11) uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: apic 8 int 22 (irq 5) uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: apic 8 int 23 (irq 11) uhci3 at pci0 dev 29 function 3 "Intel 6321ESB USB" rev 0x09: apic 8 int 22 (irq 5) ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: apic 8 int 23 (irq 11) usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb11 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xd9 pci12 at ppb11 bus 12 vga1 at pci12 dev 12 function 0 "ATI ES1000" rev 0x02 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) drm at vga1 unsupported pcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09 pciide0 at pci0 dev 31 function 1 "Intel 6321ESB IDE" rev 0x09: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility atapiscsi0 at pciide0 channel 0 drive 1 scsibus0 at atapiscsi0: 2 targets, initiator 7 cd0 at scsibus0 targ 0 lun 0: <Optiarc, DVD RW AD-7593A, 1.02> ATAPI 5/cdrom removable cd0(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 2 pciide0: channel 1 disabled (no drives) pciide1 at pci0 dev 31 function 2 "Intel 6321ESB SATA" rev 0x09: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI pciide1: using apic 8 int 20 (irq 10) for native-PCI interrupt wd0 at pciide1 channel 0 drive 0: <WDC WD5002ABYS-01B1B0> wd0: 16-sector PIO, LBA48, 476940MB, 976773168 sectors wd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5 ichiic0 at pci0 dev 31 function 3 "Intel 6321ESB SMBus" rev 0x09: apic 8 int 20 (irq 10) iic0 at ichiic0 usb1 at uhci0: USB revision 1.0 uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb2 at uhci1: USB revision 1.0 uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb3 at uhci2: USB revision 1.0 uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb4 at uhci3: USB revision 1.0 uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 midi0 at pcppi0: <PC speaker> spkr0 at pcppi0 mtrr: Pentium Pro MTRR support softraid0 at root root on wd0a swap on wd0b dump on wd0b -- Michiel van Baak [email protected] http://michiel.vanbaak.eu GnuPG key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x71C946BD "Why is it drug addicts and computer aficionados are both called users?"
