Todd Alan Smith wrote:
This only happens with SSH connections? Are the rulesets identical
between the two machines? Also, why are you still running 4.2? As I'm
sure you know, there have been many improvements to pf since that
release.
No, I also see it happening with every TCP-based protocol and port I've
tried (telnet, ftp, and iscsi)
BTW, a more appropriate subject line would have been "why is pf blocking
a connection after having already accepting it"
Yes, I know I should upgrade, especially since I bought the CDs, but I
haven't had the time yet - though this issue may force me to upgrade...
P.S. Maybe send your dmesg(s) and ruleset(s) with your next reply.
OK, see below, for the following:
- uname on firewall
- dmesg on firewall
- ifconfig -a on firewall
- ruleset on firewall
Also, so this makes more sense, here is a small network diagram
vlan4 trunk,tagged-vlans
10.0.4.6 -------- managed ------------------ carped -- internet
10.0.4.5 -------- switch ----------------- firewalls -- feed
||
||vlan1
|+-------- 10.0.1.24
+--------- 10.0.1.22
P.P.S. Part of my brain keeps thinking, "Flaky NIC?"
I was thinking the same thing - so far I:
- moved the 10.0.1.24 ethernet cable to another port in my switch
- moved the 10.0.1.24 ethernet cable to another port on the host machine
- failed the firewall over to it's CARP peer (also running 4.2)
- tried a different client computer (10.0.4.5) instead of (10.0.4.6)
-----UNAME-----
# uname -a
OpenBSD fw2.watsen.net 4.2 GENERIC.RAID#0 sparc64
-----DMESG-----
# dmesg
console is /p...@1f,0/p...@1,1/i...@7/ser...@0,3f8
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2007 OpenBSD. All rights reserved.
http://www.OpenBSD.org
OpenBSD 4.2 (GENERIC.RAID) #0: Fri Dec 28 22:26:28 EST 2007
[email protected]:/usr/src/sys/arch/sparc64/compile/GENERIC.RAID
real mem = 536870912 (512MB)
avail mem = 507109376 (483MB)
mainbus0 at root: Netra T1 200 (UltraSPARC-IIe 500MHz)
cpu0 at mainbus0: SUNW,UltraSPARC-IIe (rev 1.4) @ 500 MHz, version 0 FPU
cpu0: physical 16K instruction (32 b/l), 16K data (32 b/l), 256K
external (64 b/l)
psycho0 at mainbus0: SUNW,sabre, impl 0, version 0, ign 7c0
psycho0: bus range 0-2, PCI bus 0
psycho0: dvma map c0000000-dfffffff, iotdb 962000-9e2000
pci0 at psycho0
ppb0 at pci0 dev 1 function 1 "Sun Simba PCI-PCI" rev 0x13
pci1 at ppb0 bus 1
ebus0 at pci1 dev 12 function 0 "Sun RIO EBus" rev 0x01
"flashprom" at ebus0 addr 0-fffff not configured
clock1 at ebus0 addr 0-1fff: mk48t59
"SUNW,lomh" at ebus0 addr 200000-200003 ipl 42 not configured
"Acer Labs M7101 Power" rev 0x00 at pci1 dev 3 function 0 not configured
ebus1 at pci1 dev 7 function 0 "Acer Labs M1533 ISA" rev 0x00
power0 at ebus1 addr 2000-2007 ipl 37
com0 at ebus1 addr 3f8-3ff ipl 43: ns16550a, 16 byte fifo
com0: console
com1 at ebus1 addr 2e8-2ef ipl 43: ns16550a, 16 byte fifo
gem0 at pci1 dev 12 function 1 "Sun ERI Ether" rev 0x01: ivec 0x7c6,
address 00:03:ba:0f:2c:d3
ukphy0 at gem0 phy 1: Generic IEEE 802.3u media interface, rev. 1: OUI
0x0010dd, model 0x0002
ohci0 at pci1 dev 12 function 3 "Sun USB" rev 0x01: ivec 0x7e4, version
1.0, legacy support
pciide0 at pci1 dev 13 function 0 "Acer Labs M5229 UDMA IDE" rev 0xc3:
DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide0: using ivec 0x7cc for native-PCI interrupt
atapiscsi0 at pciide0 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <TEAC, CD-224E, 1.7A> SCSI0 5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
gem1 at pci1 dev 5 function 1 "Sun ERI Ether" rev 0x01: ivec 0x7dc,
address 00:03:ba:0f:2c:d4
ukphy1 at gem1 phy 1: Generic IEEE 802.3u media interface, rev. 1: OUI
0x0010dd, model 0x0002
ohci1 at pci1 dev 5 function 3 "Sun USB" rev 0x01: ivec 0x7e6, version
1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0: Sun OHCI root hub, rev 1.00/1.00, addr 1
usb1 at ohci1: USB revision 1.0
uhub1 at usb1: Sun OHCI root hub, rev 1.00/1.00, addr 1
ppb1 at pci0 dev 1 function 0 "Sun Simba PCI-PCI" rev 0x13
pci2 at ppb1 bus 2
siop0 at pci2 dev 8 function 0 "Symbios Logic 53c896" rev 0x07: ivec
0x7e0, using 8K of on-board RAM
scsibus1 at siop0: 16 targets
sd0 at scsibus1 targ 0 lun 0: <IBM, DNES-309170Y, SA60> SCSI3 0/direct fixed
sd0: 8683MB, 11474 cyl, 5 head, 309 sec, 512 bytes/sec, 17783301 sec total
sd1 at scsibus1 targ 1 lun 0: <IBM, DNES-309170Y, SA60> SCSI3 0/direct fixed
sd1: 8683MB, 11474 cyl, 5 head, 309 sec, 512 bytes/sec, 17783301 sec total
siop1 at pci2 dev 8 function 1 "Symbios Logic 53c896" rev 0x07: ivec
0x7e0, using 8K of on-board RAM
scsibus2 at siop1: 16 targets
em0 at pci2 dev 5 function 0 "Intel PRO/1000MT (82545EM)" rev 0x01: ivec
0x7d5, address 00:07:e9:1a:19:62
"pcons" at mainbus0 not configured
Kernelized RAIDframe activated
cd0(atapiscsi0:0:0): Check Condition (error 0x70) on opcode 0x0
SENSE KEY: Not Ready
ASC/ASCQ: Medium Not Present
siop0: target 0 now using tagged 16 bit 40.0 MHz 31 REQ/ACK offset xfers
siop0: target 1 now using tagged 16 bit 40.0 MHz 31 REQ/ACK offset xfers
raid0 at root: (RAID Level 1) total number of sectors is 16732160 (8170
MB) as root
bootpath: /p...@1f,0/p...@1,0/s...@8,0/d...@0,0
WARNING: clock gained 5 days -- CHECK AND RESET THE DATE!
swapmount: no device
raid0: Device already configured!
-----IFCONFIG-----
# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33168
groups: lo
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
gem0:
flags=8b63<UP,BROADCAST,NOTRAILERS,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
lladdr 00:03:ba:0f:2c:d3
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet6 fe80::203:baff:fe0f:2cd3%gem0 prefixlen 64 scopeid 0x1
gem1: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:03:ba:0f:2c:d4
media: Ethernet autoselect (none)
status: no carrier
inet 10.0.0.252 netmask 0xffffff00 broadcast 10.0.0.255
inet6 fe80::203:baff:fe0f:2cd4%gem1 prefixlen 64 scopeid 0x2
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:07:e9:1a:19:62
media: Ethernet autoselect (1000baseT full-duplex)
status: active
inet6 fe80::207:e9ff:fe1a:1962%em0 prefixlen 64 scopeid 0x3
enc0: flags=0<> mtu 1536
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:07:e9:1a:19:62
vlan: 1 priority: 0 parent interface: em0
groups: vlan
inet6 fe80::207:e9ff:fe1a:1962%vlan1 prefixlen 64 scopeid 0x6
inet 10.0.1.252 netmask 0xffffff00 broadcast 10.0.1.255
vlan2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:07:e9:1a:19:62
vlan: 2 priority: 0 parent interface: em0
groups: vlan
inet6 fe80::207:e9ff:fe1a:1962%vlan2 prefixlen 64 scopeid 0x7
inet 10.0.2.252 netmask 0xffffff00 broadcast 10.0.2.255
vlan3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:07:e9:1a:19:62
vlan: 3 priority: 0 parent interface: em0
groups: vlan
inet6 fe80::207:e9ff:fe1a:1962%vlan3 prefixlen 64 scopeid 0x8
inet 10.0.3.252 netmask 0xffffff00 broadcast 10.0.3.255
vlan4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:07:e9:1a:19:62
vlan: 4 priority: 0 parent interface: em0
groups: vlan
inet6 fe80::207:e9ff:fe1a:1962%vlan4 prefixlen 64 scopeid 0x9
inet 10.0.4.252 netmask 0xffffff00 broadcast 10.0.4.255
pfsync0: flags=41<UP,RUNNING> mtu 1460
pfsync: syncdev: gem1 maxupd: 128
groups: carp pfsync
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33168
groups: pflog
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:09
carp: MASTER carpdev gem0 vhid 9 advbase 1 advskew 128
groups: carp egress
inet6 fe80::200:5eff:fe00:109%carp0 prefixlen 64 scopeid 0xb
inet 96.231.191.4 netmask 0xffffff00 broadcast 96.231.191.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:01
carp: MASTER carpdev vlan1 vhid 1 advbase 1 advskew 128
groups: carp
inet6 fe80::200:5eff:fe00:101%carp1 prefixlen 64 scopeid 0xc
inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255
carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:02
carp: MASTER carpdev vlan2 vhid 2 advbase 1 advskew 128
groups: carp
inet6 fe80::200:5eff:fe00:102%carp2 prefixlen 64 scopeid 0xd
inet 10.0.2.1 netmask 0xffffff00 broadcast 10.0.2.255
carp3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:03
carp: MASTER carpdev vlan3 vhid 3 advbase 1 advskew 128
groups: carp
inet6 fe80::200:5eff:fe00:103%carp3 prefixlen 64 scopeid 0xe
inet 10.0.3.1 netmask 0xffffff00 broadcast 10.0.3.255
carp4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:04
carp: MASTER carpdev vlan4 vhid 4 advbase 1 advskew 128
groups: carp
inet6 fe80::200:5eff:fe00:104%carp4 prefixlen 64 scopeid 0xf
inet 10.0.4.1 netmask 0xffffff00 broadcast 10.0.4.255
-----RULESET-----
# cat
/etc/pf.conf
################################################################################
# macro
definitions #
################################################################################
# gateway interfaces
ext_if = "gem0"
mgt_if = "vlan1"
dmz_if = "vlan2"
com_if = "vlan3"
fam_if = "vlan4"
# gateway ips
ext_ip = "carp0"
mgt_ip = "carp1"
dmz_ip = "carp2"
com_ip = "carp3"
fam_ip = "carp4"
# networks (to the extent their needs by rules)
mgt_net = $mgt_ip:network
dmz_net = $dmz_ip:network
com_net = $com_ip:network
fam_net = $fam_ip:network
inside_net = "10.0.0.0/16"
#inside_net = "{" $mgt_net $dmz_net $fam_net $com_net "}"
# mgt machines (to the extent their needs by rules)
xyplex = "10.0.1.20"
# dmz machines (to the extent their needs by rules)
ssh_watsen_net = "10.0.2.2"
dns_watsen_net = "10.0.2.2"
www_watsen_net = "10.0.2.2"
mail_watsen_net = "10.0.2.2"
# com machines (to the extent their needs by rules)
printer = "10.0.3.3"
# fam machines (to the extent their needs by rules)
kents_pc = "10.0.4.6"
# martians (this list validated many times)
martians = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, \
10.0.0.0/8, 169.254.0.0/16, 192.0.2.0/24, \
0.0.0.0/8, 240.0.0.0/4 }"
################################################################################
# options:
"set" #
################################################################################
set skip on lo
set skip on gem1
################################################################################
# scrub rules:
"scrub" #
################################################################################
# scrub in
scrub in on $ext_if all
################################################################################
# NAT rules: "rdr", "nat",
"binat" #
################################################################################
rdr on $ext_if proto udp from any to carp0 port domain -> $dns_watsen_net
rdr on $ext_if proto tcp from any to carp0 port domain -> $dns_watsen_net
rdr on $ext_if proto tcp from any to carp0 port ssh -> $ssh_watsen_net
rdr on $ext_if proto tcp from any to carp0 port http -> $www_watsen_net
rdr on $ext_if proto tcp from any to carp0 port https -> $www_watsen_net
rdr on $ext_if proto tcp from any to carp0 port smtp -> $mail_watsen_net
rdr on $ext_if proto tcp from any to carp0 port pop3s -> $mail_watsen_net
rdr on $ext_if proto tcp from any to carp0 port imaps -> $mail_watsen_net
rdr on $ext_if proto tcp from any to carp0 port auth -> $mail_watsen_net
# the following redirects would be unneccessary if using own DNS
rdr on $dmz_if proto udp from any to carp0 port domain -> $dns_watsen_net
rdr on $dmz_if proto tcp from any to carp0 port domain -> $dns_watsen_net
rdr on $dmz_if proto tcp from any to carp0 port ssh -> $ssh_watsen_net
rdr on $dmz_if proto tcp from any to carp0 port http -> $www_watsen_net
rdr on $dmz_if proto tcp from any to carp0 port https -> $www_watsen_net
rdr on $dmz_if proto tcp from any to carp0 port smtp -> $mail_watsen_net
rdr on $dmz_if proto tcp from any to carp0 port pop3s -> $mail_watsen_net
rdr on $dmz_if proto tcp from any to carp0 port imaps -> $mail_watsen_net
rdr on $dmz_if proto tcp from any to carp0 port auth -> $mail_watsen_net
# the following redirects would be unneccessary if using own DNS
rdr on $com_if proto udp from any to carp0 port domain -> $dns_watsen_net
rdr on $com_if proto tcp from any to carp0 port domain -> $dns_watsen_net
rdr on $com_if proto tcp from any to carp0 port ssh -> $ssh_watsen_net
rdr on $com_if proto tcp from any to carp0 port http -> $www_watsen_net
rdr on $com_if proto tcp from any to carp0 port https -> $www_watsen_net
rdr on $com_if proto tcp from any to carp0 port smtp -> $mail_watsen_net
rdr on $com_if proto tcp from any to carp0 port pop3s -> $mail_watsen_net
rdr on $com_if proto tcp from any to carp0 port imaps -> $mail_watsen_net
rdr on $com_if proto tcp from any to carp0 port auth -> $mail_watsen_net
# the following redirects would be unneccessary if using own DNS
rdr on $fam_if proto udp from any to carp0 port domain -> $dns_watsen_net
rdr on $fam_if proto tcp from any to carp0 port domain -> $dns_watsen_net
rdr on $fam_if proto tcp from any to carp0 port ssh -> $ssh_watsen_net
rdr on $fam_if proto tcp from any to carp0 port http -> $www_watsen_net
rdr on $fam_if proto tcp from any to carp0 port https -> $www_watsen_net
rdr on $fam_if proto tcp from any to carp0 port smtp -> $mail_watsen_net
rdr on $fam_if proto tcp from any to carp0 port pop3s -> $mail_watsen_net
rdr on $fam_if proto tcp from any to carp0 port imaps -> $mail_watsen_net
rdr on $fam_if proto tcp from any to carp0 port auth -> $mail_watsen_net
#rdr on $fam_if from any to 10.0.1.2 -> 10.0.2.2
nat on $ext_if from $inside_net to ! $inside_net -> carp0
################################################################################
# filtering rules: "antispoof", "block",
"pass" #
################################################################################
block return log all
### rules for ext_if
###########################################################
#
# Essentially:
# - only allow select traffic in from Internet
# - all traffic can get out
# special rule for carp
pass quick on $ext_if proto carp keep state
block return in log quick on $ext_if from $martians to any
block return out log quick on $ext_if from any to $martians
# only let select traffic in from the Internet
pass in quick on $ext_if proto udp from any to $dns_watsen_net port domain
pass in quick on $ext_if proto tcp from any to $dns_watsen_net port domain
pass in quick on $ext_if proto tcp from any to $ssh_watsen_net port ssh
pass in quick on $ext_if proto tcp from any to $www_watsen_net port http
pass in quick on $ext_if proto tcp from any to $www_watsen_net port https
pass in quick on $ext_if proto tcp from any to $mail_watsen_net port smtp
pass in quick on $ext_if proto tcp from any to $mail_watsen_net port pop3s
pass in quick on $ext_if proto tcp from any to $mail_watsen_net port imaps
# let traffic with src-addr carp0 out to the Internet
pass out quick on $ext_if from carp0 to ! $inside_net
### rules for mgt_if
###########################################################
#
# Essentially:
# - no traffic originating in mgt network gets out
# - only kents_pc can get in
# special rule for carp
pass quick on $mgt_if proto carp keep state
# special rule for fw-initiated traffic
pass out log quick on $mgt_if from $mgt_if to $mgt_net
# special rule to let other firewall ssh in
pass in log quick on $mgt_if proto tcp from $mgt_net to $mgt_if port ssh
# special rule to allow Dom0's access to pkg.opensolaris.org (image-update)
pass in log quick on $mgt_if proto tcp to "72.5.123.21" port http
# I think this one was for when I was trying IPS and d/l-ed mercurial
#pass in log quick on $mgt_if proto tcp to "64.79.150.44" port https
# special rule to allow Dom0's to access DNS
pass in log quick on $mgt_if proto udp from $mgt_net to any port domain
pass in log quick on $mgt_if proto tcp from $mgt_net to any port domain
# super special rule to allow download of slim.py file
pass in log quick on $mgt_if proto tcp from $mgt_net to $www_watsen_net
port http
# normal rules
xyplex_ports = "{ 2000, 2100, 2200, 2300, 2400, 2500, 2600, 2700, 2800,
2900, \
3000, 2100, 3200, 3300, 3400, 3500, 3600, 3700, 3800,
3900, \
4000 }"
# allow kents_pc to initiate some traffic to mgt network
pass out log quick on $mgt_if proto tcp from $kents_pc to $mgt_net port ssh
pass out log quick on $mgt_if proto tcp from $kents_pc to $mgt_net port http
pass out log quick on $mgt_if proto tcp from $kents_pc to $mgt_net port
https
pass out log quick on $mgt_if proto tcp from $kents_pc to $xyplex port
$xyplex_ports
# allow kents_pc to ssh into BACKUP firewall
pass in log quick on $mgt_if proto tcp from $kents_pc to $mgt_net port ssh
### rules for dmz_if
###########################################################
#
# Essentially:
# - all traffic originating from other networks gets in (out thru
interface)
# - only specific dmz-services can initiate traffic out (in thru
interface)
# special rule for carp
pass quick on $dmz_if proto carp keep state
# special rule for fw-initiated traffic
pass out log quick on $dmz_if from $dmz_if to $dmz_net
# special rule to allow DomU's access to openbsd.org (to get install list)
pass in log quick on $dmz_if proto tcp to "129.128.5.191" port http
# special rule to allow DomU's access to mirror.rit.org (openbsd install
site)
pass in log quick on $dmz_if proto tcp to "129.21.171.98" port http
# special rule to not let any host access the firewall's vlan ip
block return in log quick on $dmz_if from $dmz_net to $dmz_if
# normal rules
# allow all networks to access DMZ
pass out log quick on $dmz_if from any to $dmz_net
# allow outbound dns client requests - 53 (udp & tcp)
pass in quick on $dmz_if proto udp from $dmz_net to any port domain
pass in quick on $dmz_if proto tcp from $dmz_net to any port domain
# allow outbound dns server replies (udp & tcp)
pass in quick on $dmz_if proto udp from $dns_watsen_net to any port domain
pass in quick on $dmz_if proto tcp from $dns_watsen_net to any port domain
# allow outbound ntp - 123 (udp & tcp)
pass in quick on $dmz_if proto udp from $dmz_net to any port ntp
pass in quick on $dmz_if proto tcp from $dmz_net to any port ntp
# allow outbound smtp - 25 (DON'T FORGET TO ENABLE INBOUND AUTH!)
pass in quick on $dmz_if proto tcp from $mail_watsen_net to any port smtp
# allow fleshclam/clamav to update (via: dig database.clamav.net on 4/18/07)
clamav_net = "{ 155.98.64.86, 205.139.192.13, 206.154.202.213,
208.67.80.27, 209.8.40.140, 64.186.240.114, 65.110.48.11, 72.21.63.182 }"
pass in quick on $dmz_if proto tcp from $mail_watsen_net to $clamav_net
port http
### rules for com_if
###########################################################
#
# Essentially:
# - allow all hosts to initiate traffic to other networks (in thru
interface)
# - only allow access to printer from fam-network (out thru interface)
# special rule for carp
pass quick on $com_if proto carp keep state
# special rule for fw-initiated traffic
pass out log quick on $com_if from $com_if to $com_net
# special rule to not let any host access the firewall's vlan ip
block return in log quick on $com_if from $com_net to $com_if
# normal rules
pass in log quick on $com_if from $com_net to any
pass out log quick on $com_if from $fam_net to $printer
### rules for fam_if
###########################################################
#
# Essentially:
# - allow all hosts to initiate traffic to other networks (in thru
interface)
# - no network can initiate traffic to fam network (out thru interface)
# special rule for carp
pass quick on $fam_if proto carp keep state
# special rule for fw-initiated traffic
pass out log quick on $fam_if from $fam_if to $fam_net
# special rule to not let any host access the firewall's vlan ip
block return in log quick on $fam_if from $fam_net to $fam_if
# normal rules
pass in log quick on $fam_if from $fam_net to any
