Hi,
as we have to connect several customers and branches with overlapping
routing/encryption domains, I started creating a box using the rdomain
feature of OpenBSD.
Routing between rdomains using pf is working flawlessly: NAT is applied
in both directions and it works even with fully overlapping routing
domains.
Routing between rdomains and a VPN destination (isakmpd running in
rdomain 0) is working as long as no NAT is tried to be applied.
Network configuration of the OBSD box:
# ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33152
priority: 0
groups: lo
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
inet 127.0.0.1 netmask 0xff000000
em0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
lladdr 00:0c:29:7c:14:1a
priority: 0
groups: egress
media: Ethernet autoselect (1000baseT full-duplex,master)
status: active
inet 88.77.88.61 netmask 0xfffffff0 broadcast 88.77.88.63
inet6 fe80::20c:29ff:fe7c:141a%em0 prefixlen 64 scopeid 0x1
em1:
flags=28b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOIN
ET6> rdomain 1 mtu 1500
lladdr 00:0c:29:7c:14:24
priority: 0
media: Ethernet autoselect (1000baseT full-duplex,master)
status: active
inet 10.0.3.2 netmask 0xffffff00 broadcast 10.0.3.255
em2:
flags=28b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOIN
ET6> rdomain 2 mtu 1500
lladdr 00:0c:29:7c:14:2e
priority: 0
media: Ethernet autoselect (1000baseT full-duplex,master)
status: active
inet 10.0.3.2 netmask 0xffffff00 broadcast 10.0.3.255
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:0c:29:7c:14:38
priority: 0
media: Ethernet autoselect (1000baseT full-duplex,master)
status: active
inet 192.168.230.2 netmask 0xfffffff0 broadcast 192.168.230.15
inet6 fe80::20c:29ff:fe7c:1438%em3 prefixlen 64 scopeid 0x4
enc0: flags=41<UP,RUNNING>
priority: 0
groups: enc
status: active
lo1: flags=28049<UP,LOOPBACK,RUNNING,MULTICAST,NOINET6> rdomain 1 mtu
33152
priority: 0
groups: lo
inet 127.0.0.1 netmask 0xff000000
lo2: flags=28049<UP,LOOPBACK,RUNNING,MULTICAST,NOINET6> rdomain 2 mtu
33152
priority: 0
groups: lo
inet 127.0.0.1 netmask 0xff000000
pfsync0: flags=41<UP,RUNNING> mtu 1500
priority: 0
pfsync: syncdev: em3 maxupd: 128 defer: off
groups: carp pfsync
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33152
priority: 0
groups: pflog
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:0a
priority: 0
carp: MASTER carpdev em0 vhid 10 advbase 1 advskew 0
groups: carp
status: master
inet6 fe80::200:5eff:fe00:10a%carp0 prefixlen 64 scopeid 0xc
inet 88.77.88.60 netmask 0xffffff00 broadcast 88.77.88.255
carp1: flags=28843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,NOINET6>
rdomain 1 mtu 1500
lladdr 00:00:5e:00:01:14
priority: 0
carp: MASTER carpdev em1 vhid 20 advbase 1 advskew 0
groups: carp
status: master
inet 10.0.3.1 netmask 0xffffff00 broadcast 10.0.3.255
carp2: flags=28843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,NOINET6>
rdomain 2 mtu 1500
lladdr 00:00:5e:00:01:1e
priority: 0
carp: MASTER carpdev em2 vhid 30 advbase 1 advskew 0
groups: carp
status: master
inet 10.0.3.1 netmask 0xffffff00 broadcast 10.0.3.255
em0 is connected to the Internet (default routing domain 0). em2 is one
of the internal networks in rdomain 2. This network is used as the
encryption domain for the vpn.
The routing tables look like this:
# route -T 0 -n show -inet
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu Prio
Iface
default 88.77.88.49 UGS 0 70 - 8
em0
10.0.3/24 127.0.0.1 UGS 38 76 33152 8
lo0
10.0.4/24 127.0.0.1 UGS 0 0 33152 8
lo0
127/8 127.0.0.1 UGRS 0 0 33152 8
lo0
127.0.0.1 127.0.0.1 UH 3 0 33152 4
lo0
192.168.230.0/28 link#4 UC 0 0 - 4
em3
88.77.88/24 link#12 UC 0 0 - 4
carp0
88.77.88.48/28 link#1 UC 2 0 - 4
em0
88.77.88.49 link#1 UHLc 1 0 - 4
em0
88.77.88.52 00:0c:29:9c:d3:02 UHLc 0 448 - 4
em0
88.77.88.60 88.77.88.60 UH 0 0 - 4
carp0
224/4 127.0.0.1 URS 0 0 33152 8
lo0
#
# route -T 2 -n show -inet
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu Prio
Iface
10.0.3/24 link#3 UC 1 0 - 4
em2
10.0.3.10 00:0c:29:89:77:e2 UHLc 2 461 - 4
em2
10.0.4/24 10.0.3.10 UGS 0 2 - 8
em2
127.0.0.1 127.0.0.1 UH 2 0 33152 4
lo2
192.168.33/24 127.0.0.1 UGS 0 1 33152 8
lo2
88.77.88.48/28 127.0.0.1 UGS 0 0 33152 8
lo2
#
There is a minimalistic set of pf rules configured:
# pfctl -s rules
block drop in log all
pass in quick on em2 proto tcp from any to any port = ssh flags S/SA
pass on lo0 all no state
pass on lo1 all no state
pass on lo2 all no state
pass on em0 proto carp all
pass on em1 proto carp all
pass on em2 proto carp all
pass on em3 proto pfsync all
match in log on enc0 inet from 192.168.33.0/24 to 10.0.15.0/24 rdr-to
10.0.3.0/24 bitmask
match out log on enc0 inet from 10.0.3.0/24 to 192.168.33.0/24 nat-to
10.0.15.0/24 bitmask static-port
pass out on em0 inet proto udp from 88.77.88.60 to 88.77.88.52 port =
isakmp
pass out on em0 inet proto udp from 88.77.88.60 to 88.77.88.52 port =
ipsec-nat-t
pass out on em0 inet proto esp from 88.77.88.60 to 88.77.88.52
pass out on em0 all flags S/SA
pass out on enc0 inet proto ipencap from 88.77.88.60 to 88.77.88.52
pass out on em1 all flags S/SA
pass out on em2 all flags S/SA
pass out on em3 all flags S/SA
pass in on em0 inet proto udp from 88.77.88.52 to 88.77.88.60 port =
isakmp
pass in on em0 inet proto udp from 88.77.88.52 to 88.77.88.60 port =
ipsec-nat-t
pass in on em0 inet proto esp from 88.77.88.52 to 88.77.88.60
pass in on enc0 inet proto ipencap from 88.77.88.52 to 88.77.88.60
pass out log on em2 inet from 192.168.33.0/24 to 10.0.3.0/24 flags S/SA
pass in log on em2 inet from 10.0.3.0/24 to 192.168.33.0/24 flags S/SA
rtable 0
pass in log on enc0 inet from 192.168.33.0/24 to 10.0.3.0/24 flags S/SA
rtable 2
pass out log on enc0 inet from 10.0.3.0/24 to 192.168.33.0/24 flags S/SA
#
The vpn configuration is simple and straight forward:
# cat /etc/ipsec.conf
ike esp from 10.0.15.0/24 (10.0.3.0/24) to 192.168.33.0/24 \
local 88.77.88.60 peer 88.77.88.52 \
main auth hmac-sha1 enc aes-256 group modp1024 \
quick auth hmac-sha1 enc aes-256 group none \
psk abctest
When I start the VPN everything looks ok:
# isakmpd -K
# ipsecctl -f /etc/ipsec.conf
# ipsecctl -vs all
FLOWS:
flow esp in from 192.168.33.0/24 to 10.0.3.0/24 peer 88.77.88.52 srcid
88.77.88.60/32 dstid 88.77.88.52/32 type use
flow esp out from 10.0.3.0/24 to 192.168.33.0/24 peer 88.77.88.52 srcid
88.77.88.60/32 dstid 88.77.88.52/32 type require
SAD:
esp tunnel from 88.77.88.60 to 88.77.88.52 spi 0x02234fca auth hmac-sha1
enc aes-256
sa: spi 0x02234fca auth hmac-sha1 enc aes
state mature replay 16 flags 4
lifetime_cur: alloc 0 bytes 1848 add 1335969276 first 1335969277
lifetime_hard: alloc 0 bytes 0 add 1200 first 0
lifetime_soft: alloc 0 bytes 0 add 1080 first 0
address_src: 88.77.88.60
address_dst: 88.77.88.52
identity_src: type prefix id 0: 88.77.88.60/32
identity_dst: type prefix id 0: 88.77.88.52/32
src_mask: 255.255.255.0
dst_mask: 255.255.255.0
protocol: proto 0 flags 0
flow_type: type use direction out
src_flow: 10.0.3.0
dst_flow: 192.168.33.0
lifetime_lastuse: alloc 0 bytes 0 add 0 first 1335969313
esp tunnel from 88.77.88.52 to 88.77.88.60 spi 0x274763cc auth hmac-sha1
enc aes-256
sa: spi 0x274763cc auth hmac-sha1 enc aes
state mature replay 16 flags 4
lifetime_cur: alloc 0 bytes 256 add 1335969276 first 1335969399
lifetime_hard: alloc 0 bytes 0 add 1200 first 0
lifetime_soft: alloc 0 bytes 0 add 1080 first 0
address_src: 88.77.88.52
address_dst: 88.77.88.60
identity_src: type prefix id 0: 88.77.88.52/32
identity_dst: type prefix id 0: 88.77.88.60/32
src_mask: 255.255.255.0
dst_mask: 255.255.255.0
protocol: proto 0 flags 0
flow_type: type use direction in
src_flow: 192.168.33.0
dst_flow: 10.0.3.0
lifetime_lastuse: alloc 0 bytes 0 add 0 first 1335969415
#
The VPN is successfully established and the correct (NATed) subnet is
uses for Phase-2 SA negotiation.
>From the logfile of the remote gateway (Check Point Firewall-1 Gaia
R75.40):
Number: 435
Date: 2May2012
Time: 14:23:40
Interface: daemon
Origin: gaia1
Type: Log
Action: Key Install
Source: obsd (88.77.88.60)
Destination: gaia1 (88.77.88.52)
Community: cp_obsd
Information: IKE: Quick Mode completion IKE IDs:
subnet: 192.168.33.0 (mask= 255.255.255.0) and subnet: 10.0.15.0 (mask=
255.255.255.0)
Source Key ID: 0x274763cc
Destination Key ID: 0x02234fca
Encryption Scheme: IKE
Encryption Methods: ESP: AES-256 + SHA1
IKE Initiator Cookie: 4df1247f162befd5
IKE Responder Cookie: 3f45060c7024c41e
IKE Phase2 Message ID: 19562ec3
VPN Peer Gateway: obsd (88.77.88.60)
Subproduct: VPN
VPN Feature: IKE
Although everything looks fine a ping the remote site from a workstation
on the local site (10.0.3/24) fails.
In a trace on pflog0 I can see, that the packets match the correct rule
but are not translated.
# tcpdump -nei pflog0
tcpdump: WARNING: snaplen raised from 116 to 160
tcpdump: listening on pflog0, link-type PFLOG
16:38:18.382729 rule 14/(match) match out on enc0: 10.0.3.10 >
192.168.33.51: icmp: echo request (DF)
16:38:19.391099 rule 14/(match) match out on enc0: 10.0.3.10 >
192.168.33.51: icmp: echo request (DF)
16:38:20.399146 rule 14/(match) match out on enc0: 10.0.3.10 >
192.168.33.51: icmp: echo request (DF)
16:38:21.407203 rule 14/(match) match out on enc0: 10.0.3.10 >
192.168.33.51: icmp: echo request (DF)
16:38:22.415234 rule 14/(match) match out on enc0: 10.0.3.10 >
192.168.33.51: icmp: echo request (DF)
16:38:23.423168 rule 14/(match) match out on enc0: 10.0.3.10 >
192.168.33.51: icmp: echo request (DF)
16:38:24.431330 rule 14/(match) match out on enc0: 10.0.3.10 >
192.168.33.51: icmp: echo request (DF)
^C
7 packets received by filter
0 packets dropped by kernel
#
# pfctl -sr -R 14
match out log on enc0 inet from 10.0.3.0/24 to 192.168.33.0/24 nat-to
10.0.15.0/24 bitmask static-port
#
# pfctl -s state | grep 192.168.33
all icmp 192.168.33.51:8 ((2) 192.168.33.51:8) <- 10.0.3.10:1981 ((2)
10.0.3.10:1981) 0:0
#
Thus the packet is dropped by the remote gateway:
Number: 441
Date: 2May2012
Time: 14:33:12
Interface: eth0
Origin: gaia1
Type: Log
Action: Drop
Source: 10.0.3.10
Destination: 192.168.33.51
Protocol: icmp
Information: ICMP: Echo Request ICMP Type: 8 ICMP
Code: 0 encryption failure: According to the policy the packet should
not have been decrypted
Encryption Scheme: IKE
Encryption Methods: ESP: AES-256 + SHA1
VPN Peer Gateway: obsd (88.77.88.60)
Subproduct: VPN
VPN Feature: VPN
Product: Security Gateway/Management
Log ID: 404821
Product Family: Network
Any hints and help is appreciated.
Cheers,
Bernd
