Thanks Steffen, but I don't think it is the case.
I shut down VTI interface toward another VPS and GRE on top of it,
enabled the plain GRE for transport SA. It works on one end, but not for
the other end which has to leave VTI with `remote any` up. How can the
transport SA match this `remote any` VTI?
Attached relevant `ip tun`, `ip link`, `ip addr` and `ip xfrm pol` for
your investigation.
On the working end:
root@ayaka:~# ip tun
test: gre/ip remote 139.162.122.174 local 149.248.36.252 dev ens3 ttl
inherit nopmtudisc
goips-i0n2: gre/ip remote 172.16.44.7 local 172.16.44.6 dev ipsec0 ttl
inherit
ipsec0: ip/ip remote 139.162.122.174 local 149.248.36.252 dev ens3 ttl
inherit nopmtudisc key 410191106
root@ayaka:~# ip link
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
mode DEFAULT group default qlen 1000
link/ether 56:00:01:c6:b6:20 brd ff:ff:ff:ff:ff:ff
12: ipsec0@ens3: <POINTOPOINT,NOARP> mtu 1446 qdisc noqueue state DOWN
mode DEFAULT group default qlen 1000
link/ipip 149.248.36.252 peer 139.162.122.174
13: goips-i0n2@ipsec0: <POINTOPOINT,NOARP,M-DOWN> mtu 1422 qdisc noqueue
state DOWN mode DEFAULT group default qlen 1000
link/gre 172.16.44.6 peer 172.16.44.7
14: test@ens3: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1400 qdisc noqueue
state UNKNOWN mode DEFAULT group default qlen 1000
link/gre 149.248.36.252 peer 139.162.122.174
root@ayaka:~# ip addr
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
group default qlen 1000
link/ether 56:00:01:c6:b6:20 brd ff:ff:ff:ff:ff:ff
inet 149.248.36.252/23 brd 149.248.37.255 scope global ens3
valid_lft forever preferred_lft forever
12: ipsec0@ens3: <POINTOPOINT,NOARP> mtu 1446 qdisc noqueue state DOWN
group default qlen 1000
link/ipip 149.248.36.252 peer 139.162.122.174
inet 172.16.44.6/31 brd 255.255.255.255 scope global ipsec0
valid_lft forever preferred_lft forever
13: goips-i0n2@ipsec0: <POINTOPOINT,NOARP,M-DOWN> mtu 1422 qdisc noqueue
state DOWN group default qlen 1000
link/gre 172.16.44.6 peer 172.16.44.7
inet 172.22.181.16/31 brd 255.255.255.255 scope global goips-i0n2
valid_lft forever preferred_lft forever
14: test@ens3: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1400 qdisc noqueue
state UNKNOWN group default qlen 1000
link/gre 149.248.36.252 peer 139.162.122.174
inet 172.22.181.16/31 scope global test
valid_lft forever preferred_lft forever
root@ayaka:~# ip xfrm pol
src 149.248.36.252/32 dst 139.162.122.174/32 proto gre
dir out priority 366975 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp spi 0xc9d4a0c3 reqid 6 mode transport
src 139.162.122.174/32 dst 149.248.36.252/32 proto gre
dir in priority 366975 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 6 mode transport
On the not working end:
root@kotone:~# ip tun
ipsec0: ip/ip remote any local 139.162.122.174 dev eth0 ttl inherit
nopmtudisc key 410190082
goips-i0n1: gre/ip remote 172.16.44.1 local 172.16.44.0 dev ipsec0 ttl
inherit
test: gre/ip remote 149.248.36.252 local 139.162.122.174 dev eth0 ttl
inherit nopmtudisc
goips-i2n5: gre/ip remote 172.16.44.6 local 172.16.44.7 dev ipsec2 ttl
inherit
ipsec2: ip/ip remote 149.248.36.252 local 139.162.122.174 dev eth0 ttl
inherit nopmtudisc key 410191106
root@kotone:~# ip link
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP mode DEFAULT group default qlen 1000
link/ether f2:3c:91:70:9c:ad brd ff:ff:ff:ff:ff:ff
12: ipsec0@eth0: <NOARP,UP,LOWER_UP> mtu 1446 qdisc noqueue state
UNKNOWN mode DEFAULT group default qlen 1000
link/ipip 139.162.122.174 brd 0.0.0.0
14: ipsec2@eth0: <POINTOPOINT,NOARP> mtu 1446 qdisc noqueue state DOWN
mode DEFAULT group default qlen 1000
link/ipip 139.162.122.174 peer 149.248.36.252
15: goips-i0n1@ipsec0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1422 qdisc
noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/gre 172.16.44.0 peer 172.16.44.1
17: goips-i2n5@ipsec2: <POINTOPOINT,NOARP,M-DOWN> mtu 1422 qdisc noqueue
state DOWN mode DEFAULT group default qlen 1000
link/gre 172.16.44.7 peer 172.16.44.6
18: test@eth0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1400 qdisc noqueue
state UNKNOWN mode DEFAULT group default qlen 1000
link/gre 139.162.122.174 peer 149.248.36.252
root@kotone:~# ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP group default qlen 1000
link/ether f2:3c:91:70:9c:ad brd ff:ff:ff:ff:ff:ff
inet 139.162.122.174/24 brd 139.162.122.255 scope global eth0
valid_lft forever preferred_lft forever
12: ipsec0@eth0: <NOARP,UP,LOWER_UP> mtu 1446 qdisc noqueue state
UNKNOWN group default qlen 1000
link/ipip 139.162.122.174 brd 0.0.0.0
inet 172.16.44.0/31 brd 255.255.255.255 scope global ipsec0
valid_lft forever preferred_lft forever
14: ipsec2@eth0: <POINTOPOINT,NOARP> mtu 1446 qdisc noqueue state DOWN
group default qlen 1000
link/ipip 139.162.122.174 peer 149.248.36.252
inet 172.16.44.7/31 brd 255.255.255.255 scope global ipsec2
valid_lft forever preferred_lft forever
15: goips-i0n1@ipsec0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1422 qdisc
noqueue state UNKNOWN group default qlen 1000
link/gre 172.16.44.0 peer 172.16.44.1
inet 172.22.181.10/31 brd 255.255.255.255 scope global goips-i0n1
valid_lft forever preferred_lft forever
17: goips-i2n5@ipsec2: <POINTOPOINT,NOARP,M-DOWN> mtu 1422 qdisc noqueue
state DOWN group default qlen 1000
link/gre 172.16.44.7 peer 172.16.44.6
inet 172.22.181.17/31 brd 255.255.255.255 scope global goips-i2n5
valid_lft forever preferred_lft forever
18: test@eth0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1400 qdisc noqueue
state UNKNOWN group default qlen 1000
link/gre 139.162.122.174 peer 149.248.36.252
inet 172.22.181.17/31 scope global test
valid_lft forever preferred_lft forever
root@kotone:~# ip xfrm pol
src 172.16.44.0/31 dst 172.16.44.0/31
dir out priority 368255 ptype main
mark 0x18730102/0xffffffff
tmpl src 139.162.122.174 dst 122.100.194.181
proto esp spi 0xd5d1c8e9 reqid 24 mode tunnel
src 172.16.44.0/31 dst 172.16.44.0/31
dir fwd priority 368255 ptype main
mark 0x18730102/0xffffffff
tmpl src 122.100.194.181 dst 139.162.122.174
proto esp reqid 24 mode tunnel
src 172.16.44.0/31 dst 172.16.44.0/31
dir in priority 368255 ptype main
mark 0x18730102/0xffffffff
tmpl src 122.100.194.181 dst 139.162.122.174
proto esp reqid 24 mode tunnel
src 139.162.122.174/32 dst 149.248.36.252/32 proto gre
dir out priority 366975 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp spi 0xc815bb54 reqid 25 mode transport
src 149.248.36.252/32 dst 139.162.122.174/32 proto gre
dir in priority 366975 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 25 mode transport
On 2018/12/21 04:38 PM, Steffen Klassert wrote:
On Sat, Dec 15, 2018 at 07:02:41PM +0800, Lemon Lam wrote:
Hello,
I recently joined DN42 with my virtual private servers, and decided to
use
GRE
and IPsec to form interconnects between servers.
I can use GRE over IPsec VTI tunnel fine, but when I simplified some
tunnels
down to GRE over IPsec transport, no incoming traffic is possible.
Please look into full description below.
[1.] One line summary of the problem:
XFRMINSTATEMODEERROR for transport mode IPsec SA when IP VTI is
active
[2.] Full description of the problem/report:
I built tunnels according to StrongSwan's guide on VTI, i.e.
using
`ip tun add ipsecvti mode vti key <hex key>`, then I add GRE on
top of
it for MPLS. Everything works great at this stage.
I want to strip it down to GRE over IPsec transport between my
VPS but
have to leave one as-is since there's endpoint with dynamic IP,
need
this
as workaround. After necessary configurations, I pinged between
transport
mode tunnel, received no response. `swanctl -l` showed increased
outgoing
traffic counter, but incoming counter stayed at zero. `tcpdump`
showed
incoming ESP packets on physical interfaces but no corresponding
packets
on GRE tunnel.
Hinted to look at `/proc/net/xfrm_stat` by developers from
StrongSwan,
found out that XFRMINSTATEMODEERROR increases by any traffic on
transport tunnel. Later experiments discovered that merely
`ip link set ipsecvti down` will let incoming traffic went
through.
This looks like you have a transport mode SA that matches
the src and dst endpoint of the vti tunnel. If this is
the case, it is a conceptional problem. VTI behaves like
an IP tunnel, it can not handle transport mode packets.
