On 9/30/20 7:23 PM, Stephen Suryaputra wrote: > On Thu, Sep 24, 2020 at 08:41:54AM -0600, David Ahern wrote: >>> We have multiple options on the table right now. One that can be done >>> without writing any code is to use an nft prerouting rule to mark >>> the packet with iif equals the tunnel and use ip rule fwmark to lookup >>> the right table. >>> >>> ip netns exec r0 nft add table ip c2t >>> ip netns exec r0 nft add chain ip c2t prerouting '{ type filter hook >>> prerouting priority 0; policy accept; }' >>> ip netns exec r0 nft rule ip c2t prerouting iif gre01 mark set 101 counter >>> ip netns exec r0 ip rule add fwmark 101 table 10 pref 999 >>> >>> ip netns exec r1 nft add table ip c2t >>> ip netns exec r1 nft add chain ip c2t prerouting '{ type filter hook >>> prerouting priority 0; policy accept; }' >>> ip netns exec r1 nft rule ip c2t prerouting iif gre10 mark set 101 counter >>> ip netns exec r1 ip rule add fwmark 101 table 10 pref 999 >>> >>> But this doesn't seem to work on my Ubuntu VM with the namespaces >>> script, i.e. pinging from h0 to h1. The packet doesn't egress r1_v11. It >>> does work on our target, based on 4.14 kernel. >> >> add debugs to net/core/fib_rules.c, fib_rule_match() to see if >> flowi_mark is getting set properly. There could easily be places that >> are missed. Or if it works on one setup, but not another compare sysctl >> settings for net.core and net.ipv4 > > Sorry, I got side-tracked. Coming back to this: I made a mistake in the > ip rule fwmark pref in the script. I have fixed it and the script is > attached (gre_setup_nft.sh). It has the nft and ip rule commands above. > The ping between h0 and h1 works. > >>> We also notice though in on the target platform that the ip rule fwmark >>> doesn't seem to change the skb->dev to the vrf of the lookup table. >> >> not following that statement. fwmark should be marking the skb, not >> changing the skb->dev. > > Yes and it causes the ping between h0 and r1 r1_v11 to not work, e.g.: > > ip netns exec h0 ping -c 1 11.0.0.1 > > Similarly, ping between r0_v00 and r1_v11 also does not work: > > ip netns exec r0 ip vrf exec vrf_r0t ping -c 1 -I 10.0.0.1 11.0.0.1 > >>> E.g., ping from 10.0.0.1 to 11.0.0.1. With net.ipv4.fwmark_reflect set, >>> the reply is generated but the originating ping application doesn't get >>> the packet. I suspect it is caused by the socket is bound to the tenant >>> vrf. I haven't been able to repro this because of the problem with the >>> nft approach above. > > To illustrate my statements above, this is what I did: > ip netns exec r1 sysctl -w net.ipv4.fwmark_reflect=1 > ip netns exec h0 ping -c 1 11.0.0.1 > PING 11.0.0.1 (11.0.0.1) 56(84) bytes of data. > 64 bytes from 11.0.0.1: icmp_seq=1 ttl=63 time=0.079 ms > > The ping between h0 and r1 r1_v11 works, but it still doesn't work for this: > ip netns exec r0 ip vrf exec vrf_r0t ping -c 1 -I 10.0.0.1 11.0.0.1 > > eventhough the reply is received by gre01: > ip netns exec r0 tcpdump -nexi gre01 > 22:10:57.173680 Out ethertype IPv4 (0x0800), length 100: 10.0.0.1 > 11.0.0.1: > ICMP echo request, id 3803, seq 1, length 64 > 0x0000: 4500 0054 1d2a 4000 4001 087e 0a00 0001 > 0x0010: 0b00 0001 0800 a410 0edb 0001 b13a 755f > 0x0020: 0000 0000 5da6 0200 0000 0000 1011 1213 > 0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 > 0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 > 0x0050: 3435 3637 > 22:10:57.173724 In ethertype IPv4 (0x0800), length 100: 11.0.0.1 > 10.0.0.1: > ICMP echo reply, id 3803, seq 1, length 64 > 0x0000: 4500 0054 c709 0000 4001 9e9e 0b00 0001 > 0x0010: 0a00 0001 0000 ac10 0edb 0001 b13a 755f > 0x0020: 0000 0000 5da6 0200 0000 0000 1011 1213 > 0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 > 0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 > 0x0050: 3435 3637 > > In summary the question is: should ip rule with action FR_ACT_TO_TBL > also change the skb->dev to the right l3mdev?
'ip rule' does not change the skb; only the vrf driver does that (and some IPv4/IPv6 code reverts it). The rule code only changes the flowi_oif setting.