Server 1:
* ixl(4) connected to the public Internet using 1500 MTU.
* Different ixl(4) used as the "trunk" interface connected to my switch
and the parent interface of all of my vlan(4)s. Also uses 1500 MTU.
* wg(4) in rdomain 1 used as the default route for rtable 1. It
connects to Server 2's wg(4) IPv6 endpoint using rtable 0. Uses 1420
MTU.
* vlan(4) in rdomain 1 that runs an e-mail server. Uses 1420 MTU.
Server 2:
* vio(4) connected to the public Internet using 1500 MTU.
* wg(4) that connects to Server 1's wg(4) IPv6 endpoint. Uses 1420 MTU.
Used as the gateway for all traffic destined to Server 1's vlan(4).
All of the IPs for the vlan(4)s in rdomain 1 on Server 1 are globally
routable. pf(4) isn't used for anything more complicated than pass/block.
For more than 5 years, this has served me well until rather abruptly I
started experiencing issues with external MTA servers both sending and
receiving e-mail. The only popular providers that didn't seem to have
issues were Gmail and Outlook. After several days I was able to identify
the issue: MTU on Server 1's wg(4) interface.
Using ping6(8), I was able to figure out that the maximum MTU I could
set for Server 1's wg(4) interface was 1408. Keeping its vlan(4) and
Server 2's wg(4) at 1420 was fine. I ended up reducing both to 1408
though to match.
ping6 -D -s 1452 works when using IPs in rdomain 0 including the IP
assigned to the ixl(4) connected to the public Internet on Server 1.
Similarly, it works when using the IP assigned to vio(4) on Server 2.
This suggests the 1500 MTU of those interfaces is fine which should
be the case since Server 1's ISP is a cable provider that doesn't use
PPPoE or similar. Server 2's provider is a VPS provider that doesn't
use PPPoE either.
Also to be clear, the maximum number of data bytes I can send using
ping6(8) is 1360 when using any of the IPs assigned to the vlan(4)s
in rdomain 1 on Server 1 _as well_ as the IP assigned to wg(4) itself (
i.e., traffic doesn't have to be sent from/to the vlan(4)s).
Can someone explain why I'd need to reduce the MTU by 12 from 1420 to
1408 and only on Server 1's wg(4)?
Does anyone have any theories as to why this would suddenly become a
problem? I've been running OpenBSD 7.9 for a few months and haven't
touched my pf(4) rules. Perhaps Path MTU Discovery (PMTUD) was saving my
ass, but now it's being blocked somewhere and doesn't work[^1]?
I've run traceroute6(8) using the vlan(4)'s IP connecting to a variety
of MTAs, and it looks damn near identical as the traceroute6(8) from
Server 2 when using vio(4)'s IP.
[^1]: I always allow ICMPv6 and ICMP, so any blocking that may be
occurring is not on my side.
