In case someone finds this thread in the future, I would like to add
that I have now received a possible solution to the problem
out-of-band. The solution is to use pair(4):
The following setup works for me, although it is a bit too convoluted:
# cat /etc/hostname.pair0
up
# cat /etc/hostname.pair1
rdomain 1 up patch pair0
# cat /etc/hostname.bridge0
add em2 add vether0 add pair0
!dhclient vether0
!dhclient pair1
In this setup, both vether0 and pair1 gets a separate IP address on
the same network but in different routing domains, but they can still
talk to each other over the patched pair+bridge.
This leads to the following mess, only possible to decipher with a
monospace font:
.----------------.---------. 10.0.0.1 .-,( ),-.
| bridge0 | em2 | dhcp server .-( )-.
| | (no ip) |---> gateway --->( internet )
| '---------' __________ '-( ).-'
.-----------. .-----------.| [_...__...°] '-.( ).-'
| vether0 | | pair0 ||
| rdomain 0 | | rdomain 0 ||
| dhcp | | (no ip) ||
| 10.0.0.2 | '-----------'| .-----------.
'-----------'-------^------' | pair1 |
| | rdomain 1 |
'---patch---| dhcp |
| 10.0.0.3 |
'-----------'
Still not sure if this is a good idea, but it is the solution to my
literal problem so I consider that one solved.
On Sat, Apr 22, 2017 at 3:49 AM, Anders Andersson <[email protected]> wrote:
> === BACKGROUND ===
>
> I'm trying to set up an OpenBSD 6.1 server having two externally visible
> IP numbers through one physical network port, each IP mapping to a
> unique MAC address[1]. I have it mostly working, but my interfaces can't
> talk to each other.
>
> All traffic should use the primary IP, and most services should listen
> on that. The secondary IP should only be used on-demand for one or two
> services.
>
> Thinking that separate routing tables can solve this, I have configured
> my network like this[2][3]:
>
> # cat hostname.em2
> up
>
> # cat hostname.vether0
> lladdr 00:00:00:00:00:02
>
> # cat hostname.vether1
> lladdr 00:00:00:00:00:03 rdomain 1
>
> # cat hostname.bridge0
> add em2 add vether0 add vether1 up
> !dhclient vether0
> !dhclient vether1
>
> # cat sysctl.conf
> net.inet.ip.forwarding=1
>
> Leading to something like this[4]:
> (full post in monospace: http://paste.debian.net/928811 )
>
> .----------------.---------. 10.0.0.1 .-,( ),-.
> | bridge0 | em2 | dhcp server .-( )-.
> | | (no ip) |---> gateway --->( internet )
> | '---------' __________ '-( ).-'
> | | [_...__...°] '-.( ).-'
> .-----------. .-----------.
> | vether0 | | vether1 |
> | rdomain 0 | | rdomain 1 |
> | dhcp | | dhcp |
> | 10.0.0.2 | | 10.0.0.3 |
> '-----------'--'-----------'
>
> Everything else should be the default, this is on a clean 6.1 install.
>
> This configuration works great, vether0 and vether1 both gets an IP
> number from my DHCP server, all traffic goes out on vether0 by default,
> but I can select vether1 manually:
>
> # traceroute -nvq1 10.0.0.1
> traceroute to 10.0.0.1 (10.0.0.1), 64 hops max, 40 byte packets
> 1 10.0.0.1 48 bytes to 10.0.0.2 0.994 ms
>
> # route -T1 exec traceroute -nvq1 10.0.0.1
> traceroute to 10.0.0.1 (10.0.0.1), 64 hops max, 40 byte packets
> 1 10.0.0.1 48 bytes to 10.0.0.3 0.984 ms
>
> I can also reach each IP from outside the box. They are going in on em2,
> through the bridge, and in to vether0 or vether1 respectively.
>
>
>
>
> === PROBLEM ===
>
> Now to my problem: I have no connection between vether0<->vether1.
>
> # traceroute -nvq1 10.0.0.3
> traceroute to 10.0.0.3 (10.0.0.3), 64 hops max, 40 byte...
> 1 *
> 2 *
> ^C
>
> If I listen with tcpdump on the bridge, I see lots of unanswered arp
> who-has:
>
> # tcpdump -nti bridge0
> tcpdump: listening on bridge0, link-type EN10MB
> arp who-has 10.0.0.3 tell 10.0.0.2
> arp who-has 10.0.0.3 tell 10.0.0.2
> ^C
>
> These packets even go out on em2 to my LAN, but no one ever answers. The
> same thing happens in reverse.
>
> I have experimented with these bridge settings:
> 'blocknonip' - adding or removing on members makes no difference
> 'discover' - should be the default, adding makes no difference
> 'learn' - should be the default, adding makes no difference
>
>
>
> === EXPECTATIONS ===
>
> I expected that someone should answer those arp who-is requests, either
> vether1 directly, or the bridge0 who should know which interfaces it
> has. Is there something I must configure to make this work, or is my
> plan flawed from the start?
>
>
>
>
>
> === INFORMATION ===
>
> Various information that could help answer my question (trimmed
> whitespace and boilerplate):
>
> # route -n show -inet
> Destination Gateway Flags Refs Use Mtu Prio Iface
> default 10.0.0.1 UGS 0 0 - 8 vether0
> 224/4 127.0.0.1 URS 0 0 32768 8 lo0
> 10.0.0/24 10.0.0.2 UCn 1 0 - 4 vether0
> 10.0.0.1 link#6 UHLch 1 1 - 3 vether0
> 10.0.0.2 00:00:00:00:00:02 UHLl 0 0 - 1 vether0
> 10.0.0.255 10.0.0.2 UHb 0 0 - 1 vether0
> 127/8 127.0.0.1 UGRS 0 0 32768 8 lo0
> 127.0.0.1 127.0.0.1 UHhl 1 2 32768 1 lo0
>
> # route -T1 -n show -inet
> Destination Gateway Flags Refs Use Mtu Prio Iface
> default 10.0.0.1 UGS 0 32 - 8 vether1
> 10.0.0/24 10.0.0.3 UCn 1 4 - 4 vether1
> 10.0.0.1 00:00:00:00:00:01 UHLch 1 3 - 3 vether1
> 10.0.0.3 00:00:00:00:00:03 UHLl 0 0 - 1 vether1
> 10.0.0.255 10.0.0.3 UHb 0 0 - 1 vether1
>
> # for if in bridge0 em2 vether{0,1}; do ifconfig $if; done
> bridge0: flags=41<UP,RUNNING>
> description: Bridge for external virtual NICs
> index 9 llprio 3
> groups: bridge
> priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp
> designated: id 00:00:00:00:00:00 priority 0
> em2 flags=3<LEARNING,DISCOVER>
> port 3 ifpriority 0 ifcost 0
> vether0 flags=3<LEARNING,DISCOVER>
> port 6 ifpriority 0 ifcost 0
> vether1 flags=3<LEARNING,DISCOVER>
> port 7 ifpriority 0 ifcost 0
> Addresses (max cache: 100, timeout: 240):
> 00:00:00:00:00:01 em2 1 flags=0<>
> em2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
> mtu 1500
> lladdr xx:xx:xx:xx:xx:xx
> description: External interface #1 (integrated NIC)
> index 3 priority 0 llprio 3
> media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
> status: active
> vether0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
> lladdr 00:00:00:00:00:02
> description: Main external IP
> index 6 priority 0 llprio 3
> groups: vether egress
> media: Ethernet autoselect
> status: active
> inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
> vether1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> rdomain 1 mtu 1500
> lladdr 00:00:00:00:00:03
> description: Secondary external IP
> index 7 priority 0 llprio 3
> groups: vether
> media: Ethernet autoselect
> status: active
> inet 10.0.0.3 netmask 0xffffff00 broadcast 10.0.0.255
>
>
>
>
>
> Footnotes:
>
> [1] My ISP implements carrier-grade NAT, with an opt-out. If I supply a
> MAC address, they will whitelist it and give it an external IP
> number over DHCP. I can do this for more than one card but they must
> have a unique MAC, meaning that I can not simply use an IP alias.
> While experimenting with the configuration, I am using my own
> internal DHCP server. The plan is to connect the OpenBSD server
> directly to the ISP.
>
> [2] It should not be necessary to have two vether, em2 should be able to
> serve the purpose of the primary interface. However, I think I'm
> hitting the bug described in the first paragraph here:
> http://undeadly.org/cgi?action=article&sid=20160725144108
>
> [3] The MAC addresses in this report are obfuscated: 10.0.0.xx maps to
> 00:00:00:00:00:xx to aid connecting a card with its IP. The virtual
> interfaces has valid MAC addresses taken from retired NICs.
>
> [4] http://search.cpan.org/dist/App-Asciio/lib/App/Asciio.pm
