On Wed, Jul 15, 2026 at 07:43:17AM +0000, Jack Ma wrote: > On Tue, Jul 14, 2026 at 08:42:01PM +0300, Ido Schimmel wrote: > > It's unclear to me why you need the UDP port based demux. On a single > > node, do you have multiple pods that belong to the same tenant and > > therefore share the same VNI? From the above, my understanding is that > > the answer is "no". > > Good question, and I think my earlier framing sent you down the wrong > path, sorry about that. The honest answer is actually "yes". By design > a node can hold several VTEPs on the same VNI, as long as they belong > to different HA sets. Anti-affinity only keeps the two replicas within > one HA set off the same node. Two pods from different HA sets on the > same VNI are free to co-locate, and that co-location is the case that > needs a per-pod port. > > > Assuming the answer is "no", instead of having multiple VXLAN sockets > > in the host netns (each bound to a different port), why not have a > > single VXLAN socket bound to the standard port (4789) and let the > > VXLAN driver perform the demux based on the VNI? [...] creating > > multiple VXLAN devices in the host netns that only differ in their > > VNI and then move each to the relevant pod netns. > > For different-VNI pods that is a clean fit, and I want to try it out > regardless. The outer packet still carries the VNI, so a shared socket > can steer by VNI without decapsulating, and the host never touches the > tenant frame. > > Where it stops working for me is two same-VNI pods on one node. They > share the underlay IP, the port (4789), and the VNI, so the outer > header is the same for both: > > outer: NodeIP:4789, VNI=X -> pod A (HA set 1) > outer: NodeIP:4789, VNI=X -> pod B (HA set 2) > > There is nothing left in the outer header to steer on, so the host > would have to look past it and terminate the tunnel itself, which is > the VTEP role I would rather keep inside the pod. A distinct port per > pod puts the distinguishing bit back in the outer header, so the host > can stay a stateless outer-UDP forwarder. > > > Given the anti-affinity constraint, you will only need to load balance > > between pods on different nodes that use the standard port and that is > > already supported by the FDB nexthop groups. > > Agreed that the across-node load balancing already works. What the > VNI-demux version costs is packing density. With one shared underlay > IP per node, VNI demux allows at most one VTEP per (VNI, node), so N > same-VNI HA sets of two replicas need 2N nodes. With a per-pod port > the same sets fit on two nodes (replica 1 on node A, replica 2 on node > B, each pod on its own port), and anti-affinity still holds. For 5 > sets on one VNI that is 10 nodes versus 2.
OK, I understand. Each tenant has its own VNI, but it can have multiple HA sets on the same node and you want to distinguish between them using the UDP destination port. > > So the port is not strictly required, and your VNI-demux model is a > real option I will prototype. The tradeoff I am weighing is density > plus keeping the host out of the tenant datapath, against the uAPI > cost, which is small: NHA_FDB_PORT is the nexthop analog of the > per-entry NDA_PORT that already exists, it is control-plane only, and > a leg with no port behaves exactly as today. I don't particularly like NHA_FDB_PORT because it's an FDB-specific encapsulation attribute and these attributes naturally belong under NHA_ENCAP. That being said, I fail to see a better alternative given that FDB nexthops don't use NHA_ENCAP / LWT infra and probably shouldn't. It's a bigger change in the datapath and I expect that we will not need too many FDB-specific attributes in the future. If you are going to post v2, please provide a more detailed motivation in the cover letter. > > Happy to keep digging if it is useful. Thanks again for thinking it > through with me. In an earlier reply you wrote "shared inner MAC/IP". The "MAC" part was intentional because you need to load balance pure L2 traffic? IOW, if your traffic is only L3, can we do the load balancing using multipath routes in the overlay instead of using FDB nexthop groups? Related: What is the use case for NHA_FDB_VNI? In my experience, the VNI is only translated when routing traffic between tenants. Why do you need this in an FDB nexthop?
