Hi all,
I'm looking for performance measuring and tuning advice for 10 gigabit
Ethernet. I have a pair of Lanner FW-8865 systems that will be used as
firewalls for the local network. Each one has a Xeon E3-1270v2 CPU,
Intel X540 10GbE NIC (PCIe 3.0 8x), and 8GB DDR3-1600 ECC RAM. Before
putting them into production I wanted to do some throughput testing,
so I connected one directly to the other (via ix0 interfaces) and used
iperf to see how much data I can push through. I also disabled pf for
now, but will do some additional testing with it enabled later on. The
kernel is 5.3 amd64 GENERIC.MP.
The initial iperf runs couldn't go beyond ~3.2 Gbps:
# server: iperf -s
# client: iperf -c 192.168.1.3
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.2 sec 3.84 GBytes 3.22 Gbits/sec
Increasing the TCP window size to 256 KB (seems to be the upper limit)
brings this up to ~4.2 Gbps:
# server: iperf -s -w 256k
# client: iperf -c 192.168.1.3 -w 256k
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 4.96 GBytes 4.22 Gbits/sec
Increasing the MTU on both ix0 interfaces to 9000 gives me ~7.2 Gbps:
# server: ifconfig ix0 mtu 9000 && iperf -s -w 256k
# client: ifconfig ix0 mtu 9000 && iperf -c 192.168.1.3 -w 256k -m
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 8.39 GBytes 7.21 Gbits/sec
[ 3] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)
This is where I'm stuck at the moment. When running iperf on
127.0.0.1, which should only test CPU and memory, I get 11.6 Gbps.
I've read the "Network Tuning and Performance Guide" @ calomel.org,
but none of the tips there help me in getting beyond 7 Gbps on the
physical interfaces.
I'm also slightly concerned about the performance at the default MTU
of 1500. Looking at `ifconfig ix0 hwfeatures` output (below), it seems
that the ix driver does not support any checksum offloading for the
X540. I wonder if that could be a reason for the poor performance?
ix0: flags=28843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,NOINET6> mtu 9000
hwfeatures=30<VLAN_MTU,VLAN_HWTAGGING> hardmtu 16110
lladdr 00:90:0b:56:12:0c
priority: 0
groups: LAN SVR
media: Ethernet autoselect (10GbaseT full-duplex)
status: active
inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
Any there any sysctl parameters that I should play with? Any other
system stats that I should monitor? I did a few runs while watching
`top` and `systat vmstat`, but didn't see any problem indications
there. I should also note that I couldn't run iperf in UDP mode - the
client segfaults any time I increase the bandwidth beyond 300 Mbps. No
idea why, but I'm more interested in TCP performance anyway.
- Max
