"I found a number of recommendations for the things to keep an eye on, but nothing that gave me any ideas on what else to try for improving the performance. "
Particularly with OpenBSD you will find that the clock speed of the core that the kernel is running on will be a limiting factor for throughput. The kernel only runs on one core at a time, and since a packet stream is linear you can't proccess more packets than you have instruction cycles on the cpu. So you will find an entertaining correlation between a CPU clock and the throughput. One trick as you may have noticed is to adjust the MTU size. Bigger packets allow you to move more data. Again the CPU clock is a limiting factor. As you add other activities that the kernel must address like pf firewalling, that will take away from time the CPU has to deal with packets. Hope this helps! I'd be keen to see the results of AMD's FX 9590 which chugs along at 4.7Ghz or more. ________________________________ From: Maxim Khitrov <[email protected]> To: OpenBSD general usage list <[email protected]> Sent: Wednesday, August 7, 2013 12:55 PM Subject: Re: 10GbE (Intel X540) performance on OpenBSD 5.3 On Wed, Aug 7, 2013 at 10:31 AM, Martin Schröder <[email protected]> wrote: > 2013/8/7 Maxim Khitrov <[email protected]>: >> I've read the "Network Tuning and Performance Guide" @ calomel.org, > > Ignore that site and search the list archives. Understood :) I found a number of recommendations for the things to keep an eye on, but nothing that gave me any ideas on what else to try for improving the performance. Specifically, I looked at netstat -m on both systems, where everything was well below the limits (<500 mbufs in use during the test). I see about 8200/8700 (ix0/total) interrupts in systat with 1500 MTU. CPU usage in top is split between two cores, one at ~80% interrupt and the other at ~80% system. Most of the time all four cores are at least 10% idle (hyper-threading is disabled in BIOS). netstat -i shows no errors for ix0 and sysctl net.inet.ip.ifq.drops is at 0 on both systems. What did surprise me is that netstat -ss (output below) shows that all received packets were hardware-checksummed, but this value is 0 for sent packets. Does this mean that ix supports checksum offloading, but only for inbound packets? This should be a bit of good news for me once I start testing forwarding performance. I assume that as long as pf doesn't modify the packet (no nat/rdr, modulate state, scrubbing, etc.), then there shouldn't be any need to recompute the checksum. Correct? ip: 39827125 total packets received 39820936 packets for this host 40 packets for unknown/unsupported protocol 77150033 packets sent from this host 39826536 input datagrams checksum-processed by hardware icmp: 147 calls to icmp_error Output packet histogram: destination unreachable: 48 Input packet histogram: echo reply: 2 destination unreachable: 40 igmp: ipencap: tcp: 77147020 packets sent 77145183 data packets (111695427326 bytes) 2 data packets (2836 bytes) retransmitted 1763 ack-only packets (4427 delayed) 6 window update packets 66 control packets 39817607 packets received 38983910 acks (for 111695426667 bytes) 18 duplicate acks 5814 packets (560082 bytes) received in-sequence 38 out-of-order packets (872 bytes) 830155 window update packets 1 packet received after close 39817153 packets hardware-checksummed 41 connection requests 10 connection accepts 49 connections established (including accepts) 43 connections closed (including 1 drop) 38983035 segments updated rtt (of 1217192 attempts) 4 retransmit timeouts 2 keepalive timeouts 2 keepalive probes sent 601 correct ACK header predictions 3276 correct data packet header predictions 20 PCB cache misses cwr by timeout: 4 10 SYN cache entries added 10 completed 3 SACK options received 1 SACK option sent udp: 3327 datagrams received 39 with no checksum 3193 input packets hardware-checksummed 47 dropped due to no socket 3280 delivered 2958 datagrams output 708 missed PCB cache esp: ah: etherip: ipcomp: carp: pfsync: divert: pflow: ip6: 2 total packets received 4 packets sent from this host Input packet histogram: ICMP6: 2 Mbuf statistics: 2 one ext mbufs divert6: icmp6: Output packet histogram: multicast listener report: 4 Histogram of error messages to be generated: pim6: rip6:
