Hello,
I have found two performance issues with MACsec on 10 Gb/s links (tested
on Intel and Broadcom NICs):
1) MACsec with encryption is much faster than MACsec without encryption
(9.8 vs 7.4 Gb/s) until 5.7, where both have poor performance
2) 5.7 introduced a severe performance impact for MACsec with and
without encryption at commit ab046a5d4be4c90a3952a0eae75617b49c0cb01b
I haven't been able to look at issue #1 yet (and I don't know where to
start) since I got sidetracked looking at issue #2.
This email is about issue #2, which results in the following in my test
setup:
* MACsec with encryption drops from 9.81 Gb/s to 1.00 Gb/s or sometimes
worse
* MACsec without encryption drops from 7.40 Gb/s to 1.80 Gb/s
I have tested a number of configurations. These tests were performed on
the following hardware:
* dual Intel Xeon E5-2680 v4 @ 2.40GHz, 14 cores each
* Intel 82599ES 10 GbE NIC
* ixgbe driver, version 5.1.0-k
I also tested on the following hardware in a more limited fashion, but
the results were consistent:
* dual Intel Xeon E5-2670 v3 @ 2.30GHz, 12 cores each
* Broadcom BCM57810 10 GbE NIC
* bnx2x driver, 1.713.36-0
Only one 10 Gb/s link was populated (i.e. no port-channel). The MTU for
the network is 9000, with a resulting MACsec MTU of 8968. All tests
were performed with only one switch in between the servers.
I tested three scenarios. The tests were run on servers that are booted
with NFS root from an identical image. The only difference was the
kernel. A script was run to create the three scenarios and run the
benchmarks, so the setups are identical across tests.
The scenarios all involved iperf3 tests of these conditions:
1) no MACsec
2) MACsec without encryption
3) MACsec with encryption
The MACsec setup was done as follows:
ip link add link em1 ms1 type macsec sci 1234 encrypt on #or omitting
the "encrypt on" for specific tests
ip macsec add ms1 tx sa 0 pn 1234 on key 01 $(printf %032d 1234)
ip macsec add ms1 rx sci 1234
ip macsec add ms1 rx sci 1234 sa 0 pn 1234 on key 01 $(printf %032d 1234)
That results in `ip macsec show` like this:
6: ms1: protect on validate strict sc off sa off encrypt on send_sci on
end_station off scb off replay off
cipher suite: GCM-AES-128, using ICV length 16
TXSC: 0000000000001234 on SA 0
0: PN 599345, state on, key 01000000000000000000000000000000
RXSC: 0000000000001234, state on
0: PN 5076769, state on, key 01000000000000000000000000000000
I tested a number of kernels (all 64 bit) including:
* 4.18.0-193.13.2.el8_2 (RHEL 8)
* 5.6.7-1.el8.elrepo (ELRepo)
* 5.7.11-1.el8.elrepo (ELRepo)
* 5.7 at tag v5.7.11 (I compiled)
* 5.7 at tag v5.7.11 with ab046a5d4be4c90a3952a0eae75617b49c0cb01b
reverted (I compiled)
I did test 4.18 <-> 5.7 (bi-directional) and both directions resulted in
poor performance. Other than that, each test was between two servers of
the same kernel version.
CONFIG_CRYPTO_AES_NI_INTEL=y is set in all kernels.
4.18 and 5.6 kernels both have very similar performance characteristics:
* 9.89 Gb/s with no macsec at all
* 7.40 Gb/s with macsec WITHOUT encryption <--- not sure why, but
turning OFF encryption slowed things down
* 9.81 Gb/s with macsec WITH encryption
With 5.7 I get:
* 9.90 Gb/s with no macsec at all
* 1.80 Gb/s with macsec WITHOUT encryption
* 1.00 Gb/s (sometimes, but often less) with macsec WITH encryption
With 5.7 but with ab046a5d4be4c90a3952a0eae75617b49c0cb01b reverted, I get:
* 9.90 Gb/s with no macsec at all
* 7.33 Gb/s with macsec WITHOUT encryption
* 9.83 Gb/s with macsec WITH encryption
On tests where performance is bad (including macsec without encryption),
iperf3 is at 100% CPU usage. I was able to run it under `perf record`on
iperf3 in a number of the tests but, unfortunately, I have had trouble
compiling perf for my own 5.7 compilations (definitely PEBKAC). If it
would be useful I can work on fixing the perf compilation issues.
For 5.7.11-1.el8.elrepo (which has the issue) I get the following top 10
items in `perf report`:
* MACsec without encryption - iperf3 instance running as server
(receives data)
29.92% iperf3 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
6.48% iperf3 [kernel.kallsyms] [k] do_syscall_64
2.92% iperf3 [kernel.kallsyms] [k] syscall_return_via_sysret
2.37% iperf3 [kernel.kallsyms] [k] entry_SYSCALL_64
2.32% iperf3 [kernel.kallsyms] [k] __skb_datagram_iter
2.26% iperf3 [kernel.kallsyms] [k] __free_pages_ok
2.09% iperf3 [kernel.kallsyms] [k] tcp_poll
1.75% iperf3 [kernel.kallsyms] [k] do_select
1.48% iperf3 [kernel.kallsyms] [k] free_one_page
1.44% iperf3 [kernel.kallsyms] [k] kmem_cache_free
* MACsec without encryption - iperf3 instance running as client (sends data)
83.63% iperf3 [kernel.kallsyms] [k] gf128mul_4k_lle
3.46% iperf3 [kernel.kallsyms] [k] ghash_update
1.48% iperf3 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.18% iperf3 [kernel.kallsyms] [k] memcpy_erms
1.17% iperf3 [kernel.kallsyms] [k] do_csum
0.50% iperf3 [kernel.kallsyms] [k] _raw_spin_lock
0.44% iperf3 [kernel.kallsyms] [k] __copy_skb_header
0.36% iperf3 [kernel.kallsyms] [k] get_page_from_freelist
0.23% iperf3 [kernel.kallsyms] [k] ixgbe_xmit_frame_ring
0.22% iperf3 [kernel.kallsyms] [k] skb_segment
* MACsec with encryption - iperf3 instance running as server (receives data)
15.66% iperf3 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
9.52% iperf3 [kernel.kallsyms] [k] do_syscall_64
3.76% iperf3 [kernel.kallsyms] [k] syscall_return_via_sysret
3.28% iperf3 [kernel.kallsyms] [k] entry_SYSCALL_64
3.22% iperf3 [kernel.kallsyms] [k] do_select
2.71% iperf3 [kernel.kallsyms] [k] tcp_poll
1.84% iperf3 [kernel.kallsyms] [k] tcp_recvmsg
1.59% iperf3 [kernel.kallsyms] [k] sock_poll
1.38% iperf3 [kernel.kallsyms] [k] __skb_datagram_iter
1.37% iperf3 [kernel.kallsyms] [k] __free_pages_ok
* MACsec with encryption - iperf3 instance running as client (sends data)
43.95% iperf3 [kernel.kallsyms] [k] gf128mul_4k_lle
17.48% iperf3 [kernel.kallsyms] [k] _aesni_enc1
9.42% iperf3 [kernel.kallsyms] [k] kernel_fpu_begin
7.75% iperf3 [kernel.kallsyms] [k] __crypto_xor
3.18% iperf3 [kernel.kallsyms] [k] crypto_ctr_crypt
2.67% iperf3 [kernel.kallsyms] [k] crypto_inc
2.30% iperf3 [kernel.kallsyms] [k] aesni_encrypt
2.05% iperf3 [kernel.kallsyms] [k] aesni_enc
1.87% iperf3 [kernel.kallsyms] [k] ghash_update
1.03% iperf3 [kernel.kallsyms] [k] kernel_fpu_end
Here is `ethtool -k em1` in case that is helpful:
Features for em1:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: on [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: on [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: on
tx-udp-segmentation: on
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: on
esp-tx-csum-hw-offload: on
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
I have lots of logs that I can provide if needed.
I thank Antoine Tenart for suggesting tests for this issue and for
narrowing down which commits to check.
Thanks,
Ryan
