W dniu 01.11.2018 o 12:09, Paweł Staszewski pisze:
rx_cqe_compress_pkts: 0
If this is a pcie bottleneck it might be useful to enable CQE
compression (to reduce PCIe completion descriptors transactions)
you should see the above rx_cqe_compress_pkts increasing when enabled.
$ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
$ ethtool --show-priv-flags enp175s0f1
Private flags for p6p1:
rx_cqe_moder : on
cqe_moder : off
rx_cqe_compress : on
...
try this on both interfaces.
Done
ethtool --show-priv-flags enp175s0f1
Private flags for enp175s0f1:
rx_cqe_moder : on
tx_cqe_moder : off
rx_cqe_compress : on
rx_striding_rq : off
rx_no_csum_complete: off
ethtool --show-priv-flags enp175s0f0
Private flags for enp175s0f0:
rx_cqe_moder : on
tx_cqe_moder : off
rx_cqe_compress : on
rx_striding_rq : off
rx_no_csum_complete: off
Enabling cqe compress changes nothing after reaching 64Gbit RX /
64Gbit/s TX on interfaces cpu's are saturated at 100%
ethtool -S enp175s0f1 | grep rx_cqe_compress
rx_cqe_compress_blks: 5657836379
rx_cqe_compress_pkts: 13153761080
ethtool -S enp175s0f0 | grep rx_cqe_compress
rx_cqe_compress_blks: 5994612500
rx_cqe_compress_pkts: 13579014869
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
- iface Rx Tx Total
==============================================================================
enp175s0f1: 27.03 Gb/s 37.09 Gb/s
64.12 Gb/s
enp175s0f0: 36.84 Gb/s 26.82 Gb/s
63.66 Gb/s
------------------------------------------------------------------------------
total: 63.85 Gb/s 63.87 Gb/s 127.72 Gb/s
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
/ iface Rx Tx Total
==============================================================================
enp175s0f1: 3.22 GB/s 4.26 GB/s
7.48 GB/s
enp175s0f0: 4.24 GB/s 3.21 GB/s
7.45 GB/s
------------------------------------------------------------------------------
total: 7.46 GB/s 7.47 GB/s
14.93 GB/s
mpstat
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: all 0.05 0.00 0.19 0.02 0.00 42.74 0.00
0.00 0.00 56.99
Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 1 0.00 0.00 0.30 0.00 0.00 0.00 0.00
0.00 0.00 99.70
Average: 2 0.00 0.00 0.20 0.00 0.00 0.00 0.00
0.00 0.00 99.80
Average: 3 0.00 0.00 0.20 1.20 0.00 0.00 0.00
0.00 0.00 98.60
Average: 4 0.10 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 5 0.00 0.00 0.10 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 6 0.10 0.00 0.20 0.00 0.00 0.00 0.00
0.00 0.00 99.70
Average: 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 11 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 12 1.40 0.00 4.50 0.00 0.00 0.00 0.00
0.00 0.00 94.10
Average: 13 0.00 0.00 1.60 0.00 0.00 0.00 0.00
0.00 0.00 98.40
Average: 14 0.00 0.00 0.00 0.00 0.00 84.10 0.00
0.00 0.00 15.90
Average: 15 0.00 0.00 0.10 0.00 0.00 93.70 0.00
0.00 0.00 6.20
Average: 16 0.00 0.00 0.10 0.00 0.00 94.31 0.00
0.00 0.00 5.59
Average: 17 0.00 0.00 0.00 0.00 0.00 95.30 0.00
0.00 0.00 4.70
Average: 18 0.00 0.00 0.00 0.00 0.00 62.80 0.00
0.00 0.00 37.20
Average: 19 0.00 0.00 0.10 0.00 0.00 98.90 0.00
0.00 0.00 1.00
Average: 20 0.00 0.00 0.00 0.00 0.00 99.30 0.00
0.00 0.00 0.70
Average: 21 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 22 0.00 0.00 0.00 0.00 0.00 99.90 0.00
0.00 0.00 0.10
Average: 23 0.00 0.00 0.10 0.00 0.00 99.90 0.00
0.00 0.00 0.00
Average: 24 0.00 0.00 0.10 0.00 0.00 97.10 0.00
0.00 0.00 2.80
Average: 25 0.00 0.00 0.00 0.00 0.00 64.06 0.00
0.00 0.00 35.94
Average: 26 0.00 0.00 0.10 0.00 0.00 88.50 0.00
0.00 0.00 11.40
Average: 27 0.00 0.00 0.00 0.00 0.00 94.10 0.00
0.00 0.00 5.90
Average: 28 0.80 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 99.20
Average: 29 0.00 0.00 0.10 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 30 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 31 0.20 0.00 0.80 0.00 0.00 0.00 0.00
0.00 0.00 99.00
Average: 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 35 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 36 0.20 0.00 0.40 0.00 0.00 0.00 0.00
0.00 0.00 99.40
Average: 37 0.00 0.00 0.10 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 38 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 39 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 40 0.10 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 99.90
Average: 41 0.10 0.00 1.20 0.00 0.00 0.00 0.00
0.00 0.00 98.70
Average: 42 0.00 0.00 0.10 0.00 0.00 78.92 0.00
0.00 0.00 20.98
Average: 43 0.00 0.00 0.00 0.00 0.00 81.00 0.00
0.00 0.00 19.00
Average: 44 0.00 0.00 0.00 0.00 0.00 82.58 0.00
0.00 0.00 17.42
Average: 45 0.00 0.00 0.00 0.00 0.00 68.97 0.00
0.00 0.00 31.03
Average: 46 0.00 0.00 0.10 0.00 0.00 79.20 0.00
0.00 0.00 20.70
Average: 47 0.00 0.00 0.00 0.00 0.00 71.33 0.00
0.00 0.00 28.67
Average: 48 0.00 0.00 0.10 0.00 0.00 72.40 0.00
0.00 0.00 27.50
Average: 49 0.00 0.00 0.00 0.00 0.00 90.79 0.00
0.00 0.00 9.21
Average: 50 0.00 0.00 0.10 0.00 0.00 93.20 0.00
0.00 0.00 6.70
Average: 51 0.00 0.00 0.00 0.00 0.00 91.70 0.00
0.00 0.00 8.30
Average: 52 0.00 0.00 0.10 0.00 0.00 79.90 0.00
0.00 0.00 20.00
Average: 53 0.00 0.00 0.00 0.00 0.00 76.20 0.00
0.00 0.00 23.80
Average: 54 0.00 0.00 0.00 0.00 0.00 89.59 0.00
0.00 0.00 10.41
Average: 55 0.00 0.00 0.10 0.00 0.00 65.97 0.00
0.00 0.00 33.93
So yes it looks like pcie x16 limit and pcie is 8/8GB/s both directions
- 16GB/s one direction
So 100Gbit/s network controller need to have pcie x32 :)
I understand in normal server host scenario (no forwarding) most traffic
is outbound or inbound - not many situations where we have 100G input
and 100G output.
I will replace then this 2 port 100G nic with two connect-x 5 100G nic's
installed in two different pcie x16
