*So my 10G/40G network performance sucks. I have a combination of
Chelsio and Intel 10 and 40G NICs. The 10G Intels do a little better. I
have dug deeper into systems with Chelsio cards than Intel cards but
what I found is that the receive buffer on the Chelsio card fills up and
then it starts sending flow control frames. I have been unable to alter
this behavior. I would say the average top rate for Chelsio is about
3Gbps (independent of whether it is a 10G/40G card)and the Intel cards
and reach but cannnot sustain 5Gbps and then regress towards 3Gbps.
The base use case is backing up data to four systems. Some of them are
Intel/Kontron S5520UR with X5650 CPUs and Chelsio T520-LL-CR cards and
the others are SuperMicro H11SSL-C EPYC 7401 based systems with
T580-SO-CR 40G cards all connected to Force10/Dell S4048-ON switches. No
errors on any interfaces other than flow control frames coming from the
servers. The performance is the same running back to back on 10G or 40G.
I setup two systems back to back and had Chelsio login to take a look.
They they set some tcp buffers and locked one ore more CPUs to the NICs
and then gave up. Here are somethings that I have tried.
*
* ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 15000
ndd -set /dev/tcp tcp_time_wait_interval 10000
# default 128
ndd -set /dev/tcp tcp_conn_req_max_q 4096
# setup for high transaction volume and drop super slow talkers
# default 1024
ndd -set /dev/tcp tcp_conn_req_max_q0 98304
# default 180000
ndd -set /dev/tcp tcp_keepalive_interval 600000
# default 180000
ndd -set /dev/tcp tcp_ip_abort_cinterval 30000
# default 60000
ndd -set /dev/tcp tcp_rexmit_interval_max 240000
# default 32768
/usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 2049
# hpn-ssh related values
ndd -set /dev/tcp tcp_recv_hiwat 491520
ndd -set /dev/tcp tcp_xmit_hiwat 491520
ndd -set /dev/tcp tcp_cwnd_max 655340
*
* # disable lso
ndd -set /dev/ip ip_lso_outbound 0# increase tcp buffersipadm set-prop
-p max_buf=4194304 tcp ipadm set-prop -p recv_buf=1048576 tcp ipadm
set-prop -p send_buf=1048576 tcp What are you doing differently to get
something near wired speed performance? I will take any delta I can get.
Thanks,*
*j. *
_______________________________________________
openindiana-discuss mailing list
[email protected]
https://openindiana.org/mailman/listinfo/openindiana-discuss
