thanks, most of that i've already tried. i don't have iperf, but i did run qperf, which shows 1.0GB/sec and 16us latency between the two hosts. but i'm not entirely sure i believe it
the only thing that made a difference so far is changing the rx tx rings up 8192 on the host. i went from 200-300MB/sec to 300-400MB/sec consistently. but i'm still far short On Fri, Aug 4, 2017 at 12:47 PM, Alex Chekholko <alex.chekho...@gmail.com> wrote: > The ESNet pages are pretty good: > https://fasterdata.es.net/host-tuning/ > https://fasterdata.es.net/network-tuning/ > > Since you're on a local low-latency link, all the stuff about high-bandwidth > high-latency links doesn't apply to you, but the host tuning parameters at > least give you somewhere to look. > > I recommend iperf for testing before IMB, make sure you can get line rate > throughput with an iperf test. > > On Fri, Aug 4, 2017 at 8:52 AM Michael Di Domenico <mdidomeni...@gmail.com> > wrote: >> >> does anyone have any additional tuning tips beyond the random things >> on the net for 10g ethernet tuning? i have two identical servers >> (Quanta Q71L-4U) attached to a single cisco 10g switch, using >> identical (Mellanox MNPH28B-XTC) cards. when i run mpi bandwidth >> tests between the two cards, i top out at 200-300MB/sec using IMB >> sendrecv. if i use lustre lnet selftests, i can crank the max to >> 500MB/sec, but i can't seem to crest this. >> >> i've monkied around with the sysctl params and the cards are in x8 >> slots at 2.5GT/s. i think they should be capable of driving at >> ~900MB/sec, but i can't seem to get there. i know i'm missing >> something... >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
