Cheers Matthew.
1) Not seeing that error, seeing this one instead:
01-Jun-2017 01:46:27.952 client: warning: client 192.168.0.23#38125
(x41fe848-f3d1-4eec-967e-039d075ee864.perf1000): error sending response: would
block
Only seeing a few of them per run (out of ~70 million requests).
Whilst I can see where this is raised in the BIND code (lib/isc/unix/socket.c
in doio_send), I don't understand the underlying reason for it being set (errno
== EWOULDBLOCK || errno == EAGAIN).
I've not bumped wmem/rmem up as much as the link (only to 16MB, not 40MB), but
no real difference after tweaks. I did another run with stupidly-large
core.{rmem,wmem}_{max,default} (64MB), this actually degraded performance a bit
so over tuning isn't good either. Need to figure out a good balance here.
I'd love to figure out what the math here should be. 'X number of simultaneous
connections multiplied by Y socket memory size = rmem' or some such.
2) I am still seeing some udp receive errors and receive buffer errors; about
1.3% of received packets.
From a 'netstat' point of view, I see:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 382976 17664 192.168.1.21:53 0.0.0.0:*
The numbers in the receive queue stay in the 200-300k range whilst the
send-queue floats around the 20-40k range. wmem already bumped.
3) Huh, didn't know about this one. Bumped up the backlog, small increase in
throughput for my tests. Still need to figure out how to read sofnet_stat. More
google-fu in my future.
After a reboot and the wmem/rmem/backlog increases, no longer any non-zero in
the 2nd column.
4) Yes, max_dgram_qlen is already set to 512.
5) Oo! new tool! :)
--
...
11 drops at location 0xffffffff815df171
854 drops at location 0xffffffff815e1c64
12 drops at location 0xffffffff815df171
822 drops at location 0xffffffff815e1c64
...
--
I'm pretty sure it's just showing more details of the 'netstat -u -s'. More
google-fu to figure out how to use that information for good rather than, well,
.. frustration? .. :)
Will keep spinning test but using smaller increments to the wmem/rmem values,
see if I can eek anything more than 360k out of it.
Thanks for your suggestions Matthew!
Stuart
-----Original Message-----
From: Mathew Ian Eis [mailto:[email protected]]
Sent: Thursday, 1 June 2017 10:30 AM
To: [email protected]
Cc: Browne, Stuart
Subject: [EXTERNAL] Re: Tuning suggestions for high-core-count Linux servers
360k qps is actually quite good… the best I have heard of until now on EL was
180k [1]. There, it was recommended to manually tune the number of subthreads
with the -U parameter.
Since you’ve mentioned rmem/wmem changes, specifically you want to:
1. check for send buffer overflow; as indicated in named logs:
31-Mar-2017 12:30:55.521 client: warning: client 10.0.0.5#51342 (test.com):
error sending response: unset
fix: increase rmem via sysctl:
net.core.rmem_max
net.core.rmem_default
2. check for receive buffer overflow; as indicated by netstat:
# netstat -u -s
Udp:
34772479 packet receive errors
fix: increase wmem and backlog via sysctl:
net.core.wmem_max
net.core.wmem_default
… and other ideas:
3. check 2nd column in /proc/net/softnet_stat for any non-zero numbers
(indicating dropped packets).
If any are non-zero, increase net.core.netdev_max_backlog
4. You may also want to want to increase net.unix.max_dgram_qlen (although
since EL7 has default this to 512, this is not much of an issue - double check
that it is 512).
5. Try running dropwatch to see where packets are being lost. If it shows
nothing then you need to look outside the system. If it shows something you may
have a hint where to tune next.
Please post your outcomes in any case, since you are already having some
excellent results.
[1] https://lists.dns-oarc.net/pipermail/dns-operations/2014-April/011543.html
Regards,
Mathew Eis
Northern Arizona University
Information Technology Services
-----Original Message-----
From: bind-users <[email protected]> on behalf of "Browne,
Stuart" <[email protected]>
Date: Wednesday, May 31, 2017 at 12:25 AM
To: "[email protected]" <[email protected]>
Subject: Tuning suggestions for high-core-count Linux servers
Hi,
I've been able to get my hands on some rather nice servers with 2 x 12 core
Intel CPU's and was wondering if anybody had any decent tuning tips to get BIND
to respond at a faster rate.
I'm seeing that pretty much cpu count beyond a single die doesn't get any
real improvement. I understand the NUMA boundaries etc., but this hasn't been
my experience on previous iterations of the Intel CPU's, at least not this
dramatically. When I use more than a single die, CPU utilization continues to
match the core count however throughput doesn't increase to match.
All the testing I've been doing for now (dnsperf from multiple sources for
now) seems to be plateauing around 340k qps per BIND host.
Some notes:
- Primarily looking at UDP throughput here
- Intention is for high-throughput, authoritative only
- The zone files used for testing are fairly small and reside completely
in-memory; no disk IO involved
- RHEL7, bind 9.10 series, iptables 'NOTRACK' firmly in place
- Current configure:
built by make with '--build=x86_64-redhat-linux-gnu'
'--host=x86_64-redhat-linux-gnu' '--program-prefix='
'--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr'
'--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
'--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64'
'--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib'
'--mandir=/usr/share/man' '--infodir=/usr/share/info' '--localstatedir=/var'
'--with-libtool' '--enable-threads' '--enable-ipv6' '--with-pic'
'--enable-shared' '--disable-static' '--disable-openssl-version-check'
'--with-tuning=large' '--with-libxml2' '--with-libjson'
'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu'
'CFLAGS= -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64
-mtune=generic -fPIC' 'LDFLAGS=-Wl,-z,relro ' 'CPPFLAGS= -DDIG_SIGCHASE -fPIC'
Things tried:
- Using 'taskset' to bind to a single CPU die and limiting BIND to '-n'
cpu's doesn't improve much beyond letting BIND make its own decision
- NIC interfaces are set for TOE
- rmem & wmem changes (beyond a point) seem to do little to improve
performance, mainly just make throughput more consistent
I've yet to investigate the switch throughput or tweaking (don't yet have
access to it).
So, any thoughts?
Stuart
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe
from this list
bind-users mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/bind-users
