Hi Chris, On Wed, Nov 26, 2014 at 01:10:21PM +0000, Chris Allen wrote: > >Does your config use a lot of regex ? This can cost a lot. Also if this > >is the case, there's a bug in 1.5.8 which magnifies the time spent > >processing regex when they're loaded from files (fixed in 1.5.9). Could > >you please also post the output of "haproxy -vv" in case some optims are > >missing. > > No, there's one regexp on a rarely-used backend.
OK. > Here's our (redacted) config: > https://gist.github.com/chrisotherwise/cfdd04216d622437f5fe More servers than what we see on average but that should have no impact. I'm seeing that you have "option httpclose". Is there any reason for this ? Maybe this is a config inherited from much older versions ? Since 1.4 we support client-side keep-alive (option http-server-close) and since 1.5 we support end-to-end keepalive. The first one will at least speed up the page load time for your visitors by reducing the number of roundtrips. The second one will reduce the CPU usage when connecting to the server, especially if you're running with conntrack enabled. On the other hand, you're running with a low maxconn count, so you way want to experiment to see if it's worth it or not. I'm seeing that you're running in round-robin mode, so if you decide to go with keep-alive, I'd suggest to enable "option prefer-last-server" to send multiple consecurive requests from a same connection to the same server. > Here's the output from haproxy -vv: > https://gist.github.com/chrisotherwise/23067a39fd59449d52f1 OK, nothing to worry about here. > Here's the output from sar when we were not too busy: > https://gist.github.com/chrisotherwise/b2f22379eb5192ea5b45 Indeed, not busy at all. > Here's the output from a pidstat of haproxy: > https://gist.github.com/acme/02f065091d9f246ceaa3 This one is more concerning. We're on regular ratios between user and system. I don't know if you have ip_conntrack/nf_conntrack loaded and enabled. It can definitely increase the system cost, especially during the connect() to the backend server. Also if you're packet-bound, you can enable "option tcp-smart-connect" in the backends in order to save one packet when reaching the servers. > > If you're close to 1 Gbps on a > >Gig NIC (tg3), you can experience some losses which further increase the > >softirq usage. > > We have 3 x 1Gb NIC. Things get hot at around 1.1Gb/s How do you perform the load balancing there ? Using bonding maybe ? If so, are you certain your switch is configured accordingly (LACP or equivalent) ? Because what you're describing really reminds me of some failed early attempts at load balancing "the dirty way" using bonding and a dumb switch, leading to 1.1 Gbps as well... Regards, Willy
