On 26/11/2014 00:19, Willy Tarreau wrote:
On Tue, Nov 25, 2014 at 09:33:30PM +0000, Chris Allen wrote:
On 25/11/2014 18:08, Lukas Tribus wrote:
I think SSL/TLS termination is the only use case where HAProxy
saturates a CPU core of a current generation 3,4Ghz+ CPU, which is why
scaling SSL/TLS is more complex, requiring nbproc> 1.
Lukas
Ok that's strange then because we don't have a very complex config.
Does your config use a lot of regex ? This can cost a lot. Also if this
is the case, there's a bug in 1.5.8 which magnifies the time spent
processing regex when they're loaded from files (fixed in 1.5.9). Could
you please also post the output of "haproxy -vv" in case some optims are
missing.
No, there's one regexp on a rarely-used backend.
Here's our (redacted) config:
https://gist.github.com/chrisotherwise/cfdd04216d622437f5fe
Here's the output from haproxy -vv:
https://gist.github.com/chrisotherwise/23067a39fd59449d52f1
Here's the output from sar when we were not too busy:
https://gist.github.com/chrisotherwise/b2f22379eb5192ea5b45
Here's the output from a pidstat of haproxy:
https://gist.github.com/acme/02f065091d9f246ceaa3
About 8 front-ends of which only 3 are busy. About 10 back-end pools
each containing
30-60 servers. Things start getting hot when we're serving about 15,000
requests per second (about 1Gb/s bandwidth).
That's not huge.
Is there any way we can find out why we're using so much cpu?
You need to check first how your CPU usage is spread between user, system
and softirq. If you see a lot of user, something is wrong either in haproxy
or in the configuration.
If you can post your config (without the private parts, you can replace IP
addresses, domain names and passwords with XXXX), that will definitely help.
We used
perf to create a flame graph which is here:
http://www.cjx.com/perf-kernel.svg - taken when the system was not very
loaded - perhaps somebody with better
knowledge of haproxy than I can tell if something is awry...
OK that shows a reasonably loaded system. If you're close to 1 Gbps on a
Gig NIC (tg3), you can experience some losses which further increase the
softirq usage.
We have 3 x 1Gb NIC. Things get hot at around 1.1Gb/s
So first, we really need to see this CPU usage breakdown.
Regards,
Willy
