Hello,
We are experiencing for the past weeks a segfault on haproxy processes when
reloading haproxy.
Each thread generates a coredump. Fortunately, this is the old process that
crashes, so there is no production impact.
The same behaviour happens with haproxy 2.0.25 compiled with OpenSSL 1.1.1l
Core was generated by `/root/haproxy -sf 42184 42185 42186 42187 42188
42189 42190 42191 42192 42193 4'.
Program terminated with signal 6, Aborted.
#0 0x00007f414c4d4495 in raise () from /lib64/libc.so.6
(gdb) bt full
#0 0x00007f414c4d4495 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f414c4d5c75 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00007f414c5123a7 in __libc_message () from /lib64/libc.so.6
No symbol table info available.
#3 0x00007f414c517dee in malloc_printerr () from /lib64/libc.so.6
No symbol table info available.
#4 0x00007f414c51ac3d in _int_free () from /lib64/libc.so.6
No symbol table info available.
#5 0x000000000047149c in ssl_sock_free_ssl_conf (conf=0x6f3d440) at
src/ssl_sock.c:3766
No locals.
#6 0x0000000000479f3d in ssl_sock_free_ssl_conf (conf=<optimized out>) at
src/ssl_sock.c:3764
No locals.
#7 ssl_sock_free_all_ctx (bind_conf=bind_conf@entry=0x6f3d160) at
src/ssl_sock.c:5144
node = 0x6f4ac60
sni = 0x6f4ac40
#8 0x000000000047a4a1 in ssl_sock_destroy_bind_conf (bind_conf=0x6f3d160)
at src/ssl_sock.c:5176
No locals.
#9 0x000000000050aae8 in deinit () at src/haproxy.c:2673
haproxy -vv output :
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
-Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
-Wno-missing-field-initializers -Wno-implicit-fallthrough
-Wno-stringop-overflow -Wtype-limits -Wshift-negative-value
-Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_THREAD=0 USE_STATIC_PCRE=1 USE_OPENSSL=1 USE_LUA=1
USE_ZLIB=1 USE_NS=
Feature list : -51DEGREES +ACCEPT4 -CLOSEFROM +CPU_AFFINITY +CRYPT_H
-DEVICEATLAS +DL +EPOLL -EVPORTS +FUTEX +GETADDRINFO -KQUEUE +LIBCRYPT
+LINUX_SPLICE +LINUX_TPROXY +LUA -MY_ACCEPT4 -MY_EPOLL -MY_SPLICE
+NETFILTER -NS -OBSOLETE_LINKER +OPENSSL -PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT
+POLL +PRCTL -PRIVATE_CACHE -PTHREAD_PSHARED -REGPARM +RT -SLZ +STATIC_PCRE
-STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -VSYSCALL -WURFL
+ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=32).
Built with OpenSSL version : OpenSSL 1.1.1t 7 Feb 2023
Running on OpenSSL version : OpenSSL 1.1.1t 7 Feb 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with zlib version : 1.2.13
Running on zlib version : 1.2.13
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.45 2021-06-15
Running on PCRE version : 8.45 2021-06-15
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTX side=FE|BE mux=H2
h2 : mode=HTTP side=FE mux=H2
<default> : mode=HTX side=FE|BE mux=H1
<default> : mode=TCP|HTTP side=FE|BE mux=PASS
Available services : none
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
I understand this is linked to some SSL config file, but fail to find what.
The haproxy config file is very large (> 1000 lines) with dozens of
frontend/backend. It will help me to know which one is it. Any "gdb" advice
how to narrow the issue down ?
Thanks,
Olivier
