Hi, this setup does not run mod_proxy_http2. Only mod_http2 was enabled. Semantic is quite important as CPU Usage isn't really "load average", but I will just give you a link for the "load average" question which will explain it better than I will ever do ;) : https://en.wikipedia.org/wiki/Load_(computing)
We do have a varnish behind apache with KeepAlive being disabled. So it might be related to this application behind apache as well, not surprised as the previous "segfault" bug I was refering to, was noticed on maybe 3 or 4 servers, while we manage hundreds of them, so it might be dependant to how they handle connections/pipe... I'm asking, just in case is keepalive necessary with http2 ? Regards, Fabien Le mar. 30 mai 2023 à 16:25, 'Stefan Eissing' via Sysadmin < sysad...@dutiko.com> a écrit : > Thanks for the information. > > Does this setup also run mod_proxy_http2? (I am investigating issue in > that module that I can somewhat reproduce) > What measurement is the "load average" you mentioned? CPU usage? > > As to the observed log entries, these appear when a client connection is > terminated and the HTTP/2 processing waits on running H2 workers to react > to an abort and stop processing. All stream (e.g. running workers) that do > not return after 60 seconds are logged and the waiting continues. After > another 60 seconds all remaining are logged again and so on. > > So if these entries for a particular connection (you see in the > "h2_stream(3805413-327-1" that it is in process 3805413 connection 327 and > stream id 1) > continue to be logged every minute, this points to a server mistake. If > the logging of such stream stops eventually, this points to badly reacting > request processing and it is the question what is running in your server > that takes this long to finish/react to an aborted connection). > > For example, if you proxy a backend with very long timeout and that > backend does not respond, these entries may happen. > > tl;dr > > The log entries could be a server bug, but as well a matter of > configuration. > > Kind Regards, > Stefan > > > Am 30.05.2023 um 11:42 schrieb root <sysad...@dutiko.com>: > > > > Package: apache2 > > Version: 2.4.56-1~deb11u2 > > Followup-For: Bug #1036213 > > > > > > I have additional information that might be related to this bug. > > It seems to be following the previous bug I opened in Bug#1033408 , > > which is now solved since 2.4.56-1~deb11u2, but now we don't have > > segfaults anymore, but the server of our customer is having a load > average > > increasing since we've updated the package, we've tested for a week and > > load average went from around 5, to 35, slowly increasing over the week > > to this value, and it would probably have increased more if we hadn't > > intervene. > > > > After witnessing this behavior I noticed these logs in error logs which > > I had never seen before: > > [...] > > [Tue May 30 08:59:29.873462 2023] [http2:warn] [pid 3805413:tid > 140434122716928] [client 81.204.51.61:56550] > h2_stream(3805413-327-1,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > > [Tue May 30 08:59:29.873486 2023] [http2:warn] [pid 3805413:tid > 140434122716928] [client 81.204.51.61:56550] > h2_stream(3805413-327-5,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > > [Tue May 30 08:59:29.873490 2023] [http2:warn] [pid 3805413:tid > 140434122716928] [client 81.204.51.61:56550] > h2_stream(3805413-327-9,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > > [Tue May 30 09:07:11.128774 2023] [http2:warn] [pid 3808854:tid > 140434047182592] [client 106.245.192.226:56994] > h2_stream(3808854-230-1,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > > [Tue May 30 09:07:11.128793 2023] [http2:warn] [pid 3808854:tid > 140434047182592] [client 106.245.192.226:56994] > h2_stream(3808854-230-5,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > > [Tue May 30 09:13:36.366838 2023] [http2:warn] [pid 3811558:tid > 140434114324224] [client 51.179.98.234:49225] > h2_stream(3811558-357-17,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > > [Tue May 30 09:13:36.366874 2023] [http2:warn] [pid 3811558:tid > 140434114324224] [client 51.179.98.234:49225] > h2_stream(3811558-357-19,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > > [Tue May 30 09:55:14.468946 2023] [http2:warn] [pid 3832829:tid > 140434089146112] [client 109.234.73.147:53640] > h2_stream(3832829-35-5,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > > [Tue May 30 09:56:14.469032 2023] [http2:warn] [pid 3832829:tid > 140434089146112] [client 109.234.73.147:53640] > h2_stream(3832829-35-5,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > > [...] > > > > After noticing these, I immediately thought about the previous bug and > > completely disabled the http2 module in apache. It immediately solved > > the load issue. > > I know it's very few information and I'm not sure it's related exactly > > to his particular bug, might need a bug report of its own, but in case, > > I don't want to duplicate and will let you choose. > > > > > > > > -- Package-specific info: > > > > -- System Information: > > Debian Release: 11.7 > > APT prefers stable-updates > > APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, > 'stable') > > Architecture: amd64 (x86_64) > > > > Kernel: Linux 5.10.0-18-amd64 (SMP w/32 CPU threads) > > Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE > not set > > Shell: /bin/sh linked to /usr/bin/dash > > Init: systemd (via /run/systemd/system) > > LSM: AppArmor: enabled > > > > Versions of packages apache2 depends on: > > ii apache2-bin 2.4.56-1~deb11u2 > > ii apache2-data 2.4.56-1~deb11u2 > > ii apache2-utils 2.4.56-1~deb11u2 > > ii dpkg 1.20.12 > > ii init-system-helpers 1.60 > > ii lsb-base 11.1.0 > > ii mime-support 3.66 > > ii perl 5.32.1-4+deb11u2 > > ii procps 2:3.3.17-5 > > > > Versions of packages apache2 recommends: > > ii ssl-cert 1.1.0+nmu1 > > > > Versions of packages apache2 suggests: > > pn apache2-doc <none> > > pn apache2-suexec-pristine | apache2-suexec-custom <none> > > ii lynx [www-browser] 2.9.0dev.6-3~deb11u1 > > > > Versions of packages apache2-bin depends on: > > ii libapr1 1.7.0-6+deb11u2 > > ii libaprutil1 1.6.1-5+deb11u1 > > ii libaprutil1-dbd-sqlite3 1.6.1-5+deb11u1 > > ii libaprutil1-ldap 1.6.1-5+deb11u1 > > ii libbrotli1 1.0.9-2+b2 > > ii libc6 2.31-13+deb11u6 > > ii libcrypt1 1:4.4.18-4 > > ii libcurl4 7.74.0-1.3+deb11u7 > > ii libjansson4 2.13.1-1.1 > > ii libldap-2.4-2 2.4.57+dfsg-3+deb11u1 > > ii liblua5.3-0 5.3.3-1.1+b1 > > ii libnghttp2-14 1.43.0-1 > > ii libpcre3 2:8.44-2+0~20210301.9+debian11~1.gbpa278ad > > ii libssl1.1 1.1.1n-0+deb11u4 > > ii libxml2 > 2.9.14+dfsg-0.1+0~20230421.14+debian11~1.gbpf14485 > > ii perl 5.32.1-4+deb11u2 > > ii zlib1g 1:1.2.11.dfsg-2+deb11u2 > > > > Versions of packages apache2-bin suggests: > > pn apache2-doc <none> > > pn apache2-suexec-pristine | apache2-suexec-custom <none> > > ii lynx [www-browser] 2.9.0dev.6-3~deb11u1 > > > > Versions of packages apache2 is related to: > > ii apache2 2.4.56-1~deb11u2 > > ii apache2-bin 2.4.56-1~deb11u2 > > > > -- Configuration Files: > > /etc/apache2/apache2.conf changed [not included] > > /etc/apache2/mods-available/mpm_event.conf changed [not included] > > /etc/apache2/ports.conf changed [not included] > > /etc/apache2/sites-available/000-default.conf changed [not included] > > > > -- no debconf information > > > > -- *Fabien Le Berre** Homme de la situation* 01 86 95 54 04 - 37 rue des Mathurins - 75008 Paris
