Hi, Naturally, I can't be certain that the "in my mind" trivial VCL changes can't be the culprit. But I just can't see the logic in those changes causing this massive change in memory usage. But I'll summarize the changes here, and maybe you can identify a suspect:
* Modified the xkey header used by the xkey vmod, adding the id of the current website * Modified the TTL, from 1w to 1h, for a specific type of resource existing in maybe 20 versions (ie different urls), each being about 5 kB in size * Modified the backend probe url, from the startpage (ie full html) to a dedicated healthcheck endpoint (much smaller footprint, and much quicker) That's it. That's all the VCL changes we made in that deployment. And, like I said, we did no changes in the OS or Varnish config. > check the difference in passes, if they are about the same, look for > hit-for-misses, We don't have those statistics from the old server, so I can't do a comparison. But here are the current statistics: MAIN.s_pass 180721 0.14 Total pass-ed requests seen MAIN.cache_hitpass 0 0.00 Cache hits for pass. MAIN.cache_hit 3718468 2.86 Cache hits MAIN.cache_hit_grace 53903 0.04 Cache grace hits MAIN.cache_hitpass 0 0.00 Cache hits for pass. MAIN.cache_hitmiss 1129 0.00 Cache hits for miss. > and lastly, look at how long Varnish is trying to cache the average object. I'm not sure how I do that. Is there a varnishstat counter I can look at? > which packages are you using? Instead of giving you the full list, I guess it makes more sense to just list the one that differ. Below is a diff output of "apt list --installed" of before and after the deploy. Regards 2c2 < accountsservice/now 0.6.55-0ubuntu12~20.04.5 amd64 [installed,upgradable to: 0.6.55-0ubuntu12~20.04.7] --- > accountsservice/focal-updates,focal-security,now 0.6.55-0ubuntu12~20.04.7 > amd64 [installed,automatic] 23,25c23,25 < bind9-dnsutils/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: 1:9.16.48-0ubuntu0.20.04.1] < bind9-host/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: 1:9.16.48-0ubuntu0.20.04.1] < bind9-libs/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: 1:9.16.48-0ubuntu0.20.04.1] --- > bind9-dnsutils/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1 > amd64 [installed,automatic] > bind9-host/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1 amd64 > [installed,automatic] > bind9-libs/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1 amd64 > [installed,automatic] 31c31 < bsdutils/focal-security,now 1:2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 1:2.34-0.1ubuntu9.4] --- > bsdutils/now 1:2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 1:2.34-0.1ubuntu9.6] 41c41 < cloud-init/now 22.4.2-0ubuntu0~20.04.2 all [installed,upgradable to: 23.4.4-0ubuntu0~20.04.1] --- > cloud-init/now 22.4.2-0ubuntu0~20.04.2 all [installed,upgradable to: > 24.1.3-0ubuntu1~20.04.1] 48c48 < cpio/focal-updates,focal-security,now 2.13+dfsg-2ubuntu0.3 amd64 [installed,automatic] --- > cpio/now 2.13+dfsg-2ubuntu0.3 amd64 [installed,upgradable to: > 2.13+dfsg-2ubuntu0.4] 56c56 < curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.21 amd64 [installed] --- > curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.22 amd64 [installed] 68c68 < distro-info-data/now 0.43ubuntu1.11 all [installed,upgradable to: 0.43ubuntu1.15] --- > distro-info-data/now 0.43ubuntu1.11 all [installed,upgradable to: > 0.43ubuntu1.16] 82c82 < fdisk/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > fdisk/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6] 119,120c119,120 < grub-efi-amd64-bin/now 2.06-2ubuntu14.1 amd64 [installed,upgradable to: 2.06-2ubuntu14.4] < grub-efi-amd64-signed/now 1.187.3~20.04.1+2.06-2ubuntu14.1 amd64 [installed,upgradable to: 1.187.6~20.04.1+2.06-2ubuntu14.4] --- > grub-efi-amd64-bin/focal-updates,focal-security,now 2.06-2ubuntu14.4 amd64 > [installed] > grub-efi-amd64-signed/focal-updates,focal-security,now > 1.187.6~20.04.1+2.06-2ubuntu14.4 amd64 [installed] 148c148 < klibc-utils/focal-updates,focal-security,now 2.0.7-1ubuntu5.1 amd64 [installed,automatic] --- > klibc-utils/focal-updates,focal-security,now 2.0.7-1ubuntu5.2 amd64 > [installed,automatic] 152c152 < landscape-common/focal-updates,now 19.12-0ubuntu4.3 amd64 [installed,automatic] --- > landscape-common/now 19.12-0ubuntu4.3 amd64 [installed,upgradable to: > 23.02-0ubuntu1~20.04.2] 154,155c154,155 < less/now 551-1ubuntu0.1 amd64 [installed,upgradable to: 551-1ubuntu0.2] < libaccountsservice0/now 0.6.55-0ubuntu12~20.04.5 amd64 [installed,upgradable to: 0.6.55-0ubuntu12~20.04.7] --- > less/now 551-1ubuntu0.1 amd64 [installed,upgradable to: 551-1ubuntu0.3] > libaccountsservice0/focal-updates,focal-security,now 0.6.55-0ubuntu12~20.04.7 > amd64 [installed,automatic] 175c175 < libblkid1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > libblkid1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] 187,190c187,190 < libc-bin/now 2.31-0ubuntu9.9 amd64 [installed,upgradable to: 2.31-0ubuntu9.14] < libc-dev-bin/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 [installed,automatic] < libc6-dev/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 [installed,automatic] < libc6/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 [installed,automatic] --- > libc-bin/now 2.31-0ubuntu9.9 amd64 [installed,upgradable to: 2.31-0ubuntu9.15] > libc-dev-bin/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 > [installed,automatic] > libc6-dev/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 > [installed,automatic] > libc6/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 > [installed,automatic] 203,204c203,204 < libcurl3-gnutls/now 7.68.0-1ubuntu2.15 amd64 [installed,upgradable to: 7.68.0-1ubuntu2.21] < libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.21 amd64 [installed,automatic] --- > libcurl3-gnutls/now 7.68.0-1ubuntu2.15 amd64 [installed,upgradable to: > 7.68.0-1ubuntu2.22] > libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.22 amd64 > [installed,automatic] 227c227 < libfdisk1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > libfdisk1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] 244,246c244,246 < libglib2.0-0/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: 2.64.6-1~ubuntu20.04.6] < libglib2.0-bin/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: 2.64.6-1~ubuntu20.04.6] < libglib2.0-data/now 2.64.6-1~ubuntu20.04.4 all [installed,upgradable to: 2.64.6-1~ubuntu20.04.6] --- > libglib2.0-0/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: > 2.64.6-1~ubuntu20.04.7] > libglib2.0-bin/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: > 2.64.6-1~ubuntu20.04.7] > libglib2.0-data/now 2.64.6-1~ubuntu20.04.4 all [installed,upgradable to: > 2.64.6-1~ubuntu20.04.7] 248c248 < libgnutls30/now 3.6.13-2ubuntu1.7 amd64 [installed,upgradable to: 3.6.13-2ubuntu1.10] --- > libgnutls30/now 3.6.13-2ubuntu1.7 amd64 [installed,upgradable to: > 3.6.13-2ubuntu1.11] 278c278 < libklibc/focal-updates,focal-security,now 2.0.7-1ubuntu5.1 amd64 [installed,automatic] --- > libklibc/focal-updates,focal-security,now 2.0.7-1ubuntu5.2 amd64 > [installed,automatic] 284c284 < libldap-2.4-2/now 2.4.49+dfsg-2ubuntu1.9 amd64 [installed,upgradable to: 2.4.49+dfsg-2ubuntu1.10] --- > libldap-2.4-2/focal-updates,focal-security,now 2.4.49+dfsg-2ubuntu1.10 amd64 > [installed,automatic] 301c301 < libmount1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > libmount1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] 306,307c306,307 < libncurses6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: 6.2-0ubuntu2.1] < libncursesw6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: 6.2-0ubuntu2.1] --- > libncurses6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 > [installed,automatic] > libncursesw6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 > [installed,automatic] 314c314 < libnghttp2-14/focal,now 1.40.0-1build1 amd64 [installed,upgradable to: 1.40.0-1ubuntu0.2] --- > libnghttp2-14/focal,now 1.40.0-1build1 amd64 [installed,upgradable to: > 1.40.0-1ubuntu0.3] 316,318c316,318 < libnspr4/focal,now 2:4.25-1 amd64 [installed,automatic] < libnss-systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] < libnss3/now 2:3.49.1-1ubuntu1.8 amd64 [installed,upgradable to: 2:3.49.1-1ubuntu1.9] --- > libnspr4/focal,now 2:4.25-1 amd64 [installed,upgradable to: > 2:4.35-0ubuntu0.20.04.1] > libnss-systemd/focal-security,now 245.4-4ubuntu3.20 amd64 > [installed,upgradable to: 245.4-4ubuntu3.23] > libnss3/now 2:3.49.1-1ubuntu1.8 amd64 [installed,upgradable to: > 2:3.98-0ubuntu0.20.04.2] 325,328c325,328 < libpam-modules-bin/now 1.3.1-5ubuntu4.6 amd64 [installed,upgradable to: 1.3.1-5ubuntu4.7] < libpam-modules/now 1.3.1-5ubuntu4.6 amd64 [installed,upgradable to: 1.3.1-5ubuntu4.7] < libpam-runtime/now 1.3.1-5ubuntu4.6 all [installed,upgradable to: 1.3.1-5ubuntu4.7] < libpam-systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] --- > libpam-modules-bin/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 amd64 > [installed,automatic] > libpam-modules/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 amd64 > [installed,automatic] > libpam-runtime/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 all > [installed,automatic] > libpam-systemd/focal-security,now 245.4-4ubuntu3.20 amd64 > [installed,upgradable to: 245.4-4ubuntu3.23] 347c347 < libprocps8/now 2:3.3.16-1ubuntu2.3 amd64 [installed,upgradable to: 2:3.3.16-1ubuntu2.4] --- > libprocps8/focal-updates,focal-security,now 2:3.3.16-1ubuntu2.4 amd64 > [installed,automatic] 375c375 < libsmartcols1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > libsmartcols1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] 386c386 < libsystemd0/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] --- > libsystemd0/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable > to: 245.4-4ubuntu3.23] 394c394 < libtinfo6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: 6.2-0ubuntu2.1] --- > libtinfo6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 > [installed,automatic] 407c407 < libuuid1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > libuuid1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] 434c434 < linux-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49] --- > linux-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: > 5.15.0.1064.73~20.04.1] 438,439c438,439 < linux-cloud-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49] < linux-cloud-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all [installed,upgradable to: 5.4.0-174.193] --- > linux-cloud-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 > [installed,upgradable to: 5.15.0.1064.73~20.04.1] > linux-cloud-tools-common/focal-updates,focal-security,now 5.4.0-182.202 all > [installed] 441c441 < linux-headers-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49] --- > linux-headers-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable > to: 5.15.0.1064.73~20.04.1] 443,444c443,444 < linux-image-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49] < linux-libc-dev/focal-updates,focal-security,now 5.4.0-174.193 amd64 [installed,automatic] --- > linux-image-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: > 5.15.0.1064.73~20.04.1] > linux-libc-dev/focal-updates,focal-security,now 5.4.0-182.202 amd64 > [installed,automatic] 447,449c447,449 < linux-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49] < linux-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all [installed,upgradable to: 5.4.0-174.193] < locales/now 2.31-0ubuntu9.9 all [installed,upgradable to: 2.31-0ubuntu9.14] --- > linux-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: > 5.15.0.1064.73~20.04.1] > linux-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all > [installed,upgradable to: 5.4.0-182.202] > locales/now 2.31-0ubuntu9.9 all [installed,upgradable to: 2.31-0ubuntu9.15] 473c473 < mount/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > mount/focal-updates,focal-security,now 2.34-0.1ubuntu9.6 amd64 > [installed,automatic] 499c499 < passwd/now 1:4.8.1-1ubuntu5.20.04.4 amd64 [installed,upgradable to: 1:4.8.1-1ubuntu5.20.04.5] --- > passwd/focal-updates,focal-security,now 1:4.8.1-1ubuntu5.20.04.5 amd64 > [installed,automatic] 555c555 < python3-jinja2/focal,now 2.10.1-2 all [installed,upgradable to: 2.10.1-2ubuntu0.2] --- > python3-jinja2/focal-updates,focal-security,now 2.10.1-2ubuntu0.2 all > [installed] 592,594c592,594 < python3-twisted-bin/now 18.9.0-11ubuntu0.20.04.2 amd64 [installed,upgradable to: 18.9.0-11ubuntu0.20.04.3] < python3-twisted/now 18.9.0-11ubuntu0.20.04.2 all [installed,upgradable to: 18.9.0-11ubuntu0.20.04.3] < python3-update-manager/now 1:20.04.10.11 all [installed,upgradable to: 1:20.04.10.20] --- > python3-twisted-bin/focal-updates,focal-security,now 18.9.0-11ubuntu0.20.04.3 > amd64 [installed,automatic] > python3-twisted/focal-updates,focal-security,now 18.9.0-11ubuntu0.20.04.3 all > [installed,automatic] > python3-update-manager/now 1:20.04.10.11 all [installed,upgradable to: > 1:20.04.10.21] 618c618 < snapd/now 2.58+20.04 amd64 [installed,upgradable to: 2.58+20.04.1] --- > snapd/now 2.58+20.04 amd64 [installed,upgradable to: 2.62+20.04] 625,627c625,627 < sudo/now 1.8.31-1ubuntu1.4 amd64 [installed,upgradable to: 1.8.31-1ubuntu1.5] < systemd-sysv/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] < systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] --- > sudo/focal-updates,focal-security,now 1.8.31-1ubuntu1.5 amd64 > [installed,automatic] > systemd-sysv/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable > to: 245.4-4ubuntu3.23] > systemd/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: > 245.4-4ubuntu3.23] 637c637 < ubuntu-advantage-tools/now 27.13.5~20.04.1 amd64 [installed,upgradable to: 31.2~20.04] --- > ubuntu-advantage-tools/now 27.13.5~20.04.1 amd64 [installed,upgradable to: > 31.2.3~20.04] 648,649c648,649 < update-manager-core/now 1:20.04.10.11 all [installed,upgradable to: 1:20.04.10.20] < update-notifier-common/now 3.192.30.16 all [installed,upgradable to: 3.192.30.17] --- > update-manager-core/now 1:20.04.10.11 all [installed,upgradable to: > 1:20.04.10.21] > update-notifier-common/now 3.192.30.16 all [installed,upgradable to: > 3.192.30.19] 654,655c654,655 < util-linux/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] < uuid-runtime/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > util-linux/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] > uuid-runtime/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] _______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
