Sorry Batanun, this thread got lost in my inbox. Would you be able to upgrade to 7.5 and see if you get the same results? I'm pretty sure it's a jemalloc issue, but upgrading should make it clear. You are on Ubuntu, right? Which version? -- Guillaume Quintard
On Mon, May 20, 2024 at 1:50 AM Batanun B <[email protected]> wrote: > > Sorry, I should have been clearer, I meant: where are the varnish > packages coming from? Are they from the official repositories, from > https://packagecloud.io/varnishcache/ or built from source maybe? > > Ah, I see. They come from varnishcache packagecloud. More specifically, we > use: > > > https://packagecloud.io/install/repositories/varnishcache/varnish60lts/script.deb.sh > > > > you should really invest some time in something like prometheus, it > would probably have made the issue obvious > > Yes, in hindsight we definitely should have done that. I will discuss this > with my coworkers going forward. > > > > Is there any chance you can run the old version on the server to explore > the differences? > > Possibly, for a limited time. If so, what types of tests would I do? And > how long time would I need to run the old version? > > Note that with our setup, we wouldn't be able to run two different images > at the same time, in the same environment, with both recieving traffic. So > all traffic would be routed to this version (multiple servers, but all > running the same image). > > An alternative approach that I'm considering, is to switch to the old > image, but manually update the VCL to the new version. If the problem > remains, then the issue is almost certainly with the VLC. But if the > problem disapears, then it's more likely something else. > > > > what's the output of: varnishstat -1 -f '*g_bytes' > > SMA.default.g_bytes 10951750929 . Bytes outstanding > SMA.large.g_bytes 8587329728 . Bytes outstanding > SMA.Transient.g_bytes 3177920 . Bytes outstanding > > So, the default storage usage has gone up with 2GB since my first message > here, while the others have remained the same. Meanwhile, the total memory > usage of Varnish has gone up to 26 GB, an increase of 3 GB. So now the > overhead has gone up with 1GB to a total of 6 GB. > > Going forward, it will be interesting to see how the memory consumption > changes after the default storage has reached its max (2 GB from where it > is now). If we're lucky, it will stabilize, and then I'm not sure if it's > worth it to troubleshoot any further. Otherwise, the free memory would get > a bit too close to zero for our comfort, with no indication of stopping. > > Does Varnish keep track of total available OS memory, and start releasing > memory by throwing out objects from the cache? Or will it continue to eat > memory until something fails? > > > > have you tweaked any workspaces/thread parameters? > > Nope. As I said, we haven't changed any OS or Varnish configuration. > _______________________________________________ > varnish-misc mailing list > [email protected] > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >
_______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
