On Fri, 06 Sep 2024 13:23:32 +0000 Alan Daniels wrote:>
I am running a small number of Squid servers on the boundary of my network. I
am seeing what looks like a memory leak in the application when it receives a
relatively large number of connections.
> A connection rate of 20-30 requests per second for 5 minutes will
cause the memory to increase by about 100MB and the memory never gets
released. We are running this in Kubernetes pods, the memory keeps
increasing until the pod exhausts the memory limit it has and gets killed.
> I believe the leak is occurring in the SSL handling portion of the
servers code, the reason for this is when I run tests scripts to trigger
this, I wasn't able to replicate this when I used the scheme http, I was
only able to replicate this when I used https.
Please be aware of several relevant details:
* The squid.conf shown is for a regular HTTP-only proxy. Any TLS/SSL
traffic can only go through such a proxy as CONNECT tunnels.
* CONNECT tunnels consume 2 TCP sockets/connections each with a
minimum 64KB buffer. Due to their opaque nature they are kept open until
closed by explicit disconnection by either server or client.
* The provided list of requests and times taken, show a large number
of test requests taking upwards of 30min to complete.
This means that we can estimate the expected memory consumption from the
known RPS rate.
25 RPS * 5min * 2 connection * 64KB buffer
=> 25 * 300 * 128
=> 7500 * 128 = 960000
=> 937.5 MB
.. so under peak load you can expect around 1GB of memory usage.
This is confirmed by the provided top "VIRT" reading after testing.
Which shows 926 MB memory used at peak. Over half of it has been
released back to the OS already, some will have been held to assist with
future or current load.
I do not see any clear evidence here of a memory leak. Not to say there
isn't one, just that the evidence given does not show one clearly.
Better evidence would be:
1) a time-series graph of memory usage vs connected clients.
2) before and after copies of the "mgr:mem" report from Squid. With
matching 'top' command output. Maybe also matching "mgr:filedescriptors"
listings.
HTH
Amos
