salut guillaume,
nothing else is different between the v1 and v2 setup? (/tmp is tmpfs on
v2 setup perhaps?)
stijn
On 5/22/25 11:10, Guillaume COCHARD via slurm-users wrote:
Hello,
We've noticed a recent change in how MaxRSS is reported on our cluster.
Specifically, the MaxRSS value for many jobs now often matches the allocated
memory, which was not the case previously. It appears this change is due to how
Slurm accounts for memory when copying large files, likely as a result of
moving from cgroup v1 to cgroup v2.
Here’s a simple example:
copy_file.sh
#!/bin/bash
cp /distributed/filesystem/file5G /tmp
cp /tmp/file5G ~
Two jobs with different memory allocations:
Job 1
sbatch -c 1 --mem=1G copy_file.sh
seff <jobid>
Memory Utilized: 1021.87 MB
Memory Efficiency: 99.79% of 1.00 GB
Job 2
sbatch -c 1 --mem=10G copy_file.sh
seff <jobid>
Memory Utilized: 4.02 GB
Memory Efficiency: 40.21% of 10.00 GB
With cgroup v1, this script typically showed minimal memory usage. Now, under
cgroup v2, memory usage appears inflated and depends on the allocated memory,
which seems wrong.
I believe this behavior aligns with similar issues raised by the Kubernetes
community [1], and is consistent with how memory.current behaves in cgroup v2
[3].
According to Slurm’s documentation about cgroup v2, "this plugin provides cgroup's
memory.current value from the memory interface, which is not equal to the RSS value
provided by procfs. Nevertheless it is the same value that the kernel uses in its OOM
killer logic." [2]
While technically correct, this seems to mark a significant change in what MaxRSS and
"Memory Efficiency" actually measure and renders those metrics almost useless.
Our Configuration:
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup,task/affinity
Question:
Is there a way to restore more realistic MaxRSS values — specifically, ones
that exclude file-backed page cache — while still using cgroup v2?
Thanks,
Guillaume
References:
[1] https://github.com/kubernetes/kubernetes/issues/118916
[2] https://slurm.schedmd.com/cgroup_v2.html#limitations
[3] https://facebookmicrosites.github.io/cgroup2/docs/memory-controller.html
--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
