Looking into this more it looks like memory.max_usage_in_byte and memory.usage_in_bytes also count file cache. Which is very surprising and not at all useful. But total_rss in memory.stat shows a more correct number. Looking at that one for a real job gives me around 30 GB, which matches my other data and expectations. But my issue remains that sstat and sacct report only a few MB for memory stats.
On Mon, 9 Nov 2020 at 18:49, Patrik Andersson <patrik.s.anders...@gmail.com> wrote: > We are using cgroups to track resource usage of our jobs. The jobs are run > in docker with docker's --parent-cgroup flag pointing at the slurm job's > cgroup. This works great for limiting memory usage. > > Unfortunately the maximum memory usage, maxRSS, is not accurately reported > in sacct. While the cgroup's memory.max_usage_in_bytes does show accurate > numbers. > > Looking at the cgroup: > >> /sys/fs/cgroup/memory/slurm/uid_500/job_31626/memory.max_usage_in_bytes:1132154880 >> # 1GB >> /sys/fs/cgroup/memory/slurm/uid_500/job_31626/memory.use_hierarchy:1 >> /sys/fs/cgroup/memory/slurm/uid_500/job_31626/memory.stat:rss 0 >> /sys/fs/cgroup/memory/slurm/uid_500/job_31626/memory.stat:total_rss 524288 >> > > Looking at sacct: > >> $ sacct -j 31626 -o >> jobid,AveRSS,MaxRSS,AveVMSize,MaxVMSize,ReqMem,TotalCPU >> >> JobID AveRSS MaxRSS MaxVMSize >> 31626.batch 28600K 28600K 77900K > > > I expected that we would get some of the cgroup stats since we are using > cgroup plugins. > > lines from slurm.conf > >> JobAcctGatherFrequency=30 >> >> JobAcctGatherType=jobacct_gather/cgroup >> >> ProctrackType=proctrack/cgroup >> >> TaskPlugin=task/affinity,task/cgroup >> >> SelectTypeParameters=CR_Core_Memory,CR_CORE_DEFAULT_DIST_BLOCK >> > > cgroup.conf > >> CgroupAutomount=yes >> >> CgroupMountpoint=/sys/fs/cgroup >> >> >> >> ### Task/cgroup Plugin ### >> >> # Constrain allowed cores to the subset of allocated resources. >> >> # This functionality makes use of the cpuset subsystem >> >> ConstrainCores=yes >> >> ConstrainKmemSpace=yes >> >> ConstrainRAMSpace=yes >> >> ConstrainSwapSpace=yes >> >> ConstrainDevices=no >> >> MinKmemSpace=30 >> >> MinRAMSpace=30 >> >> # Set a default task affinity to bind each step task to a subset of the >> >> # allocated cores using sched_setaffinity >> >> # /!\ This feature requires the Portable Hardware Locality (hwloc) library >> >> TaskAffinity=no >> >
