Public bug reported:
Package: slurm-wlm / slurmd 25.11.2-1ubuntu2 (universe, resolute release pocket)
Ubuntu 26.04.1 LTS, glibc 2.43-2ubuntu2.4, kernel 7.0.0-34-generic, systemd
259, cgroup v2
PROBLEM
sacct and sstat never report memory usage: MaxRSS, AveRSS and TRESUsageInMax
are always empty,
for every job and every step (including long completed jobs). TotalCPU is
populated (it comes
from rusage, not from the gather plugin). sstat on a running step is empty too,
and AveCPU shows
a garbage value (213503982334-14:25:51), so slurmstepd is not delivering any
gathered data.
It happens with both JobAcctGatherType=jobacct_gather/cgroup (our production
setting) and
jobacct_gather/linux. It is not a cgroup problem: the step cgroup has
memory.current and
memory.peak, and task/cgroup enforces the limits correctly.
Relevant slurm.conf on the production nodes:
JobAcctGatherType=jobacct_gather/cgroup,
JobAcctGatherFrequency=30, ProctrackType=proctrack/cgroup,
TaskPlugin=task/affinity,task/cgroup,
AccountingStorageType=accounting_storage/slurmdbd.
EVIDENCE
With SlurmdDebug=debug and DebugFlags=JobAccountGather, slurmd logs a single
poll attempt and
then nothing for the whole step, even with --acctg-freq=task=10:
jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded
jobacct_gather/cgroup: jag_common_poll_data: JAG: cont_id hasn't been set yet
not running poll
... (no further poll until the step ends) ...
jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded
CAUSE
Upstream SchedMD fixed this in 25.11.7 (also in 26.05.0). The changelog says:
"Fix JobAccountGather failing on glibc 2.43+ due to a sscanf() %Nc behavior
change."
Upstream commit on branch slurm-25.11: 0d323a1ed8 "jobacct_gather - Remove
unused 'cmd' string"
(cherry-picked from master 6e2c994986, upstream issue 50879). Its message
explains that the
sscanf(sbuf, "%d (%39c", ...) in _get_process_data_line()
(src/plugins/jobacct_gather/common/common_jag.c) was broken by glibc 2.43
(glibc commit
2b16c76609): %Nc now succeeds only if N characters are available, so process
names shorter than
39 characters made it fail, leaving prec->pid unset. That produces exactly the
log line above.
The file is shared by jobacct_gather/linux and jobacct_gather/cgroup, so both
plugins are
affected (I checked this by reading the code; only the linux plugin was tested
end to end).
Resolute ships glibc 2.43 together with Slurm 25.11.2, so every Resolute
system is affected.
CONFIRMATION
Reproduced in two containers from the same ubuntu:26.04 image (glibc
2.43-2ubuntu2.4), single
node Slurm with jobacct_gather/linux, JobAcctGatherFrequency=5,
accounting_storage/none, one job
allocating 400 MB, and "sstat -j <id>.batch" while the job runs:
- Slurm 25.11.2 (slurm-wlm from the Ubuntu archive): MaxRSS and AveRSS empty,
AveCPU=213503982334-14:25:51; slurmd log says "JAG: no pids in this
container".
- Slurm 25.11.7 (upstream tarball built on the same image): MaxRSS=420708K,
AveRSS=420708K;
the log shows the per-task "mem size/max" values.
The attached archive slurm-repro-docker.tar.gz contains the reproduction files
(Dockerfile.pkg,
Dockerfile.src, slurm.conf, cgroup.conf, run-test.sh, compose.yml) and the
upstream patch
slurm-25.11-jobacct-glibc-2.43-sscanf.patch. Usage after extracting: "docker
compose run --rm pkg"
and "docker compose run --rm src".
WHAT IS NOT AFFECTED
CPU and memory limits still work: a 2-CPU job running 30 busy loops used 2.01
cores, and a
512 MB job allocating 3 GB ended OUT_OF_MEMORY (exit 137, oom_kill in
slurmd.log).
HOW TO REPRODUCE
sbatch --mem=1G -c1 --wrap 'python3 -c "x=bytearray(400*2**20);
x[::4096]=b\"a\"*len(x[::4096]); import time; time.sleep(70)"'
sacct -j <jobid> -o JobID,MaxRSS,AveRSS,TRESUsageInMax # all empty
sstat -j <jobid>.batch -o JobID,MaxRSS,AveCPU # while running:
empty / garbage
Expected: MaxRSS around 400M.
REQUEST
Please update slurm-wlm in resolute (SRU) to a release containing the upstream
fix (>= 25.11.7),
or backport the one-file upstream patch to 25.11.2 (it only drops an unused
%39c field from one
sscanf, so the regression risk is low). The development series already has
26.05.4-1ubuntu1,
which includes the fix.
** Affects: slurm-wlm (Ubuntu)
Importance: Undecided
Status: New
** Tags: glibc-2.43 resolute
** Attachment added: "code to reproduce the issue + patch"
https://bugs.launchpad.net/bugs/2169842/+attachment/6006815/+files/slurm-repro-docker.tar.gz
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2169842
Title:
slurmd 25.11.2 on Resolute: job accounting gather never polls (empty
MaxRSS/sstat); upstream 25.11.7 fixes JobAccountGather on glibc 2.43
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/slurm-wlm/+bug/2169842/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
