Nico, yep that’s a very annoying bug as we do the same here with job
efficiency. It was patched in 18.08.05. However the db still needs to be
cleaned up. We are working on a script to fix this. When we are done, we'll
offer it up to the list.
Best,
Chris
—
Christopher Coffey
High-Performance Computing
Northern Arizona University
928-523-1167
On 2/1/19, 8:47 AM, "slurm-users on behalf of nico.faer...@id.unibe.ch"
<slurm-users-boun...@lists.schedmd.com on behalf of nico.faer...@id.unibe.ch>
wrote:
Hi
While doing some statistics on efficient CPU usage, I realized that sacct
is reporting inexplicable (at least for me) high values for TotalCPU, UserCPU
and SystemCPU. Here is a simple example (each job step is a infinite while
loop):
sacct -j 64338003
--format=jobid,elapsed,ncpus,cputime,totalcpu,usercpu,systemcpu,nodelist
JobID Elapsed NCPUS CPUTime TotalCPU UserCPU
SystemCPU NodeList
------------ ---------- ---------- ---------- ---------- ----------
---------- ---------------
64338003 00:02:29 4 00:09:56 13:19:41 13:19:36
00:05.054 anode033
64338003.ba+ 00:02:31 4 00:10:04 00:09.017 00:04.003
00:05.014 anode033
64338003.ex+ 00:02:30 4 00:10:00 00:00.001 00:00:00
00:00.001 anode033
64338003.0 00:02:32 1 00:02:32 03:19:52
03:19:52 00:00.013 anode033
64338003.1 00:02:32 1 00:02:32 03:19:54
03:19:54 00:00.008 anode033
64338003.2 00:02:32 1 00:02:32 03:19:53 03:19:53
00:00.010 anode033
64338003.3 00:02:32 1 00:02:32 03:19:52
03:19:52 00:00.007 anode033
I would expect CPUTime to be the upper limit for TotalCPU.
Looking at cpuacct.stat for job step3:
cat /cgroup/cpuacct/slurm/uid_6994/job_64338003/step_3/cpuacct.stat
user 14902 (~149 = 00:02:29)
system 0
This value corresponds to the expected CPU usage of a single job step.
We are running Slurm 18.08.4 with
JobAcctGatherType=jobacct_gather/cgroup
Does anyone have an explanation for those high values reported by sacct?
Best,
Nico
Universitaet BernAbt. Informatikdienste
Nico Färber
High Performance Computing
Gesellschaftsstrasse 6
CH-3012 Bern
Raum 104
Tel. +41 (0)31 631 51 89
