Hi Martin, to my very best knowledge MaxRSS does report aggregated memory consumption of all tasks but including all the shared libraries that the individual processes uses, even though a shared library is only loaded into memory once regardless of how many processes use it.
So shared libraries do count multiple times (once for every individual process) to MaxRSS when summed up. This can even result in a higher MaxRSS value reported by sacct than the total amount of memory that is physically available. For exactly that reason we have decided to use `JobAcctGatherParams=UsePss´ in slurm.conf as we think proportional set size (PSS) is more useful than RSS because when the PSS for all processes are summed together, that seems to be a better representation for the "true" total memory consumption of the job. Also see: https://en.wikipedia.org/wiki/Proportional_set_size https://bugs.schedmd.com/show_bug.cgi?id=9010 I would also be interested what others think. Best regards Jürgen * Ohlerich, Martin <martin.ohler...@lrz.de> [221102 11:53]: > Dear "Commiserates". > > I wonder a bit about the meaning of MaxRSS. The documentation says: > "Maximum resident set size of all tasks in job." > > To what refers here "maximum"? The maximum over job period, I > understand hopefully correctly. But it does not seem to be the size > of all tasks (summed up, so-to-speak), but the maximum size of RSS > of that task with the largest RSS of all tasks during the job's > period. Right? > > In any case, I observed something like this: > > login08:~> sacct -j 2408392 -o 'maxrss,maxrssnode%20' > MaxRSS MaxRSSNode > ---------- -------------------- > 102124920K i02r09c03s02 > > ... so 102 GB if I counted the decimal positions correctly. On the other > hand, for specifically this node, I actually only have > > i02r09c03s02:~> cat /proc/meminfo > MemTotal: 98436736 kB > > i.e. 98 GB RAM ... > > Does anybody know whether there is a reasonable explanation how this can be? > Specifically is the situation even worse, if MaxRSS is the maximum RSS of > only one task (rank) on that node. What about the other tasks (which > certainly also consume memory). And also the OS is quite large on these > disk-less compute nodes. > > Would be nice if you could share any ideas about my finding. Thank you! > Kind regards, > Martin >
smime.p7s
Description: S/MIME cryptographic signature
