Dear All, I am facing an issue in SLURM(20.11.8), in which sreport cluster utilization is 100%, and when I run sreport cluster userutilizationbyaccount, Some user utilisation is greater than 100%, three users including root showing utilisation over 250%, making overall utilisation 500% (though user has not submitted any job in past one week)
It was showing some runaway jobs, but we cleared it, then again, it was showing same runaway jobs, and we cleared it again. (both manually/through command) Before that, we had encountered an issue in the past in which, in our cluster with primary and backup slurm controller, we kept a common mount point for the "StateSaveLocation" /var/share/slurm/ctld. Then we observed a strange behaviour that " If the mount point is present and the service is restarted on the primary controller then it replaces all the statesavelocation files." This resulted in cancellation of all the jobs (running, pending state), reservations and assigns the JobID from 1 for newly submitted jobs. If the SateSaveLocation is kept on local file system instead of shared mount point then everything works fine even after restarting the slurmctld service. After that issue, utilisation is higher than expected, though it has not impacted any real job utilisation. Also, we have removed those user's account in SLURM, yet it is still showing their utilisation Please help in resolving this issue. Thanks and Regards, Shubham Mehta HPC Technology CDAC Pune ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
