Re: [slurm-users] Rate-limiting sbatch and srun

2022-07-19 Thread Christopher Samuel
On 7/18/22 3:45 pm, gphipps wrote: Everyone so often one of our users accidentally writes a “fork-bomb” that submits thousands of sbatch and srun requests per second. It is a giant DDOS attack on our scheduler. Is there a way of rate limiting these requests before they reach the daemon? Yes

Re: [slurm-users] slurmctld up and running but not really working

2022-07-19 Thread Ole Holm Nielsen
Hi Julien, Apparently your slurmdbd is quite happy, but it seems that your slurmctld StateSaveLocation has been corrupted: [2022-07-19T15:17:58.356] error: Node state file /var/lib/slurm-llnl/slurmctld/node_state too small [2022-07-19T15:17:58.356] error: NOTE: Trying backup state save file.

[slurm-users] slurmctld up and running but not really working

2022-07-19 Thread Julien Rey
Hello, I am currently facing an issue with an old install of slurm (17.02.11). However, I cannot upgrade this version because I had troubles with database migration in the past (when upgrading to 17.11) and this install is set to be replaced in the next coming monthes. For the time being I ha

[slurm-users] slurmrestd does not show GrpTRESMins billing association

2022-07-19 Thread Oren Shani
Hi All, I am trying to get the equivalent of this line from *scontrol show association *via slurmrestd: GrpTRESMins=cpu=N(58),mem=N(178688),energy=N(0),node=N(29), *billing=7(58),* fs/disk=N(0),vmem=N(0),pages=N(0),gres/gpu=N(0),license/interactive=N(0) But while cpu and mem are shown in bot

Re: [slurm-users] Rate-limiting sbatch and srun

2022-07-19 Thread Ole Holm Nielsen
On 7/19/22 08:15, Ole Holm Nielsen wrote: On 7/19/22 00:45, gphipps wrote: Everyone so often one of our users accidentally writes a “fork-bomb” that submits thousands of sbatch and srun requests per second. It is a giant DDOS attack on our scheduler. Is there a way of rate limiting these reque