Thanks, I will look into that.
Cheers
Geoff
Message: 1
Date: Tue, 19 Jul 2022 21:38:56 -0700
From: Christopher Samuel <ch...@csamuel.org>
To: slurm-users@lists.schedmd.com
Subject: Re: [slurm-users] Rate-limiting sbatch and srun
Message-ID: <6f7850ba-8b66-47a0-7312-8c222f1ff...@csamuel.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
On 7/18/22 3:45 pm, gphipps wrote:
> Everyone so often one of our users accidentally writes a ?fork-bomb?
> that submits thousands of sbatch and srun requests per second. It is a
> giant DDOS attack on our scheduler. Is there a way of rate limiting
> these requests before they reach the daemon?
Yes there is, you can use the Slurm cli_filter to do this.
https://slurm.schedmd.com/cli_filter_plugins.html
If you use the lua plugin you can write what you need in that; though of
course it would need careful thought as you would need somewhere to
store state on the node (writeable by users), a way of counting the
frequency of the RPCs and introducing increasing delays (up to a point)
if it's out of control and then decaying that delay time down when the
RPCs from that user cease/decrease.
All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA
End of slurm-users Digest, Vol 57, Issue 26
*******************************************
