On 7/18/22 3:45 pm, gphipps wrote:
Everyone so often one of our users accidentally writes a “fork-bomb”
that submits thousands of sbatch and srun requests per second. It is a
giant DDOS attack on our scheduler. Is there a way of rate limiting
these requests before they reach the daemon?
Yes there is, you can use the Slurm cli_filter to do this.
https://slurm.schedmd.com/cli_filter_plugins.html
If you use the lua plugin you can write what you need in that; though of
course it would need careful thought as you would need somewhere to
store state on the node (writeable by users), a way of counting the
frequency of the RPCs and introducing increasing delays (up to a point)
if it's out of control and then decaying that delay time down when the
RPCs from that user cease/decrease.
All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA
