Re: [slurm-users] Rate Limiting of RPC calls

2021-02-19 Thread Kevin Buckley
On 2021/02/17 14:02, Kota Tsuyuzaki wrote: .. so that I'm realizing that sacct may exhaust the resources more rapidly than squeue on mysql point of view because, as I understand correctly, squeue doesn't affects such a mysql db query performance. Any thoughts? Best, Kota Pulling together so

Re: [slurm-users] Rate Limiting of RPC calls

2021-02-16 Thread Kota Tsuyuzaki
rm-users On Behalf Of Kota > Tsuyuzaki > Sent: Friday, February 12, 2021 6:14 PM > To: 'Slurm User Community List' ; 'Christopher > Samuel' > Subject: Re: [slurm-users] Rate Limiting of RPC calls > > Thanks Guys! >

Re: [slurm-users] Rate Limiting of RPC calls

2021-02-12 Thread Kota Tsuyuzaki
-2837 - > -Original Message- > From: slurm-users On Behalf Of Kevin > Buckley > Sent: Wednesday, February 10, 2021 11:32 AM > To: Slurm User Community List ; Christopher > Samuel > Subject: Re: [slurm-users] Rate Limiting of RPC c

Re: [slurm-users] Rate Limiting of RPC calls

2021-02-09 Thread Kevin Buckley
On 2021/02/10 09:33, Christopher Samuel wrote: Also getting users to use `sacct` rather than `squeue` to check what state a job is in can help a lot too, it reduces the load on slurmctld. That raises an interesting take on the two utilities, Chris, in that 1) It should be possible to write a

Re: [slurm-users] Rate Limiting of RPC calls

2021-02-09 Thread Christopher Samuel
On 2/9/21 5:08 pm, Paul Edmon wrote: 1. Being on the latest release: A lot of work has gone into improving RPC throughput, if you aren't running the latest 20.11 release I highly recommend upgrading.  20.02 also was pretty good at this. We've not gone to 20.11 on production systems yet, but I

Re: [slurm-users] Rate Limiting of RPC calls

2021-02-09 Thread Paul Edmon
We've hit this before several times. The tricks we've used to deal with this are: 1. Being on the latest release: A lot of work has gone into improving RPC throughput, if you aren't running the latest 20.11 release I highly recommend upgrading.  20.02 also was pretty good at this. 2. max_rpc

[slurm-users] Rate Limiting of RPC calls

2021-02-09 Thread Kota Tsuyuzaki
Hello guys, In our cluster, sometimes new incoming member accidentally creates too many slurm RPC calls (sbatch, sacct, etc), then slurmctld, slurmdbd, and mysql may be overloaded. To prevent such a situation, I'm looking for something like RPC Rate Limit for users. Does Slurm supports such a Ra