> IMHO the RPC rate limiting should be considered a best practice, and I > wouldn't think that it's a "dirty" configuration. You need Slurm 23.02 or > later for this. Some details are discussed in this Wiki page: Dirty in a way that levels are so low that they break some other service in order to determine which service is making that calls. You know, breaking things isn't the best practice ;) I totally agree that RPC rate limiting overall is a good practice, and perhaps it should be enabled by default in SLURM.
Regards Patryk. On 25/05/07 10:13AM, Ole Holm Nielsen via slurm-users wrote: > On 5/7/25 09:57, Patryk Bełzak via slurm-users wrote: > > Hi, > > why you think it's an authentication requests? As far as I understand > > multiple UIDs are asking for job and partition info. It's unlikely that all > > of them perform that kind of requests the same way and in the same time, so > > I think you should look for some external program that may do that - i.e. > > some monitoring tool? Or reporting tool? I'm not sure if API calls are also > > registered as RPC in controller logs. > > > > Dirty (but maybe effective) way of discovering what makes all of that calls > > is to set the RPC rate limit to some low value and see what stopped working > > ;) https://slurm.schedmd.com/slurm.conf.html#OPT_rl_enable > > IMHO the RPC rate limiting should be considered a best practice, and I > wouldn't think that it's a "dirty" configuration. You need Slurm 23.02 or > later for this. Some details are discussed in this Wiki page: > > https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_configuration/#rpc-rate-limiting > > IHTH, > Ole > > > > On 25/05/06 02:38PM, Guillette, Jeremy via slurm-users wrote: > > [-- Type: text/plain; charset=Windows-1252, Encoding: quoted-printable, > > Size: 9.3K --] > > > Hello, > > > I’m trying to figure out why we’ve been seeing an increase in network > > > traffic in our AWS-based cluster, which uses Amazon’s parallel cluster > > > tool. After an incident a couple weeks ago, I turned on debug2 logging on > > > the slurmd processes, and I’m seeing huge numbers of `REQUEST_GETPW` and > > > `REQUEST_GETGR` requests going to the slurmd processes. I briefly turned > > > on debug2 logging for `slurmctld` as well, and I’m seeing lots of RPC > > > calls, but not as many as the `REQUEST_GETPW` requests that I’ve seen on > > > compute node slurmd processes. > > > Here’s a sample from the slurmctld log: > > > ``` > > > [2025-04-28T15:11:05.436] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=20 > > > [2025-04-28T15:11:05.450] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=2971 > > > [2025-04-28T15:11:05.451] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=2971 > > > [2025-04-28T15:11:05.451] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=16 > > > [2025-04-28T15:11:05.461] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=2788 > > > [2025-04-28T15:11:05.461] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=2788 > > > [2025-04-28T15:11:05.461] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=12 > > > [2025-04-28T15:11:05.517] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=2916 > > > [2025-04-28T15:11:05.518] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=2916 > > > [2025-04-28T15:11:05.518] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=14 > > > [2025-04-28T15:11:05.628] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=3405 > > > [2025-04-28T15:11:05.629] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=3405 > > > [2025-04-28T15:11:05.629] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=14 > > > [2025-04-28T15:11:05.740] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=2189 > > > [2025-04-28T15:11:05.740] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=2189 > > > [2025-04-28T15:11:05.740] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=15 > > > [2025-04-28T15:11:05.845] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=2209 > > > [2025-04-28T15:11:05.846] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=2209 > > > [2025-04-28T15:11:05.846] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=4106 > > > [2025-04-28T15:11:05.846] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=14 > > > [2025-04-28T15:11:05.847] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=4106 > > > [2025-04-28T15:11:05.847] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=11 > > > [2025-04-28T15:11:05.938] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=3400 > > > [2025-04-28T15:11:05.938] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=3400 > > > [2025-04-28T15:11:05.938] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=18 > > > [2025-04-28T15:11:06.903] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=3449 > > > [2025-04-28T15:11:06.904] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=3449 > > > [2025-04-28T15:11:06.904] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=15 > > > [2025-04-28T15:11:07.175] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=3722 > > > [2025-04-28T15:11:07.176] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=3722 > > > [2025-04-28T15:11:07.177] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=254 > > > [2025-04-28T15:11:07.205] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=4040 > > > [2025-04-28T15:11:07.206] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=4040 > > > [2025-04-28T15:11:07.206] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=17 > > > [2025-04-28T15:11:07.237] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=2990 > > > [2025-04-28T15:11:07.238] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=2990 > > > [2025-04-28T15:11:07.239] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=15 > > > [2025-04-28T15:11:07.284] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=2920 > > > [2025-04-28T15:11:07.285] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=2920 > > > [2025-04-28T15:11:07.285] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=15 > > > [2025-04-28T15:11:07.370] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=3236 > > > [2025-04-28T15:11:07.371] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=3236 > > > [2025-04-28T15:11:07.371] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=17 > > > [2025-04-28T15:11:08.463] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=2848 > > > [2025-04-28T15:11:08.464] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=2848 > > > [2025-04-28T15:11:08.464] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=14 > > > [2025-04-28T15:11:08.691] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=2627 > > > [2025-04-28T15:11:08.692] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=2627 > > > [2025-04-28T15:11:08.692] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=18 > > > [2025-04-28T15:11:08.873] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=3729 > > > [2025-04-28T15:11:08.874] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=3729 > > > [2025-04-28T15:11:08.875] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=196 > > > [2025-04-28T15:11:08.881] debug2: Processing RPC: REQUEST_JOB_INFO_SINGLE > > > from UID=3461 > > > [2025-04-28T15:11:08.882] debug2: Processing RPC: REQUEST_PARTITION_INFO > > > from UID=3461 > > > [2025-04-28T15:11:08.882] debug2: _slurm_rpc_dump_partitions, size=1253 > > > usec=10 > > > ``` > > > And from slurmd: > > > ``` > > > [2025-04-27T19:45:01.353] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:02.475] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:02.491] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:02.496] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:02.496] [59253.batch] debug: Handling REQUEST_GETGR > > > [2025-04-27T19:45:02.497] [59253.batch] debug: Handling REQUEST_GETGR > > > [2025-04-27T19:45:02.497] [59253.batch] debug: Handling REQUEST_GETGR > > > [2025-04-27T19:45:02.501] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:02.504] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:02.507] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:02.513] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:02.518] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:02.518] [59253.batch] debug: Handling REQUEST_GETGR > > > [2025-04-27T19:45:02.606] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:02.607] [59253.batch] debug: Handling REQUEST_GETGR > > > [2025-04-27T19:45:04.988] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:04.992] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:04.995] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:04.999] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.011] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.016] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.033] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.045] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.048] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.057] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.073] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.077] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.110] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.143] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.143] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.144] [59253.batch] debug: Handling REQUEST_GETGR > > > [2025-04-27T19:45:05.152] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.152] [59253.batch] debug: Handling REQUEST_GETGR > > > [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.159] [59253.batch] debug: Handling REQUEST_GETGR > > > [2025-04-27T19:45:05.167] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.167] [59253.batch] debug: Handling REQUEST_GETGR > > > [2025-04-27T19:45:05.170] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.172] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.174] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.203] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.204] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.207] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.316] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.318] [59253.batch] debug: Handling REQUEST_GETPW > > > [2025-04-27T19:45:05.321] [59253.batch] debug: Handling REQUEST_GETPW > > > ``` > > > This level of debugging makes the logs pretty huge, but if seeing a whole > > > log file is helpful, I can make something available. > > > Any ideas on next steps for figuring out what’s going on? It seems like > > > something is asking for authentication a whole lot, but it’s not clear to > > > me what or why. We do use munge for Slurm authentication, and SSSD to > > > work with LDAP for user authentication. > > > -Jeremy Guillette > > > — > > > > > > Jeremy Guillette > > > > > > Software Engineer, FAS Academic Technology | Academic Technology > > > Harvard University Information Technology > > > P: (617) 998-1826 | W: huit.harvard.edu > > > (he/him/his) > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
smime.p7s
Description: S/MIME cryptographic signature
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
