Re: [slurm-users] limit the queued jobs

[Ole Holm Nielsen]

On 7/11/22 07:55, Purvesh Parmar wrote:

Hi,
Thank you for the response. However, MaxJobsAccrue only limits to 
assigning priorities to jobs i.e. Maximum number of pending jobs able to 
accrue age priority at any given time. But it does not stop/ allow  user 
from submitting further jobs.


I see.  Then you probably want to use the MaxSubmitJobs limit in stead.

See also my Wiki page:
https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#maxjobcount-limit

/Ole

On Mon, 11 Jul 2022 at 11:07, Ole Holm Nielsen > wrote:


Hi Purvesh,

On 7/11/22 03:37, Purvesh Parmar wrote:
 > I want to limit the queued jobs per user to 5 which means, system
should
 > not allow more than 5 jobs per user to remain in queue (not running
and
 > waiting for resources) and only 4 jobs to run at any given time. To
 > summarize, I want to implement a policy of 4 jobs per user in the
running
 > state and 5 jobs per user in the queue at any given time.
 >
 > I have achieved implementation of the limit of running jobs to 4 by
 > sacctmger  set MaxJobs=4. This works..But, could not find any
 > parameter which takes care of only queued jobs limit. How to do this?

Probably you want to use the MaxJobsAccrue limit, see the sacctmgr manual
page.

[slurm-users] how do slurmctld determine whether a compute node is not responding?

[taleintervenor]

Hi, all:

 

Recently we found some strange log in slurmctld.log about node not
responding, such as:

[2022-07-09T03:23:10.692] error: Nodes node[128-168,170-178] not responding

[2022-07-09T03:23:58.098] Node node171 now responding

[2022-07-09T03:23:58.099] Node node165 now responding

[2022-07-09T03:23:58.099] Node node163 now responding

[2022-07-09T03:23:58.099] Node node172 now responding

[2022-07-09T03:23:58.099] Node node170 now responding

[2022-07-09T03:23:58.099] Node node175 now responding

[2022-07-09T03:23:58.099] Node node164 now responding

[2022-07-09T03:23:58.099] Node node178 now responding

[2022-07-09T03:23:58.099] Node node177 now responding

Meanwhile, checking slurmd.log and nhc.log on those node all seem to be ok
at the reported timepoint.

So we guess it's slurmctld launch some detection towards those compute node
and didn't get response, thus lead to slurmctld thinking those node to be
not responding.

Then the question is what detect action do slurmctld launched? How did it
determine whether a node is responsive or non-responsive?

And is it possible to customize slurmctld's behavior on such detection, for
example wait timeout or retry count before determine the node to be not
responding?

Re: [slurm-users] how do slurmctld determine whether a compute node is not responding?

[Kamil Wilczek]

Hello,

I know that this is not quite the answer, but you could additionally
(and maybe you already did this :)) check if this is not a network
problem:

* Are the nodes available outside of Slurm during that time? SSH, ping?
* If you have a monitoring system (Prometheus, Icinga, etc.), are
  there any issues reported?

And lastly, did you try to set log level to "debug" for "slurmd"
and "slurmctld"?

Kind Regards
--

W dniu 11.07.2022 o 09:32, taleinterve...@sjtu.edu.cn pisze:

Hi, all:

Recently we found some strange log in slurmctld.log about node not 
responding, such as:


[2022-07-09T03:23:10.692] error: Nodes node[128-168,170-178] not responding

[2022-07-09T03:23:58.098] Node node171 now responding

[2022-07-09T03:23:58.099] Node node165 now responding

[2022-07-09T03:23:58.099] Node node163 now responding

[2022-07-09T03:23:58.099] Node node172 now responding

[2022-07-09T03:23:58.099] Node node170 now responding

[2022-07-09T03:23:58.099] Node node175 now responding

[2022-07-09T03:23:58.099] Node node164 now responding

[2022-07-09T03:23:58.099] Node node178 now responding

[2022-07-09T03:23:58.099] Node node177 now responding

Meanwhile, checking slurmd.log and nhc.log on those node all seem to be 
ok at the reported timepoint.


So we guess it’s slurmctld launch some detection towards those compute 
node and didn’t get response, thus lead to slurmctld thinking those node 
to be not responding.


Then the question is what detect action do slurmctld launched? How did 
it determine whether a node is responsive or non-responsive?


And is it possible to customize slurmctld’s behavior on such detection, 
for example wait timeout or retry count before determine the node to be 
not responding?




--
Kamil Wilczek  [https://keys.openpgp.org/]
[D415917E84B8DA5A60E853B6E676ED061316B69B]
Laboratorium Komputerowe
Wydział Matematyki, Informatyki i Mechaniki
Uniwersytet Warszawski

ul. Banacha 2
02-097 Warszawa

Tel.: 22 55 44 392
https://www.mimuw.edu.pl/
https://www.uw.edu.pl/


OpenPGP_signature
Description: OpenPGP digital signature

Re: [slurm-users] how do slurmctld determine whether a compute node is not responding?

[Ole Holm Nielsen]

On 7/11/22 09:32, taleinterve...@sjtu.edu.cn wrote:
Recently we found some strange log in slurmctld.log about node not 
responding, such as:


[2022-07-09T03:23:10.692] error: Nodes node[128-168,170-178] not responding

[2022-07-09T03:23:58.098] Node node171 now responding

[2022-07-09T03:23:58.099] Node node165 now responding

[2022-07-09T03:23:58.099] Node node163 now responding

[2022-07-09T03:23:58.099] Node node172 now responding

[2022-07-09T03:23:58.099] Node node170 now responding

[2022-07-09T03:23:58.099] Node node175 now responding

[2022-07-09T03:23:58.099] Node node164 now responding

[2022-07-09T03:23:58.099] Node node178 now responding

[2022-07-09T03:23:58.099] Node node177 now responding

Meanwhile, checking slurmd.log and nhc.log on those node all seem to be ok 
at the reported timepoint.


So we guess it’s slurmctld launch some detection towards those compute 
node and didn’t get response, thus lead to slurmctld thinking those node 
to be not responding.


Such node warnings could be caused by a broken network.  Or by your DNS 
servers not responding to DNS lookups so that "node177" is unknown, for 
example.


Then the question is what detect action do slurmctld launched? How did it 
determine whether a node is responsive or non-responsive?


And is it possible to customize slurmctld’s behavior on such detection, 
for example wait timeout or retry count before determine the node to be 
not responding?


See the slurm.conf parameters displayed by:

# scontrol show config | grep Timeout

We normally use this:

SlurmctldTimeout= 600 sec
SlurmdTimeout   = 300 sec

/Ole

[slurm-users] 答复: how do slurmctld determine whether a compute node is not responding?

[taleintervenor]

Hello, Kamil Wilczek:

Well I agree that the non-responding case may caused by network unstable, since 
our slurm cluster has 2 part nodes geographical distant distributed with only 
ethernet link them. Those reported nodes are all in one building while the 
slurmctld node in another building.
But we can do nothing about the network infrastructure, so we are more 
interested in adjusting slurm to make it tolerate such short-time non 
responding case.
Or is there possible to tell slurmctld do those detection through certain proxy 
node? For example we have slurmctld backup node in the same building with those 
reported compute nodes, if slurm can use this backup controller node to execute 
detection towards some part of compute nodes, the result may be more stable.

-邮件原件-
发件人: Kamil Wilczek <> 
发送时间: 2022年7月11日 15:53
收件人: Slurm User Community List ; 
taleinterve...@sjtu.edu.cn
主题: Re: [slurm-users] how do slurmctld determine whether a compute node is not 
responding?

Hello,

I know that this is not quite the answer, but you could additionally (and maybe 
you already did this :)) check if this is not a network
problem:

* Are the nodes available outside of Slurm during that time? SSH, ping?
* If you have a monitoring system (Prometheus, Icinga, etc.), are
   there any issues reported?

And lastly, did you try to set log level to "debug" for "slurmd"
and "slurmctld"?

Kind Regards
-- 

W dniu 11.07.2022 o 09:32, taleinterve...@sjtu.edu.cn pisze:
> Hi, all:
> 
> Recently we found some strange log in slurmctld.log about node not 
> responding, such as:
> 
> [2022-07-09T03:23:10.692] error: Nodes node[128-168,170-178] not 
> responding
> 
> [2022-07-09T03:23:58.098] Node node171 now responding
> 
> [2022-07-09T03:23:58.099] Node node165 now responding
> 
> [2022-07-09T03:23:58.099] Node node163 now responding
> 
> [2022-07-09T03:23:58.099] Node node172 now responding
> 
> [2022-07-09T03:23:58.099] Node node170 now responding
> 
> [2022-07-09T03:23:58.099] Node node175 now responding
> 
> [2022-07-09T03:23:58.099] Node node164 now responding
> 
> [2022-07-09T03:23:58.099] Node node178 now responding
> 
> [2022-07-09T03:23:58.099] Node node177 now responding
> 
> Meanwhile, checking slurmd.log and nhc.log on those node all seem to 
> be ok at the reported timepoint.
> 
> So we guess it’s slurmctld launch some detection towards those compute 
> node and didn’t get response, thus lead to slurmctld thinking those 
> node to be not responding.
> 
> Then the question is what detect action do slurmctld launched? How did 
> it determine whether a node is responsive or non-responsive?
> 
> And is it possible to customize slurmctld’s behavior on such 
> detection, for example wait timeout or retry count before determine 
> the node to be not responding?
> 

--
Kamil Wilczek  [https://keys.openpgp.org/] 
[D415917E84B8DA5A60E853B6E676ED061316B69B]
Laboratorium Komputerowe
Wydział Matematyki, Informatyki i Mechaniki Uniwersytet Warszawski

ul. Banacha 2
02-097 Warszawa

Tel.: 22 55 44 392
https://www.mimuw.edu.pl/
https://www.uw.edu.pl/

Re: [slurm-users] 答复: how do slurmctld determine whether a compute node is not responding?

[Kamil Wilczek]

I think that the previous answer from ole.h.niel...@fysik.dtu.dk
might be helpful in that case. But whether this is in the same
building or in some more distatnt location the timeout shouldn't
exceed a second or two. I do not understand why the timeouts are
set so high by default -- workloads messing up the network?

My defaults are (I did not change them):

BatchStartTimeout   = 10 sec
EioTimeout  = 60
GetEnvTimeout   = 2 sec
MessageTimeout  = 10 sec
PrologEpilogTimeout = 65534
ResumeTimeout   = 60 sec
SlurmctldTimeout= 120 sec
SlurmdTimeout   = 300 sec
SuspendTimeout  = 30 sec
TCPTimeout  = 2 sec
UnkillableStepTimeout   = 60 sec

The "TCPTimeout" might also be a viable option, because this would
be the first limit to reach and nodes under heavy load might have
problems with network operation, especially if some resources
are not reserved for the OS.

--

W dniu 11.07.2022 o 10:27, taleinterve...@sjtu.edu.cn pisze:

Hello, Kamil Wilczek:

Well I agree that the non-responding case may caused by network unstable, since 
our slurm cluster has 2 part nodes geographical distant distributed with only 
ethernet link them. Those reported nodes are all in one building while the 
slurmctld node in another building.
But we can do nothing about the network infrastructure, so we are more 
interested in adjusting slurm to make it tolerate such short-time non 
responding case.
Or is there possible to tell slurmctld do those detection through certain proxy 
node? For example we have slurmctld backup node in the same building with those 
reported compute nodes, if slurm can use this backup controller node to execute 
detection towards some part of compute nodes, the result may be more stable.

-邮件原件-
发件人: Kamil Wilczek <>
发送时间: 2022年7月11日 15:53
收件人: Slurm User Community List ; 
taleinterve...@sjtu.edu.cn
主题: Re: [slurm-users] how do slurmctld determine whether a compute node is not 
responding?

Hello,

I know that this is not quite the answer, but you could additionally (and maybe 
you already did this :)) check if this is not a network
problem:

* Are the nodes available outside of Slurm during that time? SSH, ping?
* If you have a monitoring system (Prometheus, Icinga, etc.), are
there any issues reported?

And lastly, did you try to set log level to "debug" for "slurmd"
and "slurmctld"?

Kind Regards


--
Kamil Wilczek  [https://keys.openpgp.org/]
[D415917E84B8DA5A60E853B6E676ED061316B69B]
Laboratorium Komputerowe
Wydział Matematyki, Informatyki i Mechaniki
Uniwersytet Warszawski

ul. Banacha 2
02-097 Warszawa

Tel.: 22 55 44 392
https://www.mimuw.edu.pl/
https://www.uw.edu.pl/


OpenPGP_signature
Description: OpenPGP digital signature

[slurm-users] Frontend node mode issues identified in v22.05.2

[Jordi Blasco]

Hi,

I use the front-end node mode
 to emulate a real cluster
in order to validate the Slurm configuration in a Docker container and
develop custom plugins. With versions 21.08.8-2 and 22.05.2, slurmd is
complaining about not being able to find the frontend node.

slurmd -D -vvv
...
slurmd: error: _find_node_record: lookup failure for node "slurm-simulator"
slurmd: error: _find_node_record: lookup failure for node
"slurm-simulator", alias "slurm-simulator"
slurmd: error: slurmd initialization failed
[root@slurm-simulator /]#
[root@slurm-simulator /]# squeue -V
slurm 22.05.2

With version 20.11.9 and using the same slurm configuration, it works as
usual. The only difference is the flags used in the v22 to integrate
cgroups v2.

Any suggestions?

Thanks in advance.

Jordi

Re: [slurm-users] Frontend node mode issues identified in v22.05.2

[Ole Holm Nielsen]

On 7/11/22 12:54, Jordi Blasco wrote:
I use the front-end node mode 
 to emulate a real 
cluster in order to validate the Slurm configuration in a Docker container 
and develop custom plugins. With versions 21.08.8-2 and 22.05.2, slurmd is 
complaining about not being able to find the frontend node.


slurmd -D -vvv
...
slurmd: error: _find_node_record: lookup failure for node "slurm-simulator"
slurmd: error: _find_node_record: lookup failure for node 
"slurm-simulator", alias "slurm-simulator"

slurmd: error: slurmd initialization failed


This could be a DNS lookup issue.  Can you ping the node named 
"slurm-simulator"?


/Ole

Re: [slurm-users] Frontend node mode issues identified in v22.05.2

[Jordi Blasco]

Thank Ole,

I checked the /etc/nsswitch.conf and I have even setup a dnsmasq service,
just in case.

[root@slurm-simulator /]# cat /etc/nsswitch.conf | grep hosts
# Valid databases are: aliases, ethers, group, gshadow, hosts,
hosts:  files dns myhostname

[root@slurm-simulator /]# ping slurm-simulator -c 1
PING slurm-simulator (172.17.0.4) 56(84) bytes of data.
64 bytes from slurm-simulator (172.17.0.4): icmp_seq=1 ttl=64 time=0.022 ms

--- slurm-simulator ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.022/0.022/0.022/0.000 ms

[root@slurm-simulator /]# cat /etc/resolv.conf | grep -v "^#"
nameserver 172.17.0.4
nameserver 172.31.0.2
search eu-west-3.compute.internal
[root@slurm-simulator /]# host slurm-simulator
slurm-simulator has address 172.17.0.4
[root@slurm-simulator /]# host 172.17.0.4
4.0.17.172.in-addr.arpa domain name pointer slurm-simulator.


Regards,

Jordi



On Mon, 11 Jul 2022 at 23:09, Ole Holm Nielsen 
wrote:

> On 7/11/22 12:54, Jordi Blasco wrote:
> > I use the front-end node mode
> >  to emulate a real
> > cluster in order to validate the Slurm configuration in a Docker
> container
> > and develop custom plugins. With versions 21.08.8-2 and 22.05.2, slurmd
> is
> > complaining about not being able to find the frontend node.
> >
> > slurmd -D -vvv
> > ...
> > slurmd: error: _find_node_record: lookup failure for node
> "slurm-simulator"
> > slurmd: error: _find_node_record: lookup failure for node
> > "slurm-simulator", alias "slurm-simulator"
> > slurmd: error: slurmd initialization failed
>
> This could be a DNS lookup issue.  Can you ping the node named
> "slurm-simulator"?
>
> /Ole
>
>

[slurm-users] Is there a way create reservations w/o being Operator or Admin?

[David Henkemeyer]

I would like to remove the restriction that users must be at least operator
level to do "scontrol create reservation".  So, either I could:

   - Change the default AdminLevel to operator.  Is that possible?
   - Remove the restriction that a user has to be operator to create a
   reservation.  Is that possible?

My motivation for this is as follows: currently, we don't have any users
defined in the slurm accounting DB, and this works for almost everything
our users do (srun, sbatch, squeue, scancel, etc).  I would prefer to keep
it that way, so we can avoid having to add all our users into the
accounting DB (and then add new users when we bring new people onboard).

Thanks in advance,
David

[slurm-users] Allow SFTP on a specific compute node

[Ratnasamy, Fritz]

Hello,

 Currently, our cluster does not allow ssh to compute nodes for users
unless they have
a running job on that compute node. I believe a system admin has set up a
PAM module
that does the block. Whn trying ssh, this is the message returned:
Access denied by pam_slurm_adopt: you have no active jobs on this node
Connection closed by 10.135.242.188 port 22

However, we would like to allow sftp on a specific compute node for
specific users.
Any idea on how to do that?
Thanks,


*Fritz Ratnasamy*

Data Scientist

Information Technology

The University of Chicago

Booth School of Business

5807 S. Woodlawn

Chicago, Illinois 60637

Phone: +(1) 773-834-4556

Re: [slurm-users] Allow SFTP on a specific compute node

[Ole Holm Nielsen]

On 7/12/22 06:51, Ratnasamy, Fritz wrote:
  Currently, our cluster does not allow ssh to compute nodes for users 
unless they have
a running job on that compute node. I believe a system admin has set up a 
PAM module

that does the block. Whn trying ssh, this is the message returned:
Access denied by pam_slurm_adopt: you have no active jobs on this node
Connection closed by 10.135.242.188 port 22

However, we would like to allow sftp on a specific compute node for 
specific users.

Any idea on how to do that?


SFTP uses the SSH protocol, see 
https://en.wikipedia.org/wiki/Secure_file_transfer_program


With pam_slurm_adopt you can only connect with SSH to a node where you 
have a running job, which is a Good Thing.


What's the usage scenario for needing file transfer access to a compute 
node running someone else's job?


/Ole

Re: [slurm-users] Allow SFTP on a specific compute node

[Jake Jellinek]

I cannot think of any way to do this within the Slurm configuration

I would solve this by having a wrapper run at boot time which started a new 
sshd process on a different port which you secured (ie only that user could 
connect) and then start this as part of your boot time scripts
If your script was started on one of your ‘special’ machines, start the second 
instance of sshd…..if not, do nothing

Hope that helps

On 12 Jul 2022, at 05:53, Ratnasamy, Fritz  
wrote:


Hello,

 Currently, our cluster does not allow ssh to compute nodes for users unless 
they have
a running job on that compute node. I believe a system admin has set up a PAM 
module
that does the block. Whn trying ssh, this is the message returned:
Access denied by pam_slurm_adopt: you have no active jobs on this node
Connection closed by 10.135.242.188 port 22

However, we would like to allow sftp on a specific compute node for specific 
users.
Any idea on how to do that?
Thanks,


Fritz Ratnasamy
Data Scientist
Information Technology
The University of Chicago
Booth School of Business
5807 S. Woodlawn
Chicago, Illinois 60637
Phone: +(1) 773-834-4556