Thank you to both Kilian and Chris, I have running on the slurm server to report once when any of the nodes go into “Drain” State:
sudo -u slurm bash -c “strigger --set -D -p /etc/slurm/triggers/slurm_admin_notify --flags=perm" /bin/mail -s “ClusterName DrainedNode:$*” our_admin_email_address Exactly what we needed. Success. Thank you! Jodie Center for Advanced Computing Cornell University On Sep 19, 2018, at 5:14 PM, Christopher Benjamin Coffey <chris.cof...@nau.edu> wrote: Killian, thank you very much! Never noticed the perm flag! Best, Chris — Christopher Coffey High-Performance Computing Northern Arizona University 928-523-1167 On 9/19/18, 10:01 AM, "slurm-users on behalf of Kilian Cavalotti" <slurm-users-boun...@lists.schedmd.com on behalf of kilian.cavalotti.w...@gmail.com> wrote: On Wed, Sep 19, 2018 at 9:21 AM Christopher Benjamin Coffey <chris.cof...@nau.edu> wrote: > The only thing that I've gotten working so far is this: > sudo -u slurm bash -c "strigger --set -D -n cn15 -p > /common/adm/slurm/triggers/nodestatus" > > So, that will run the nodestatus script which emails when the node cn15 gets > set into drain state. What I'd like to do, which I haven't put time into > figuring out, is how to setup a persistent trigger that can run when ANY node > goes into drain state. Let me know if you figure that out. As you can see > above, the trigger has to be setup by the slurm user. strigger takes a "--flags=perm" option, which makes the trigger permanent and doesn't purge it after the event happened. So it doesn't need tb re-armed in the trigger script for the next event. Also, not specifying any job id nor node name in the strigger command will make the trigger apply to all nodes. That's how we set our default "down" trigger for all nodes on our cluster: # su -s /bin/bash -c "strigger --set --down --program=/share/admin/scripts/slurm/triggers/down.sh --flags=perm" slurm Also note that the script given to "--program" needs to be executable by user Slurm on the controller node(s). Cheers, -- Kilian
