Hi Andy, I followed your advice - md5sum /etc/munge/munge.key and they are the same on all systems.
what else can it be? I will cheek the clock on the systems again as suggested by Chris Best, Eric _____________________________________________________________________________________________________ Eric F. Alemany System Administrator for Research Division of Radiation & Cancer Biology Department of Radiation Oncology Stanford University School of Medicine Stanford, California 94305 Tel:1-650-498-7969<tel:1-650-498-7969> No Texting Fax:1-650-723-7382<tel:1-650-723-7382> On May 7, 2018, at 5:39 PM, Riebs, Andy <andy.ri...@hpe.com<mailto:andy.ri...@hpe.com>> wrote: The /etc/munge/ munge.key is different on the systems. Try md5sum /etc/munge/munge.key on both systems to see if they are the same... -- Andy Riebs andy.ri...@hpe.com<mailto:andy.ri...@hpe.com> Hewlett-Packard Enterprise +1 404 648 9024 ________________________________ From: slurm-users <slurm-users-boun...@lists.schedmd.com<mailto:slurm-users-boun...@lists.schedmd.com>> on behalf of Eric F. Alemany <ealem...@stanford.edu<mailto:ealem...@stanford.edu>> Sent: Monday, May 7, 2018 7:40:53 PM To: Slurm User Community List Subject: Re: [slurm-users] Nodes are down after 2-3 minutes. Hi Chris, I followed the link as well as the instruction on “Securing the installation” and “Testing the installation” The only thing that i am not able to do is: Check if a credential can be remotely decoded eric@radoncmaster:/etc/munge$ munge -n | ssh e...@radonc01.stanford.edu<mailto:e...@radonc01.stanford.edu> unmunge unmunge: Error: Invalid credential Did i do something wrong somewhere ? Thank you for your help _____________________________________________________________________________________________________ Eric F. Alemany System Administrator for Research Division of Radiation & Cancer Biology Department of Radiation Oncology Stanford University School of Medicine Stanford, California 94305 Tel:1-650-498-7969<tel:1-650-498-7969> No Texting Fax:1-650-723-7382<tel:1-650-723-7382> On May 7, 2018, at 3:42 PM, Chris Samuel <ch...@csamuel.org<mailto:ch...@csamuel.org>> wrote: On Tuesday, 8 May 2018 8:38:47 AM AEST Eric F. Alemany wrote: I thought i did but I will do it again If that doesn't work then check the "Securing the Installation" and "Testing the Installation" parts of the munge docs here (ignore the installation part): https://github.com/dun/munge/wiki/Installation-Guide Good luck! Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC