Thanks for your help, Prentice.
Sorry, yes – centos 7.5 installed on a fresh HDD. I rebooted and checked that
chronyd is disabled. ntpd is running. The rest of the cluster uses centos 7.5
and ntp so it’s possible, although maybe not ideal.
I’m running ntpq on the new compute node. It is looking to the slurm head node
which is also set up as the ntp server. Here’s the output:
[root ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
HEADNODE_IP .XFAC. 16 u - 1024 0 0.000 0.000 0.000
It was a bit of a pain to get set up. The time difference was several hours so
ntp would have taken ages to fix on its own. I have used ntpdate successfully
on the existing compute nodes, but got a “no server suitable for
synchronization found” error here. ‘ntpd -gqx’ timed out. So in order to set
the time, I had to point ntp to the default centos pool of ntp servers to set
the time and then point it back to the headnode. After that, ‘ntpd -gqx’ ran
smoothly and I assume (based on the ntpq output) that it worked. Running ‘date’
on the new compute and existing head node simultaneously returns the same time
to within ~1 sec rather than the 7:30 gap from the log file.
Not sure if it’s relevant to this problem, but the new compute node is on a
different subnet connected to a different port than the existing compute nodes.
This is the first time that I’ve set up a node on a different subnet. I figured
it be simple to point slurm to the new node, but I didn’t anticipate ntp and
munge issues.
Thanks,
Gard
From: slurm-users <slurm-users-boun...@lists.schedmd.com> on behalf of Prentice
Bisbal <pbis...@pppl.gov>
Reply-To: Slurm User Community List <slurm-users@lists.schedmd.com>
Date: Tuesday, October 27, 2020 at 12:22 PM
To: "slurm-users@lists.schedmd.com" <slurm-users@lists.schedmd.com>
Subject: Re: [slurm-users] [External] Munge thinks clocks aren't synced
You don't specify what OS or version you're using. If you're using RHEL 7 or a
derivative, chrony is used by default over ntpd, so there could be some
confusion between chronyd and ntpd. If you haven't done so already, I'd check
to see which daemon is actually running on your system.
Can you share the complete output of ntpq -p with us, and let us know what
nodes the output is from? You might want to run 'ntpdate' before starting ntpd.
If the clocks are too far off, either ntpd won't correct the time, or it will
take a long time. ntpdate immediately syncs up the time between servers.
I would make sure ntpdate is installed and enabled, then reboot both compute
nodes. This will make sure that ntpdate is called at startup before ntpd, and
will then make sure all start using the correct time.
--
Prentice
On 10/27/20 2:08 PM, Gard Nelson wrote:
Hi everyone,
I’m adding a new node to an existing cluster. After installing slurm and the
prereqs, I synced the clocks with ntpd. When I run ‘ntpq -p’, I get 0.0 for
delay, offset and jitter. (the slurm head node is also the ntp server) ‘date’
also gives me identical times for the head and compute nodes. However, when I
start slurmd, I get a munge error about the clocks being out of sync. From the
slurmctld log:
[2020-10-27T11:02:06.511] node NEW_NODE returned to service
[2020-10-27T11:02:07.265] error: Munge decode failed: Rewound credential
[2020-10-27T11:02:07.265] ENCODED: Tue Oct 27 11:09:45 2020
[2020-10-27T11:02:07.265] DECODED: Tue Oct 27 11:02:07 2020
[2020-10-27T11:02:07.265] error: Check for out of sync clocks
[2020-10-27T11:02:07.265] error: slurm_unpack_received_msg:
MESSAGE_NODE_REGISTRATION_STATUS has authentication error: Rewound credential
[2020-10-27T11:02:07.265] error: slurm_unpack_received_msg: Protocol
authentication error
[2020-10-27T11:02:07.275] error: slurm_receive_msg [HEAD_NODE_IP:PORT]:
Unspecified error
I restarted ntp, munge and the slurm daemons on both nodes before this last
error was generated. Any idea what’s going on here?
Thanks,
Gard
CONFIDENTIALITY NOTICE
This e-mail message and any attachments are only for the use of the intended
recipient and may contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended
recipient, any disclosure, distribution or other use of this e-mail message or
attachments is prohibited. If you have received this e-mail message in error,
please delete and notify the sender immediately. Thank you.
--
Prentice Bisbal
Lead Software Engineer
Research Computing
Princeton Plasma Physics Laboratory
http://www.pppl.gov<https://urldefense.com/v3/__http:/www.pppl.gov__;!!LM3lv1w8qtQ!AUViCRtpIXKV37Z4WGp5j64ppClYVIuzUEXXvfoDHHD_tVjDVMA9b2gBHtaWUHsEPdvmkQ$>
