Slurmdbd has an issue and from the logs is still trying to load the old
version:
[2021-01-22T14:17:18.430] MySQL server version is: 5.5.68-MariaDB
[2021-01-22T14:17:18.433] error: Database settings not recommended values:
innodb_buffer_pool_size innodb_log_file_size innodb_lock_wait_timeout
[2021-01-22T14:17:18.528] Accounting storage MYSQL plugin loaded
[2021-01-22T14:17:18.529] error: chdir(/var/log): Permission denied
[2021-01-22T14:17:18.529] chdir to /var/tmp
*[2021-01-22T14:17:18.531] slurmdbd version 20.02.3
started[2021-01-22T14:56:40.334] error: g_slurm_auth_unpack: remote
plugin_id 144 not found*
[2021-01-22T14:56:40.334] error: slurm_unpack_received_msg: Invalid
Protocol Version 9216 from uid=-1 from problem connection: Socket operation
on non-socket
[2021-01-22T14:56:40.334]* error: slurm_unpack_received_msg: Incompatible
versions of client and server code*
[2021-01-22T14:56:40.345] error: CONN:7 Failed to unpack SLURM_PERSIST_INIT
message
[2021-03-03T09:49:57.607] Terminate signal (SIGINT or SIGTERM) received
[2021-03-03T09:49:57.610] Unable to remove pidfile '/var/run/slurmdbd.pid':
Permission denied
But I know it's updated:
rpm -qa|grep slurmdbd
slurm20-slurmdbd-20.11.3-mybuild.x86_64
And the pid file is not there:
ls -l /var/run/slurmdbd.pid
ls: cannot access /var/run/slurmdbd.pid: No such file or directory
And on the service file:
cat /usr/lib/systemd/system/slurmdbd.service
[Unit]
RequiresMountsFor=/cm/shared
Description=Slurm DBD accounting daemon
After=network.target munge.service
ConditionPathExists=/etc/slurm/slurmdbd.conf
[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/slurmdbd
*ExecStart=/cm/shared/apps/slurm/20.11.3/sbin/slurmdbd -D $SLURMDBD_OPTIONS*
ExecReload=/bin/kill -HUP $MAINPID
LimitNOFILE=65536
I reinstalled the slurmdbd file that is local:
Dependencies Resolved
==================================================================================================================================================================================================
Package Arch
Version Repository
Size
==================================================================================================================================================================================================
Reinstalling:
slurm20-slurmdbd x86_64
20.11.3-mybuild
/slurm20-slurmdbd-20.11.3-mybuild.x86_64 2.3 M
Transaction Summary
==================================================================================================================================================================================================
Reinstall 1 Package
Total size: 2.3 M
Installed size: 2.3 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : slurm20-slurmdbd-20.11.3-mybuild.x86_64
1/1
Verifying : slurm20-slurmdbd-20.11.3-mybuild.x86_64
1/1
Installed:
slurm20-slurmdbd.x86_64 0:20.11.3-mybuild
What did I miss? In the upgrade page
<https://slurm.schedmd.com/quickstart_admin.html> I see this:
The libslurm.so version is increased every major release. So things like
MPI libraries with Slurm integration should be recompiled. Sometimes it
works to just symlink the old .so name(s) to the new one, but this has no
guarantee of working.
So I have this:
locate libslurm.so
/cm/shared/apps/slurm/20.11.3/lib64/libslurm.so
/cm/shared/apps/slurm/20.11.3/lib64/libslurm.so.36
/cm/shared/apps/slurm/20.11.3/lib64/libslurm.so.36.0.0
Is there some other place the old version is being referenced?
