On 7/14/22 18:49, Timony, Mick wrote:
Which database server and version do you run, MySQL or MariaDB? What's
your Slurm version?
mariadb 5.5.68 and a patched version of slurm 21.08.7
We run the same MariaDB on CentOS 7.9, and Slurm 21.08.8-2.
Did you already make appropriate database purges to reduce the size? I
have some notes in my Wiki page
https://wiki.fysik.dtu.dk/niflheim/Slurm_database#setting-database-purge-parameters
<https://wiki.fysik.dtu.dk/niflheim/Slurm_database#setting-database-purge-parameters>
No, we have not made any changes yet as am concerned that it will cause
performance issues.
IMHO, database purging is an important way to *improve* the performance!
Reducing the database size is beneficial, also for making the daily
database dump faster. As I wrote in the above Wiki page, you should
introduce purging very gently and decrease the purging intervals in small
steps over a number of weeks:
A monthly purge operation can be a huge amount of work for a database depending
on its size, and you certainly want to cut down the amount of work required
during the purges. If you did not use purges before, it is probably a good idea
to try out a series of daily purges starting with:
PurgeEventAfter=2000days
PurgeJobAfter=2000days
PurgeResvAfter=2000days
PurgeStepAfter=2000days
PurgeSuspendAfter=2000days
If this works well over a few days, decrease the purge interval 2000days little
by little and try again (1800, 1500, etc) until you after many iterations come
down to the desired final purge intervals.
Please note that it's acceptable for the slurmdbd to be slow, or even down
for many hours. The slurmctld will cache all data (up to a certain limit)
while the slurmdbd is not responding.
/Ole
