There may be prettier ways, but this gets the job done. Captures the output
from each sbatch command to get a job ID, colon separates the ones in the
second group, and removes the trailing colon before submitting the last job:
#!/bin/bash
JOB1=$(sbatch job1.sh | awk '{print $NF}')
echo "Submitted job 1, id ${JOB1}"
JOB2N=""
JOB2N_FILES="job2.sh job3.sh"
for job in ${JOB2N_FILES}; do
JOB2N="${JOB2N}$(sbatch --dependency=afterok:$JOB1 $job | awk '{print $NF}'):"
echo "Submitted $job, list is now ${JOB2N}"
done
JOB2N=$(echo ${JOB2N} | sed 's/:$//g')
echo "Submitting last job"
sbatch --dependency=afterok:$JOB2N joblast.sh
From: slurm-users <slurm-users-boun...@lists.schedmd.com> on behalf of Jason
Simms <sim...@lafayette.edu>
Date: Tuesday, March 2, 2021 at 1:18 PM
To: Slurm User Community List <slurm-users@lists.schedmd.com>
Subject: [slurm-users] Managing Multiple Dependencies
External Email Warning
This email originated from outside the university. Please use caution when
opening attachments, clicking links, or responding to requests.
________________________________
Hello all,
I am relatively new to the nuances of handling complex dependencies in Slurm,
so I'm hoping the hive mind can help. I have a user wanting to accomplish the
following:
* submit one job
* submit multiple jobs that are dependent on the output from the first job
(so they just need to launch once the first job has completed)
* submit one final job dependent on all the previous jobs completing
Is there a way to do this cleanly? So it's a three stage process. I have ideas
in my head of writing Slurm JobIDs to a file, reading them out, and managing
dependencies that way, but perhaps there is a more efficient way (or perhaps
not!).
Warmest regards,
Jason
--
Jason L. Simms, Ph.D., M.P.H.
Manager of Research and High-Performance Computing
XSEDE Campus Champion
Lafayette College
Information Technology Services
710 Sullivan Rd | Easton, PA 18042
Office: 112 Skillman Library
p: (610) 330-5632
