[slurm-users] Re: Software builds using slurm

2024-06-10 Thread Renfro, Michael via slurm-users
At a certain point, you’re talking about workflow orchestration. Snakemake [1] and its slurm executor plugin [2] may be a starting point, especially since Snakemake is a local-by-default tool. I wouldn’t try reproducing your entire “make” workflow in Snakemake. Instead, I’d define the roughly 60

[slurm-users] Re: Software builds using slurm

2024-06-10 Thread Cutts, Tim via slurm-users
You have two options for managing those dependencies, as I see it) 1. you use SLURM’s native job dependencies, but this requires you to create a build script for SLURM 2. You use make to submit the jobs, and take advantage of the -j flag to make it run lots of tasks at once, just use a jo