Re: [slurm-users] RFC: Slurm Tool to Automate and Track Large Job Arrays

2019-01-18 Thread Alex Chekholko
Almost every place I worked built some site-specific tools for managing jobs that some people found very useful. E.g. https://github.com/StanfordBioinformatics/SJM http://clusterjob.org/ There have also been some efforts to standardize this sort of thing: https://www.commonwl.org/ I have not use

[slurm-users] RFC: Slurm Tool to Automate and Track Large Job Arrays

2019-01-18 Thread Erik Surface
Hi, I am a slurm end-user needing to run ~250k jobs, each takes ~2-4 hrs. With the traffic on our cluster and a limit of 7000 job submissions at a time, it will take about a month to run the full set, if we are lucky. I built a generic tool (in bash, currently) that automates the tracking and subm