I'm setting up an EC2 SLURM cluster and when an instance doesn't resume fast enough I get an error like:
node c7-c5-24xl-464 not resumed by ResumeTimeout(600) - marking down and power_save I keep running into issues where my cloud nodes do not show up in sinfo and I can't display their information with scontrol. This makes it difficult to know which of my CLOUD nodes are available for scheduling and which are down for some reason and can't be used. I haven't figured out when slurm will show a cloud node and when it won't and this make it pretty hard to manage the cluster. Would I be better off just removing the CLOUD attribute on my EC2 nodes? What is the advantage of making them CLOUD nodes if it just make it more difficult to manage the cluster?
