Re: [slurm-users] [External] Re: Slurm 22.05.8 - salloc not starting shell on remote host

2023-05-19 Thread Ryan Novosielski
Make sure you don’t have a firewall blocking connections back to the login node from the cluster. We had that problem at Rutgers before. Sent from my iPhone On May 19, 2023, at 13:13, Prentice Bisbal wrote:  Brian, Thanks for the reply, and I was hoping that would be the fix, but that doesn

[slurm-users] Question about PMIX ERROR messages being emitted by some child of srun process

2023-05-19 Thread Pritchard Jr., Howard
HI, So I’m testing the use of Open MPI 5.0.0 pre-release with the Slurm/PMIx setup currently on NERSC Perlmutter system. First off, if I use the PRRte launch system, I don’t see the issue I’m raising here. But, many NERSC users prefer to use the srun “native” launch method with applications co

Re: [slurm-users] [External] Re: Slurm 22.05.8 - salloc not starting shell on remote host

2023-05-19 Thread Prentice Bisbal
This is fixed. I was a little overzealous in my IPtables rules on the login host and was restricting traffic from the compute node back to the login node. Thanks to Ryan and Brian for the quick replies offering suggestions. Prentice On 5/19/23 1:11 PM, Prentice Bisbal wrote: Brian, Thanks

Re: [slurm-users] [External] Re: Slurm 22.05.8 - salloc not starting shell on remote host

2023-05-19 Thread Prentice Bisbal
Brian, Thanks for the reply, and I was hoping that would be the fix, but that doesn't seem to be the case. I'm using 22.05.8, which isn't that old. I double-checked the documentation archives for version 22.05.08's documetation, and setting LaunchParameters=use_interactive_step should be va

Re: [slurm-users] Slurm 22.05.8 - salloc not starting shell on remote host

2023-05-19 Thread Brian Andrus
Defaulting to a shell for salloc is a newer feature. For your version, you should:     srun -n 1 -t 00:10:00 --mem=1G --pty bash Brian Andrus On 5/19/2023 8:24 AM, Ryan Novosielski wrote: I’m not at a computer, and we run an older version of Slurm yet so I can’t say with 100% confidence that

Re: [slurm-users] Slurm 22.05.8 - salloc not starting shell on remote host

2023-05-19 Thread Ryan Novosielski
I’m not at a computer, and we run an older version of Slurm yet so I can’t say with 100% confidence that his this has changed and I can’t be too specific, but I know that this is the behavior you should expect from that command. I believe that there are configuration options to make it behave di

Re: [slurm-users] How do I set SBATCH_EXCLUSIVE to its default value?

2023-05-19 Thread Jeffrey T Frey
> I get that these correspond > > --exclusive=userexport SBATCH_EXCLUSIVE=user > --exclusive=mcs export SBATCH_EXCLUSIVE=mcs > But --exclusive has a default behavior if I don't assign it a value. What do > I set SBATCH_EXCLUSIVE to, to get the same default behavior? Try setting

Re: [slurm-users] HELP: error between compilation and execution on gpu cluster

2023-05-19 Thread Saksham Pande 5-Year IDD Physics
Hello, is writing the --with-pmi flag sufficient or do I have to write it in the form --with-pmi=, where it points to a directory, and if so, where? Slightly confused by the syntax provided in the documentation. [sakshamp.phy20.itbhu@login2]$ srun --mpi=list srun: MPI types are... srun: cray_shast

[slurm-users] Slurm 22.05.8 - salloc not starting shell on remote host

2023-05-19 Thread Prentice Bisbal
I'm setting up Slurm from scratch for the first time ever. Using 22.05.8 since I haven't had a changed to upgrade our DB server to 23.02 yet. When I try to use salloc to get a shell on a compute node (ranger-s22-07), I end up with a shell on the login node (ranger): [pbisbal@ranger ~]$ salloc

Re: [slurm-users] HELP: error between compilation and execution on gpu cluster

2023-05-19 Thread Saksham Pande 5-Year IDD Physics
Thank you for responding. The output of ompi_info regarding configuration is Configure command line: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking'

[slurm-users] How do I set SBATCH_EXCLUSIVE to its default value?

2023-05-19 Thread Carl Ponder
The SBATCH_EXCLUSIVE environment-variable is supposed to be equivalent to using the --exclusive flag on the command-line or in the sbatch-header *--exclusive*[={user|mcs}] The job allocation can not share nodes with other running jobs (or just other users with the "=user" option or with

Re: [slurm-users] HELP: error between compilation and execution on gpu cluster

2023-05-19 Thread Juergen Salk
Hi, I am not sure if this related to GPUs. I rather think the issue has to do with how your OpenMPI has been built. What does ompi_info command show? Look for "Configure command line" in the output. Does this include '--with-slurm' and '--with-pmi' flags? To my very best knowledge, both flags