Hi Jeffrey, Jeffrey T Frey <f...@udel.edu> writes:
> On a cluster running Slurm 17.11.8 (cons_res) I can submit a job that > requests e.g. 2 nodes with unique features on each: > > $ sbatch --nodes=2 --ntasks-per-node=1 --constraint="[256GB*1&192GB*1]" … > > The job is submitted and runs as expected: on 1 node with feature > "256GB" and 1 node with feature "192GB." A similar job on a cluster > running 20.11.1 (cons_res OR cons_tres, tested with both) fails to > submit: > > sbatch: error: Batch job submission failed: Requested node configuration is > not available > > I enabled debug5 output with NodeFeatures: > > [2020-12-16T08:53:19.024] debug: JobId=118 feature list: [512GB*1&768GB*1] > [2020-12-16T08:53:19.025] NODE_FEATURES: _log_feature_nodes: FEAT:512GB > COUNT:1 PAREN:0 OP:XAND ACTIVE:r1n[00-47] AVAIL:r1n[00-47] > [2020-12-16T08:53:19.025] NODE_FEATURES: _log_feature_nodes: FEAT:768GB > COUNT:1 PAREN:0 OP:END ACTIVE:r2l[00-31] AVAIL:r2l[00-31] > [2020-12-16T08:53:19.025] NODE_FEATURES: valid_feature_counts: feature:512GB > feature_bitmap:r1n[00-47],r2l[00-31],r2x[00-10] > work_bitmap:r1n[00-47],r2l[00-31],r2x[00-10] tmp_bitmap:r1n[00-47] count:1 > [2020-12-16T08:53:19.025] NODE_FEATURES: valid_feature_counts: feature:768GB > feature_bitmap:r1n[00-47],r2l[00-31],r2x[00-10] > work_bitmap:r1n[00-47],r2l[00-31],r2x[00-10] tmp_bitmap:r2l[00-31] count:1 > [2020-12-16T08:53:19.025] NODE_FEATURES: valid_feature_counts: > NODES:r1n[00-47],r2l[00-31],r2x[00-10] HAS_XOR:T status:No error > [2020-12-16T08:53:19.025] select/cons_tres: _job_test: SELECT_TYPE: test 0 > pass: test_only > [2020-12-16T08:53:19.026] debug2: job_allocate: setting JobId=118_* to > "BadConstraints" due to a flaw in the job request (Requested node > configuration is not available) > [2020-12-16T08:53:19.026] _slurm_rpc_submit_batch_job: Requested node > configuration is not available > > My syntax agrees with the 20.11.1 documentation (online and man pages) so it > seems correct — and it works fine in 17.11.8. Any ideas? We don't use features, so I am only guessing that you have defined the features incorrectly. Do you have something like Feature=512GB,768GB ? However, you might want to consider memory as a consumable resource and allow Slurm to find available nodes, rather than defining the maximum memory as a feature. Regards Loris -- Dr. Loris Bennett (Hr./Mr.) ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de
