Hello, I have met a similar issue with slurmrestd authentication failed error, similar question: https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html
I have installed `slurm 21.08.6` on CentOS 7.9.2009 container, basic service is running fine ``` [root@slurmctl supervisor]# ls -l /.dockerenv -rwxr-xr-x. 1 root root 0 Mar 17 23:17 /.dockerenv [root@slurmctl supervisor]# srun --partition normal hostname slurmctl ``` Slurmrestd is compiled with `--enable-slurmrestd` successfully, slurmrestd JWT configuration is as follows: ``` [root@slurmctl slurmctld]# dd if=/dev/random of=/var/spool/slurm/jwt_hs256.key bs=32 count=1 [root@slurmctl slurmctld]# scontrol show config |grep -i auth AuthAltTypes = auth/jwt AuthAltParameters = jwt_key=/var/spool/slurm/jwt_hs256.key AuthInfo = (null) AuthType = auth/munge [root@slurmctl slurmctld]# ll -l /var/spool/slurm/jwt_hs256.key -rw-r--r--. 1 root root 32 Mar 17 23:21 /var/spool/slurm/jwt_hs256.key # start slurmrestd process [root@slurmctl slurmctld]# SLURMRESTD_SECURITY=disable_user_check /usr/sbin/slurmrestd -vvvv 0.0.0.0:19090 # check process and environ [root@slurmctl slurmctld]# ps -ef |grep slurmrestd root 1235 236 0 23:26 ? 00:00:00 /usr/sbin/slurmrestd -vvvvv 0.0.0.0:19090 [root@slurmctl slurmctld]# cd /proc/1235/ [root@slurmctl 1235]# cat environ | tr '\0' "\n" TERM=xterm TINI_VERSION=v0.18.0 SHLVL=1 HOSTNAME=slurmctl SUPERVISOR_ENABLED=1 SUPERVISOR_PROCESS_NAME=slurmrestd PWD=/ SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock SUPERVISOR_GROUP_NAME=slurmrestd PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin HOME=/root SLURMRESTD_SECURITY=disable_user_check _=/usr/bin/supervisord ``` I have generated correct token to request, but slurmrestd log message reported authentication failed: Unspecified error ``` [root@slurmctl 1235]# scontrol token username=slurm SLURM_JWT=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw [root@slurmctl 1235]# token="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw" [root@slurmctl 1235]# curl 172.17.0.4:19090/openapi -H "X-SLURM-USER-TOKEN: $token" -H "X-SLURM-USER-NAME: slurm" Authentication failure slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[172.17.0.1]:38090] attempting user_name slurm token authentication pass through slurmrestd: error: operations_router: [[172.17.0.1]:38090] authentication failed: Unspecified error slurmrestd: debug2: _on_message_complete_request: [[172.17.0.1]:38090] on_http_request rejected: Unspecified error ``` But! I have found that I'm setting `SLURM_JWT` environment variable for process, whatever token value is authenticated normally ``` # start process with SLURM_JWT [root@slurmctl 1235]# cd /proc/2108/ [root@slurmctl 2108]# cat environ |tr "\0" "\n" TERM=xterm TINI_VERSION=v0.18.0 SHLVL=1 HOSTNAME=slurmctl SLURM_JWT=randomtoken SUPERVISOR_ENABLED=1 SUPERVISOR_PROCESS_NAME=slurmrestd PWD=/ SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock SUPERVISOR_GROUP_NAME=slurmrestd PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin HOME=/root SLURMRESTD_SECURITY=disable_user_check _=/usr/bin/supervisord # request OK [root@slurmctl 2108]# curl 172.17.0.4:19090/slurm/v0.0.35/jobs -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: slurm" [ ] [root@slurmctl 2108]# curl 172.17.0.4:19090/slurm/v0.0.35/jobs -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: errorvalue" [ ] ``` So, I'm confused about JWT authentication. Q1: What is used for the `SLURM_JWT` environment variable, is it required for JWT? Related search from github source repo: https://github.com/SchedMD/slurm/search?q=SLURM_JWT Q2: How to use slurmrestd JWT authentication? -------- Thanks, Chenyang Yan
