Hello, We had the same error and we fixed it by adding `Environment="SLURM_JWT=daemon"` to the [Service] section of the unit file (in our case /usr/lib/systemd/system/slurmrestd.service ).
We have a bug (feature?) that makes us unable to use root or slurm user as user for slurmrestd service so maybe you'll encounter that as well. According to the release notes ( https://github.com/SchedMD/slurm/blob/master/NEWS ) this should be starting on version 22.05.0pre1 but it is already the case in our version 21.08.6. We have an administrator user for this can of usage. Our service file is like this: $ cat /usr/lib/systemd/system/slurmrestd.service [Unit] Description=Slurm REST daemon After=network.target munge.service ConditionPathExists=/etc/slurm/slurm.conf [Service] User=slurmadm Type=simple EnvironmentFile=-/etc/sysconfig/slurmrestd Environment="SLURM_JWT=daemon" ExecStart=/usr/sbin/slurmrestd -v -a rest_auth/jwt localhost:6820 ExecReload=/bin/kill -HUP $MAINPID [Install] WantedBy=multi-user.target Guillaume De: "Chenyang Yan" <memory.ya...@gmail.com> À: slurm-us...@schedmd.com Envoyé: Samedi 19 Mars 2022 14:09:06 Objet: [slurm-users] Slurmrestd authentication failed: Unspecified error Hello, I have met a similar issue with slurmrestd authentication failed error, similar question: [ https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html | https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html ] I have installed `slurm 21.08.6` on CentOS 7.9.2009 container, basic service is running fine ``` [root@slurmctl supervisor]# ls -l /.dockerenv -rwxr-xr-x. 1 root root 0 Mar 17 23:17 /.dockerenv [root@slurmctl supervisor]# srun --partition normal hostname slurmctl ``` Slurmrestd is compiled with `--enable-slurmrestd` successfully, slurmrestd JWT configuration is as follows: ``` [root@slurmctl slurmctld]# dd if=/dev/random of=/var/spool/slurm/jwt_hs256.key bs=32 count=1 [root@slurmctl slurmctld]# scontrol show config |grep -i auth AuthAltTypes = auth/jwt AuthAltParameters = jwt_key=/var/spool/slurm/jwt_hs256.key AuthInfo = (null) AuthType = auth/munge [root@slurmctl slurmctld]# ll -l /var/spool/slurm/jwt_hs256.key -rw-r--r--. 1 root root 32 Mar 17 23:21 /var/spool/slurm/jwt_hs256.key # start slurmrestd process [root@slurmctl slurmctld]# SLURMRESTD_SECURITY=disable_user_check /usr/sbin/slurmrestd -vvvv [ http://0.0.0.0:19090/ | 0.0.0.0:19090 ] # check process and environ [root@slurmctl slurmctld]# ps -ef |grep slurmrestd root 1235 236 0 23:26 ? 00:00:00 /usr/sbin/slurmrestd -vvvvv [ http://0.0.0.0:19090/ | 0.0.0.0:19090 ] [root@slurmctl slurmctld]# cd /proc/1235/ [root@slurmctl 1235]# cat environ | tr '\0' "\n" TERM=xterm TINI_VERSION=v0.18.0 SHLVL=1 HOSTNAME=slurmctl SUPERVISOR_ENABLED=1 SUPERVISOR_PROCESS_NAME=slurmrestd PWD=/ SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock SUPERVISOR_GROUP_NAME=slurmrestd PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin HOME=/root SLURMRESTD_SECURITY=disable_user_check _=/usr/bin/supervisord ``` I have generated correct token to request, but slurmrestd log message reported authentication failed: Unspecified error ``` [root@slurmctl 1235]# scontrol token username=slurm SLURM_JWT=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw [root@slurmctl 1235]# token="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw" [root@slurmctl 1235]# curl [ http://172.17.0.4:19090/openapi | 172.17.0.4:19090/openapi ] -H "X-SLURM-USER-TOKEN: $token" -H "X-SLURM-USER-NAME: slurm" Authentication failure slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[172.17.0.1]:38090] attempting user_name slurm token authentication pass through slurmrestd: error: operations_router: [[172.17.0.1]:38090] authentication failed: Unspecified error slurmrestd: debug2: _on_message_complete_request: [[172.17.0.1]:38090] on_http_request rejected: Unspecified error ``` But! I have found that I'm setting `SLURM_JWT` environment variable for process, whatever token value is authenticated normally ``` # start process with SLURM_JWT [root@slurmctl 1235]# cd /proc/2108/ [root@slurmctl 2108]# cat environ |tr "\0" "\n" TERM=xterm TINI_VERSION=v0.18.0 SHLVL=1 HOSTNAME=slurmctl SLURM_JWT=randomtoken SUPERVISOR_ENABLED=1 SUPERVISOR_PROCESS_NAME=slurmrestd PWD=/ SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock SUPERVISOR_GROUP_NAME=slurmrestd PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin HOME=/root SLURMRESTD_SECURITY=disable_user_check _=/usr/bin/supervisord # request OK [root@slurmctl 2108]# curl [ http://172.17.0.4:19090/slurm/v0.0.35/jobs | 172.17.0.4:19090/slurm/v0.0.35/jobs ] -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: slurm" [ ] [root@slurmctl 2108]# curl [ http://172.17.0.4:19090/slurm/v0.0.35/jobs | 172.17.0.4:19090/slurm/v0.0.35/jobs ] -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: errorvalue" [ ] ``` So, I'm confused about JWT authentication. Q1: What is used for the `SLURM_JWT` environment variable, is it required for JWT? Related search from github source repo: [ https://github.com/SchedMD/slurm/search?q=SLURM_JWT | https://github.com/SchedMD/slurm/search?q=SLURM_JWT ] Q2: How to use slurmrestd JWT authentication? -------- Thanks, Chenyang Yan
