I am seeing occasional problems where ansible-playbook fails during python interpreter discovery, and would appreciate any and all advice. The only diagnostics I can find in ansible debug output is what appears to be a SIGPIPE return code.
Background: I am running rocky8, (Rocky Linux release 8.4 (Green Obsidian)), and using ansible-7.2.0-1.el8.noarch, ansible-core-2.14.2-3.el8.x86_64 with python 3.11.2 on one node and deploying to many other nodes also running rocky8 with python 3.6 installed. I have an application the is installing and configuring software on all the nodes using ansible playbooks. Usually this runs without a problem but occasionally I get a failures in a playbook when it is attempting python interpreter discovery. The playbooks are run multiple times on each node to install/configure different services so the falling playbook is executed 1 or more times before an execution instance fails. If I execute the playbook again after the failure it runs ok. I verified that /usr/bin/python3.6, /usr/bin/python3 and /usr/libexec/platform-python all exist on each node before I start the application Debug output: This is the output, with -vvvvv, when one of the playbooks fails. Attempting python interpreter discovery <10.42.3.199> ESTABLISH SSH CONNECTION FOR USER: root <10.42.3.199> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s) <10.42.3.199> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no) <10.42.3.199> SSH: ANSIBLE_REMOTE_PORT/remote_port/ansible_port set: (-o)(Port=22) <10.42.3.199> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/root/.ssh/id_rsa") <10.42.3.199> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no) <10.42.3.199> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="root") <10.42.3.199> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10) <10.42.3.199> SSH: Set ssh_common_args: () <10.42.3.199> SSH: Set ssh_extra_args: () <10.42.3.199> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/root/.ansible/cp/1a125e8154") <10.42.3.199> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/root/.ssh/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/1a125e8154"' 10.42.3.199 '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'python3.10'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.9'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.8'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"'' <10.42.3.199> (-13, b'', b"OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021 My reading of the ansible source code indicates that the output of the last line is (rcode=-13, stdout, stderr). The return code is from the subprocess.Popen call in the run_cmd. A return of a negative number indicates a the process was terminated by a signal: "13 SIGPIPE Terminate Write to pipe with no readers". Here is the output from a successful run of the playbook with the same arguments: Attempting python interpreter discovery <10.42.3.199> ESTABLISH SSH CONNECTION FOR USER: root <10.42.3.199> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s) <10.42.3.199> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no) <10.42.3.199> SSH: ANSIBLE_REMOTE_PORT/remote_port/ansible_port set: (-o)(Port=22) <10.42.3.199> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/root/.ssh/id_rsa") <10.42.3.199> SSH: ansible_password/ansible_ssh_password not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no) <10.42.3.199> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="root") <10.42.3.199> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10) <10.42.3.199> SSH: Set ssh_common_args: () <10.42.3.199> SSH: Set ssh_extra_args: () <10.42.3.199> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/root/.ansible/cp/7dfa49af55") <10.42.3.199> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/root/.ssh/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/1a125e8154"' 10.42.3.199 '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'python3.10'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.9'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.8'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"'' <10.42.3.199> (0, b'PLATFORM\nLinux\nFOUND\n/usr/bin/python3.6\n/usr/bin/python3\n/usr/libexec/platform-python\nENDFOUND\n', b"OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021 -- You received this message because you are subscribed to the Google Groups "Ansible Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/fa6cd9cb-efc8-4bf3-8cd1-8e0c07a4a025n%40googlegroups.com.
