On Fri, 18 Apr 2025, Colin Watson wrote:
Ugh, I didn't realize your password would show up in the backtrace! Sorry about that - please change it as soon as possible.
Yeah, I caught that after I sent the message and already have. Ah well, it was overdue for rotation anyway.
valid. Therefore I think we must be dealing with action at a distance from some previous memory corruption, which is going to be a pain to track down. It might be in openssh-server, and the timing suggests that it probably is; but it might also be in any other PAM module used in the auth phase.
Before I continue, I just remembered another issue (possibly PAM-related) which had come up irregularly enough to forget about, but may be smoke here.
Every so often, logging in normally behaves but also spits out:"When trying to update a password, this return status indicates that the value provided as the current password is not correct."
which is some sort of Samba error. Maybe there's an interaction here. Now as to your new instructions:
Now try logging in again until you hit a crash, and then look in "sudo journalctl -u ssh.service | less" for the output of valgrind; each instance of its output will start with a line saying "Memcheck, a memory error detector", and each line will have "==PID==" in it for some process ID. I don't think the output is likely to include your password this time, but it will probably be worth checking it over just in case.
Typical such output attached.
Separately, it might also be helpful for me to have a copy of your /etc/pam.d/common-auth file, so I can see which other modules are being run
Attached. It should be the Sid default, modulo anything frobbing it during updates.
# # /etc/pam.d/common-auth - authentication settings common to all services # # This file is included from other service-specific PAM config files, # and should contain a list of the authentication modules that define # the central authentication scheme for use on the system # (e.g., /etc/shadow, LDAP, Kerberos, etc.). The default is to use the # traditional Unix authentication mechanisms. # # As of pam 1.0.1-6, this file is managed by pam-auth-update by default. # To take advantage of this, it is recommended that you configure any # local modules either before or after the default block, and use # pam-auth-update to manage selection of other modules. See # pam-auth-update(8) for details. # here are the per-package modules (the "Primary" block) auth [success=2 default=ignore] pam_unix.so nullok auth [success=1 default=ignore] pam_winbind.so krb5_auth krb5_ccache_type=FILE cached_login try_first_pass # here's the fallback if no module succeeds auth requisite pam_deny.so # prime the stack with a positive return value if there isn't one already; # this avoids us returning an error just because nothing sets a success code # since the modules above will each just jump around auth required pam_permit.so # and here are more per-package modules (the "Additional" block) auth required pam_ecryptfs.so unwrap # end of pam-auth-update config
Apr 19 03:16:56 solid-state-entity sshd[252834]: ==252834== Memcheck, a memory error detector Apr 19 03:16:56 solid-state-entity sshd[252834]: ==252834== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al. Apr 19 03:16:56 solid-state-entity sshd[252834]: ==252834== Using Valgrind-3.24.0 and LibVEX; rerun with -h for copyright info Apr 19 03:16:56 solid-state-entity sshd[252834]: ==252834== Command: /usr/lib/openssh/sshd-session -D -R Apr 19 03:16:56 solid-state-entity sshd[252834]: ==252834== Apr 19 03:16:59 solid-state-entity unix_chkpwd[252836]: password check failed for user (frink) Apr 19 03:17:00 solid-state-entity sshd-session[252834]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.90 user=frink Apr 19 03:17:00 solid-state-entity sshd-session[252834]: pam_winbind(sshd:auth): getting password (0x00000388) Apr 19 03:17:00 solid-state-entity sshd-session[252834]: pam_winbind(sshd:auth): pam_get_item returned a password Apr 19 03:17:00 solid-state-entity sshd-session[252834]: pam_winbind(sshd:auth): user 'frink' granted access Apr 19 03:17:00 solid-state-entity sshd-session[252834]: pam_unix(sshd:account): could not identify user (from getpwnam(SSE\frink)) Apr 19 03:17:00 solid-state-entity sshd-session[252834]: Failed password for frink from 192.168.1.90 port 52056 ssh2 Apr 19 03:17:00 solid-state-entity sshd-session[252834]: fatal: Access denied for user frink by PAM account configuration [preauth] Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== HEAP SUMMARY: Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== in use at exit: 478,256 bytes in 5,205 blocks Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== total heap usage: 8,993 allocs, 3,788 frees, 1,524,157 bytes allocated Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== LEAK SUMMARY: Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== definitely lost: 256 bytes in 3 blocks Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== indirectly lost: 5,329 bytes in 17 blocks Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== possibly lost: 0 bytes in 0 blocks Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== still reachable: 446,251 bytes in 5,182 blocks Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== suppressed: 26,420 bytes in 3 blocks Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== Rerun with --leak-check=full to see details of leaked memory Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== For lists of detected and suppressed errors, rerun with: -s Apr 19 03:17:00 solid-state-entity sshd[252834]: ==252834== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
