oss-sec mailing list archives
Re: "sshole": password authentication bypass: OpenSSH + musl libc
From: Solar Designer <solar () openwall com>
Date: Mon, 16 Nov 2026 08:34:43 +0100
Hi, I'm sorry I'm late to comment in this thread, but I feel a few aspects are still misunderstood by a lot of people here. On Thu, Nov 05, 2026 at 09:14:35AM -0500, Jan Schaumann wrote:
OpenSSH just published OpenSSH 10.6p2 with a critical fix for a password authentication bypass on systems that use musl libc and have PAM disabled. Alpine Linux has been called out as one such affected system: https://www.openssh.org/releasenotes.html#10.6p2* sshd(8): avoid password authentication bypass that is exploitable through sshd(8) that affects some Linux systems when built and run with PAM support disabled. We consider the root cause of this bug to be outside sshd, but it is exploitable via sshd and can be avoided there. This bug was reported by Özgün Kültekin of Wakefield (wakefield.co)The vulnerability itself is described here: https://sshhole.com/ (The history of how we got here is actually fun.) That is, for any account that has an invalid password hash field of '*', you can log in by entering a password >257 characters, which will cause crypt(3) in musl libc to return '*'.
Kind of, and that's indeed embarrassing, but it misses important detail. musl's crypt(3) only returns "*" when the setting (aka salt) string passed to it is not "*", so it reasonably expected the caller doing the typical strcmp() would see no match. What OpenSSH did was authenticate one user while taking the password hash field from another user. That could be a reasonable way to mitigate timing leaks as long as authentication result is discarded. The problem is it wasn't. Such fake yet real authentication against another user's password hash was unexpected by musl developers. It was well understood before that blindly returning "*" on error would be a vulnerability, which would not have been limited to OpenSSH. When the passed setting string is "*", musl returns "x" instead. My original design in crypt_blowfish used two different failure tokens, "*0" and "*1". This is what libxcrypt inherited, unless it's configured with --disable-failure-tokens, in which case it returns NULL. We're lucky that "*0" isn't normally found in the password hash field... unless a password changing program would obtain this error return from crypt(3) and blindly put it in there. Even then, normally authentication would fail because a second call with that as the setting would return "*1"... unless it's by OpenSSH prior to the recent fix. This incident is a reminder that we should probably bite the bullet and start returning NULL. There should be way fewer programs unprepared to handle NULL now than there were back when we chose to use failure tokens instead. Some distros that use libxcrypt already --disable-failure-tokens for quite a while. musl is likely going to start returning NULL in a future release, but for now will likely move to the same "*0" and "*1" failure tokens that libxcrypt uses. Besides the risk of crashes on NULL dereference, I also worry that in some wrappers and scripting languages NULL may be mapped to an empty string or (more reasonably) to a non-object. I am seeing the latter when I call crypt() from Perl on Rocky Linux 9 and 10. If I check with defined(), it is not. But if I blindly use it as a string, it's empty. If this happens when setting a new password, the empty string could be blindly put into a password hash field. That could be later interpreted as the user having no password (the traditional Unix behavior for empty password hash field) or if it's passed into Perl's crypt() as the salt then we'll get a non-object again, which may behave like an empty string again and the two may match. That would mean successful authentication with no/empty or with any password, regardless of the intended password. I suspect this sort of problem is not limited to Perl. It may also not be limited to treatment of non-objects as empty strings - they could also be stored as-is and then passed back into crypt() and used in a comparison, possibly resulting in a match between two non-objects. Overall, I am not sure NULL is actually safer than "*0" and "*1".
The code fix (checking for MIN_PASSWORD_HASH_LEN instead of using musl crypt(3)'s default "*") is here: https://github.com/openssh/openssh-portable/commit/4353fdc43ef646c43a8dda7025cc1a898caf1482 Setting `PasswordAuthentication no` in /etc/ssh/sshd_config mitigates the problem (at the obvious cost of not being able to do password auth).
Yes. As others also pointed out elsewhere, OpenSSH should also make sure it always fails authentication (regardless of password hash length) when it performs fake authentication as timing leak defense. On Thu, Nov 05, 2026 at 11:10:52PM +0000, lists () notatla org uk wrote:
If an error string is output does not : or multiple :: better express the aim of never seeing this as field content (because it is field separator) ?
I guess the idea is that crypt(3) could return ":" on error to ensure no match with anything that came from a passwd format file, where that's the field separator. Yes, but that would be differently dangerous: if a password changing program blindly puts that error string into a field in such file, that would look like an empty field on subsequent reads, and further fields would be shifted. A constant ":" return (not two different failure tokens) could also match a previously returned value like that if the storage format isn't a passwd format file (e.g., LDAP or SQL database). OpenBSD bcrypt used to return ":" on error until 2015, but switched to returning NULL since. As I recall, OpenBSD's own passwd(1) program was safe from this, but I wouldn't be surprised if the ":" return got copied into other projects where it isn't necessarily safe. Finally, I include below two postings by Rich Felker to the musl list. Alexander https://www.openwall.com/lists/musl/2026/11/09/5 On Mon, Nov 09, 2026 at 08:56:50AM -0500, Rich Felker wrote:
The correct usage of crypt for password authentication is asserting
that, if P is the candidate password and H is the account password
"hash", strcmp(crypt(P,H),H)==0. Allowing the user in without checking
that is potentially a password-authentication-bypass bug.
I put "hash" in quotation marks because it might not be an actual
hash; it can be a string chosen by the tooling or a user manually
editing the passwd/shadow files to produce something unmatchable. So
an application should not be making assumptions about its form.
Through at least 1.2.6, musl uses failure tokens "*" and "x" for cases
where the password should never match. These include unsupported
password length (for hash types that need to restrict it) and
unsupported or invalid setting/salt strings.
It isn't decided yet, but 1.2.7 will probably switch to using failure
tokens matching libxcrypt's ("*0" and "*1") which are slightly safer
against some forms of misuse by the caller. Future versions may drop
failure tokens entirely and switch to treating these conditions as
errors (returning null).
I would not document future behavior, and would document existing
behavior only as an implementation detail, not an application-facing
contract. The only contract with the application is that crypt(P,H)
will never return a string identical to H except on valid password (in
the absence of theoretical hash collisions) regardless of the validity
of P and H.
https://www.openwall.com/lists/musl/2026/11/06/7 On Fri, Nov 06, 2026 at 01:54:55PM -0500, Rich Felker wrote:
On Thu, Nov 05, 2026 at 10:07:13AM -0500, Rich Felker wrote:Versions of OpenSSH prior to 10.6p2 using musl are affected by an authentication bypass when UsePAM=no and PasswordAuthentication=yes. Anyone who cannot immediately upgrade should edit /etc/ssh/sshd_config to set: PasswordAuthentication no and restart sshd after ensuring you have working pubkey access. Dropbear and other non-OpenSSH software using password-based logins is not affected. It is my position that this is not a bug in musl just an erroneous use by the caller. But the mechanism by which this happened is something we have been wanting to change for a while now, and it very likely will be changed in the upcoming release cycle. Details to follow soon.OK, some details on the mechanism here, starting with backroung and moving step-by-step: 1. The way you validate a password against a "hash" (which is actually a combination of a 'setting' prefix and hashed password) is comparing the result of crypt(password,hash) against hash. Only if they compare equal should the user be granted access. 2. Mechanically, any character or sequence of characters that do not form a valid hash (and that do not contain : or \n) can be placed in the password field of the shadow file (or legacy passwd file) to ensure that it's impossible to login as that user with a password, since for such a hash, crype(pass,hash)!=hash is always true. The conventional "no password access" value is "*". 3. In order to avoid leaking information about which user accounts exist and have password-based login available, OpenSSH has a clever hack to, when the user's hashed password is a single character like "*" that can never be matched, use a different user's password hash as the setting/salt. Originally they used root's: https://github.com/openssh/openssh-portable/commit/9286875a73b2de7736b5e50692739d314cd8d9dc and later switched to searching the user database until finding a user with a password: https://github.com/openssh/openssh-portable/commit/dbf788b4d9d9490a5fff08a7b09888272bb10fcc Then, after obtaining the other_user_hash, OpenSSH checks the condition: crypt(pass,other_user_hash) == hash and, despite having already determined that password logins are disabled for this user, allows the login the proceed if the condition is met. 4. There is a historical disagreement on how crypt() should handle exceptional conditions. Pre-POSIX, crypt was not permitted to return a null pointer, and a lot of software did (and some apparently still does) assume this. POSIX does not define any error conditions except ENOSYS (which was probably about export controls), so defining a behavior when the operation cannot be completed with the provided parameters is entirely up to the implementation. Some implementations -- mainly OpenBSD, FreeBSD, and recent Fedora -- return a null pointer and report the condition in errno. A number of implementations including NetBSD, musl, and most glibc-based Linux distros (now using libxcrypt) return what's come to be known as a "failure token" -- a string that guarantees crypt(pass,failure_token) != failure_token for any possible password pass. The failure token musl uses is "*", except when the input salt/setting is "*"; in that case, it returns "x", so that: crypt(pass,"*") != "*" is satisfied to prevent any password from validating. Other implementations commonly use "*0" and "*1" for these roles. 5. Having a failure token not match any password is contingent on the caller of crypt passing the correct hash as the settign argument to crypt. While it's guaranteed that: crypt(pass,failure_token) != failure_token it is impossible to guarantee that: crypt(pass,some_other_hash) != failure_token But that is exactly what OpenSSH was testing, by passing a hash obtained from searching for another user account with a password. 6. While many password hash algorithms always succeed as long as the setting string is not invalid, SHA256 and SHA512 -- not the base hash algorithms themselves, but the way they're used in multiple rounds as password hashes -- have quadratic time in password length, and thus require treating excessively long passwords as an error condition to avoid DoS in any process that validates passwords. So, for an excessively long password pass, and some_other_hash taken from a user who has a SHA256 or SHA512 hashed password: crypt(pass,some_other_hash) == failure_token 7. For any choice of failure_token, if it ends up in the password field of the shadow file, it will allow password bypass for that user, because: crypt(pass,some_other_hash) == failure_token == user_hash Since by convention (what's implemented by shadow tooling, etc.) "*" is used for disabling password access to accounts and is also the failure token musl uses, this condition was met. Mitigation, fix, and hardening: As noted in the advisory, disabling password authentication entirely mitigates the issue. There are a number of other configurations that are also unaffected: In order to be affected, the target account must have "*" as its password hash (attempting to prevent password access), there must be at least one other account that does have a password set, and the first account found with a password (in order of enumerating /etc/passwd) must be using SHA256 or SHA512 type password hashes. OpenSSH fixed the bug by hard-coding failure after doing the timing-masking crypt call with the wrong hash, so that if the target user's password hash is shorter than 4 character (not a valid hash; must be a failure token) the login attempt always fails: https://github.com/openssh/openssh-portable/commit/4353fdc43ef646c43a8dda7025cc1a898caf1482 This should fix the issue for any system where failure_token is under 4 bytes (all real world systems). As for hardening musl against this kind of issue, pursuing a switch to returning an error (null+errno) has been an open issue for a long time now, and one I'd like to assess in the next release cycle. In the immediate, I am going to propose a patch switching to the more common "*0"/"*1" convention for failure tokens. This does not prevent misuse, but it's safer under the assumption that the conventional "*" is what appears in shadow. This should be able to go into the pending release and would be easy for distros to apply as a backport if desired.
Current thread:
- Re: "sshole": password authentication bypass: OpenSSH + musl libc, (continued)
- Re: "sshole": password authentication bypass: OpenSSH + musl libc Loganaden Velvindron (Nov 06)
- Re: "sshole": password authentication bypass: OpenSSH + musl libc Michael Orlitzky (Nov 06)
- Message not available
- Message not available
- Re: "sshole": password authentication bypass: OpenSSH + musl libc theknell (Nov 06)
- Re: "sshole": password authentication bypass: OpenSSH + musl libc Richard Kettlewell (Nov 07)
- Re: "sshole": password authentication bypass: OpenSSH + musl libc Loganaden Velvindron (Nov 06)
- Re: "sshole": password authentication bypass: OpenSSH + musl libc Peter Gutmann (Nov 06)
- Re: "sshole": password authentication bypass: OpenSSH + musl libc Matt Johnston (Nov 06)
- Message not available
- Re: "sshole": password authentication bypass: OpenSSH + musl libc theknell (Nov 07)
