Charting the SSH Multiverse

Charting the SSH Multiverse

Speaker: HD Moore

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: https://www.youtube.com/watch?v=Gm_X52BU-hw

Reading time: 8 minutes

TL;DR

SSH was once considered a solved problem, but since 2023 it has become a hotbed of novel vulnerabilities, backdoors, and implementation quirks spanning dozens of incompatible forks and third-party libraries. HD Moore, co-founder and CEO of RunZero and the creator of Metasploit, presented a comprehensive survey of the SSH ecosystem's fragmentation — and unveiled Shambl, an open-source SSH enumeration tool designed to systematically find the bugs hiding in non-standard implementations.

Introduction

For much of its thirty-year history, SSH occupied a comfortable corner of the security landscape: patched, well-understood, and rarely exciting. Public-key authentication was considered a reliable backstop, and OpenSSH's dominance meant that most systems ran essentially the same codebase. Then, between 2023 and 2025, the protocol exploded: a state-actor-linked backdoor nearly shipped to millions of Linux systems, a remote root code execution vulnerability surfaced in OpenSSH's default configuration for the first time in two decades, and a cascade of implementation flaws emerged from the long tail of obscure SSH libraries embedded in enterprise products.

Moore, returning to BSidesSF for the first time since 2011, brought with him eight months of systematic research into the SSH ecosystem's fragmentation. His core argument: the SSH "multiverse" — the dozens of independent implementations and forks that collectively outnumber genuine OpenSSH deployments — is riddled with bugs that neither the OpenSSH security model nor conventional scanning tools are designed to find.

▶ Watch: Introduction and SSH Background (00:00)

How Much Information Does SSH Leak Before Authentication?

One of the talk's most striking sections addressed how much an attacker — or a defender — can learn from an SSH server before ever authenticating. Moore laid out a progression of information sources available in the pre-authentication phase:

TCP/IP stack fingerprinting: Even before an SSH banner is transmitted, the TCP window size and scaling factors reveal the underlying kernel version. Combining the SSH banner (e.g., "OpenSSH 7.4") with the MSS scaling value from the TCP handshake can narrow down the exact Linux distribution — sometimes to a single candidate — with a single packet and a single connection.

Distribution-specific package versions: Debian and Ubuntu embed the specific package version in the SSH banner string. An observer can determine not only what distribution a server runs, but whether its SSH package is currently patched, without sending a single authentication attempt.

Key exchange fingerprinting: The specific combination of key exchange algorithms, ciphers, and MAC algorithms a server proposes during the KEX_INIT phase constitutes a fingerprint unique enough to identify vendor implementations. Cisco IOS, Juniper, and various embedded device SSH stacks all advertise characteristic algorithm sets.

Public key probing without a private key: SSH's two-phase public-key authentication process — "will you accept this key?" followed by "here's proof I own it" — allows an attacker to test whether a server accepts a given public key without possessing the corresponding private key. Given a known bad actor's public key (Moore cited the Jia Tan persona as an example), an attacker could mass-scan the entire internet in roughly four hours to identify every server that actor has access to.

▶ Watch: Pre-authentication Information Disclosure (02:00)

The SSH Ecosystem: A Fragmented Multiverse

Moore traced the lineage of SSH implementations from Tatu Ylonen's original 1995 freeware through the OpenBSD-led fork that became OpenSSH Portable, the proprietary Tectia/SSH Communications Security branch, and the proliferating family of embedded libraries. The ecosystem today includes standalone implementations like WolfSSH, Cisco IOS SSH, LSSH, and PKIX, along with widely embedded libraries: LibSSH, Go's x/crypto/ssh, Python's Paramiko, Java's Apache Mina, and Erlang's built-in SSH module.

None of these implementations are identical. Moore described spending eight months diffing every SSH implementation he could find, discovering that even implementations that nominally derive from OpenSSH have diverged in security-critical ways. PKIX, for instance, frequently inverts conditional logic when copying security fixes from OpenSSH — getting the sense of the check wrong while still applying a superficially similar change. No exploitable vulnerabilities have yet emerged from those inversions, but Moore was not optimistic about their long-term safety.

The most damning example was Microsoft's implementation. When Windows finally shipped OpenSSH by default, it introduced a subtle but critical modification: it made the most important cryptographic comparison function — used for secure key comparison — no longer timing-safe, and introduced an out-of-bounds memory read, all in the name of carriage-return/line-feed compatibility. The modification effectively backdoored the most security-sensitive function in the codebase. Moore and colleagues presented the finding at a prior conference, and Microsoft has since addressed it.

▶ Watch: SSH Implementation Fragmentation (08:00)

2023–2025: An Explosion of SSH Vulnerabilities

Moore catalogued the cascade of high-profile SSH vulnerabilities that transformed the protocol from "boring" to "critical" in under two years:

The Terrapin attack (2023): Fabian Bäumer at Ruhr University Bochum demonstrated a protocol-layer attack that could truncate packets on the client side, exploit a cipher negotiation edge case, and effectively downgrade the connection to a weaker cipher suite than the parties intended.

The XZ/liblzma backdoor (2024): The Jia Tan persona — now widely attributed to a state actor — spent years building trust as an open-source contributor before embedding a sophisticated backdoor into the XZ compression library, which is linked by systemd and, on Debian/Ubuntu/Fedora systems, indirectly by OpenSSH. The backdoor was keyed to the attacker's private key and would have given them silent remote access to any affected system. It was caught at the last minute by Andres Freund before reaching stable release channels.

RegreSSHion (CVE-2024-6387): A Qualys-discovered race condition in OpenSSH's signal handling code that enabled unauthenticated remote root code execution in the default configuration — the first such vulnerability in OpenSSH in roughly twenty years.

MOVEit SSH vulnerability: MOVEit's custom SSH implementation (licensed from IPWorks) contained a bug where a filename embedded in the bytes of a public key during the handshake would cause the server to read and authenticate against that file. WatchTower researchers turned it into full remote code execution by having the server parse a specially crafted log file containing an embedded key in an unusual format.

▶ Watch: Major SSH Vulnerabilities of 2023–2024 (14:00)

Shambl: A Custom SSH Protocol Stack for Finding Hidden Bugs

To systematically probe the SSH multiverse, Moore and the RunZero team built Shambl — a custom SSH protocol stack implemented in Go that can step through any state of the SSH state machine and inject arbitrary protocol messages at each stage. The tool is available as a single Go binary deployable on any operating system.

Shambl's capabilities include:

• Auth bypass probing: Testing whether a server permits channel opens, shell requests, or port forwards before completing authentication.
• Public key enumeration: Spraying millions of public keys at a server to determine which identities have access, without possessing any corresponding private keys. Moore demonstrated this against Dropbear servers using GitHub's database of 117 million user public keys.
• Bad key detection via badkeys.info: Integration with Hanno Böck's badkeys project, which maintains a database of known-leaked SSH keys — including private keys extracted from the Fortinet configuration dump leaked to the dark web — allowing instant identification of compromised host keys.
• Blind exec detection: Shambl identified a vulnerability in Erlang's built-in sshd (CVE-2025-32433) where sending a channel-open followed immediately by an exec request — without waiting for the channel-open reply — caused the server to execute the command with no authentication. The bug is subtle because the server sends no reply, making it look identical to normal behavior on many legitimate implementations.

▶ Watch: Shambl Tool and Recent Findings (18:00)

The Erlang blind exec bug (pre-authentication remote code execution affecting Erlang/OTP releases before 27.3.3) illustrates the difficulty of automated testing: the server simply doesn't respond, making it indistinguishable from dozens of other implementations — including Cisco IOS — that also ignore the exec request without actually running anything. Shambl resolves the ambiguity by checking whether the output stream subsequently contains command output.

Moore also described RunZero's contribution of a unified TLS/SSH cryptography library built on Go tip that speaks SSL3 through TLS 1.3, post-quantum algorithms, and all SSH cipher suites from a single code path — with a permissive ASN parser sourced from the Zmap project that can parse malformed certificates rather than failing silently. The library is available at xcrypto and is being proposed as a dependency for tools like Nuclei.

Notable Quotes

"Until about 2023, SSH was pretty boring. And then last year, everything exploded. And this year it's somehow even more exciting." — ▶ 00:00

"They managed to screw up literally the most important function in the entire code base... for carriage return line feed compatibility. For a crypto comparison, they made it no longer timing safe." — ▶ 12:00

"You can take Shambl and it's a binary that runs on any OS. You can just throw it at any network you want, and it just randomly rains shells out." — ▶ 26:01

Key Takeaways

• SSH leaks extensive information before authentication — kernel versions, distribution patch levels, and vendor identity are all recoverable from the banner, TCP fingerprint, and key exchange negotiation alone.
• The SSH ecosystem is deeply fragmented: beyond OpenSSH, dozens of independent implementations and forks carry implementation bugs that OpenSSH does not have, and many have never been systematically audited.
• Three vulnerability classes dominate SSH in the wild: hard-coded or reused host keys, authentication bypass flaws, and pre-authentication port forwarding — all detectable with the right tooling.
• Shambl is a practical blue-team tool: organizations can use it to verify that decommissioned SSH keys are no longer accepted on any internal host, audit for pre-auth port forwarding vulnerabilities, and check host keys against the badkeys.info leaked-key database.
• Post-quantum SSH is coming: Shambl's next release will add post-quantum algorithm support via a unified Go cryptography library, giving security teams early visibility into which implementations support the upcoming NIST-standardized algorithms.