Exploiting the not so misuse-resistant AES-GCM API of OpenSSL

Exploiting the Not-So-Misuse-Resistant AES-GCM API of OpenSSL

Speaker: Félix Charette

Conference: NorthSec 2025 — May 15–16, 2025, Marché Bonsecours, Montreal

Watch on YouTube: https://www.youtube.com/watch?v=MB4H2r7AebA

Reading time: ~7 minutes

TL;DR

AES-GCM is theoretically sound, but OpenSSL's bindings for Ruby and PHP contain a well-documented — yet widely overlooked — flaw: neither language's standard decrypt function validates that the authentication tag is the correct length. A one-byte tag is accepted as readily as a full sixteen-byte tag. This reduces the integrity check from a cryptographically strong guarantee to a brute-force target of 256 guesses, enabling ciphertext forgery and, under certain conditions, full plaintext recovery via a format-validity oracle attack. Félix Charette walks through how to identify vulnerable codepaths, craft shortened-tag payloads, flip ciphertext bits under CTR mode, and decrypt unknown values by observing parsing behaviour.

Introduction

Cryptographic libraries exist to make secure implementations accessible to developers who are not cryptographers. The implicit promise is that using the library correctly — following its documented API — will produce a secure result. But "correct usage" is rarely self-evident, and APIs that fail to enforce their own invariants silently degrade the security guarantee they are supposed to provide.

AES-GCM is an Authenticated Encryption with Associated Data (AEAD) scheme. Its authentication tag is the mechanism that guarantees ciphertext integrity: any tampering with the ciphertext should cause tag verification to fail, preventing decryption. The tag is only as strong as the guarantee that it is checked in full.

Charette's talk opens by revealing that this guarantee is not enforced in the OpenSSL bindings shipped with Ruby and PHP. The official documentation for both languages explicitly warns that tag length is not verified by the decrypt function — but warnings in documentation are not the same as enforcement in code. In practice, developers copy-paste working code, pass whatever bytes they have as the tag, and the decrypt call succeeds. The door to ciphertext manipulation is left wide open.

▶ Watch: AES-GCM fundamentals and the authentication tag (0:00)

AES-GCM Internals and Where the Weakness Lives

AES-GCM combines two operations. Counter Mode (CTR) converts the AES block cipher into a stream cipher: a nonce and counter are encrypted with the secret key to produce a keystream, which is then XORed with the plaintext to yield ciphertext. Because XOR is its own inverse, decryption is identical to encryption. This also means that any bit flipped in the ciphertext causes the same bit to flip in the plaintext — a property that CTR mode attacks exploit directly.

The second component is GHASH, a Galois field hash function that takes the ciphertext and any additional authenticated data, derives a subkey from the encryption key, and produces a 16-byte authentication tag. The tag is recomputed on the receiving end before decryption occurs. If the recomputed tag does not match the received tag, decryption should be aborted.

The vulnerability is not in the algorithm. It is in the API contract. OpenSSL's low-level decrypt function accepts whatever bytes the caller passes as the tag and compares only the bytes it receives — it does not check that the tag is the required 16 bytes. Ruby and PHP surface this as-is. As Charette demonstrates with a short PHP snippet, encrypting the string "HackThePlanet" produces a 16-byte tag, but passing only the first byte of that tag to the decrypt function successfully decrypts the message. With a one-byte tag, an attacker needs at most 256 attempts to find a matching tag for any modified ciphertext.

▶ Watch: The misuse resistance problem and the one-byte tag demo (4:00)

Identifying Vulnerable Endpoints

Charette provides a systematic method for identifying targets in real applications. The starting point is any endpoint that handles encrypted data using OpenSSL AES-GCM: session cookies, encrypted request parameters, authorization tokens, or any binary blob passed between client and server.

Once a candidate ciphertext is identified, the next step is understanding its serialization format. Charette describes three common patterns. The first is JSON with distinct fields for the nonce, tag, and ciphertext — the easiest case, because the tag field can simply be replaced with a shorter value. The second is a delimiter-separated string (pipes, commas), where the same substitution applies. The third is raw concatenation, where the nonce, ciphertext, and tag are packed end-to-end and length assumptions are baked into the parsing logic.

The concatenation format introduces an additional exploitation surface. If the parser extracts fields using substring operations that derive offsets from the total length, supplying a shorter-than-expected total input can cause the extracted "tag" to overlap with, or be entirely contained within, the ciphertext region. Charette walks through Ruby and PHP substring implementations to show how inputs shorter than nonce_length + 1 + tag_length bytes produce misaligned extractions that result in a one-byte effective tag without the attacker needing to explicitly control the tag field at all.

▶ Watch: Serialization formats and identifying the tag length check (6:00)

Ciphertext Forgery via CTR Bit-Flipping

With the authentication tag reduced to one byte, an attacker can treat the ciphertext as if it were protected only by CTR mode — which provides confidentiality but zero integrity. The classical CTR attack proceeds as follows.

The attacker identifies a position in the plaintext they wish to modify — for example, a field in a JSON session token. If the attacker knows the current plaintext value at that position (because it is predictable, reflected in a response, or controlled via input), they XOR the corresponding ciphertext bytes with the known plaintext and then XOR again with the desired replacement value. The resulting modified ciphertext, when decrypted, produces the attacker's chosen plaintext at the target position.

The only remaining obstacle is the authentication tag. With a one-byte tag, the attacker submits the forged ciphertext with each of the 256 possible tag values in turn. On a web application, the oracle is the server's response: a valid decryption returns a normal application response, while an invalid tag returns an error. On average, the correct tag is found in 128 requests. For a session-cookie forgery attack, this means escalating from an unauthenticated visitor to an authenticated user in under 256 HTTP requests.

Charette notes that the attack's effectiveness depends on knowing at least a portion of the plaintext. When the plaintext structure is known (JSON with predictable key names, for example), identifying target positions is straightforward even without full knowledge of the values.

▶ Watch: CTR bit-flipping and tag brute-force forgery (12:00)

Plaintext Recovery via Format Validity Oracle

The format validity oracle extends the attack to plaintext recovery — decrypting unknown values without the key. The technique is analogous to a padding oracle attack but depends on application-layer parsing behaviour rather than cryptographic padding.

The principle is that the application decrypts the ciphertext and then parses the result. If the parser is strict — for example, if it expects valid JSON and throws an exception when the result is malformed — the attacker can observe whether a given manipulation produced valid or invalid output. This gives one bit of information per query about the plaintext at the target position.

Charette walks through an example with a JSON-encrypted cookie containing the value user: Alice. By XORing a byte at a target position with specific values and submitting the result with each of 256 possible tags, the attacker can determine whether the decrypted byte falls inside or outside a set of valid JSON characters (printable ASCII, no control characters, no unescaped quotes). Iterating across positions and narrowing the character set with each round, the attacker progressively recovers the plaintext.

The technique generalises to other structured formats: URL query strings, compressed data (which carry CRC checksums that serve as built-in validity signals), XML, and any other format with predictable structural constraints. The only requirement is that the application behaves observably differently — in response code, response body, or timing — when decryption produces valid versus invalid structured data.

▶ Watch: Format validity oracle and plaintext recovery (14:00)

Notable Quotes

"OpenSSL has implemented those functions, and Ruby and PHP decided to use those instead of developing their own. The documentation says 'the length of the tag is not checked by the function.' What does that mean?"

"We take that tag, we take the first byte only, that's what we send, and sure enough, it prints HackThePlanet. So we've decrypted successfully our ciphertext without the integrity check — or at least it's only one byte. It's two hundred and fifty-six requests, and we are sure to get the tag."

"Imagine a web server — you're sending your cookie, it says 'you're not authenticated.' You try another tag, 'you're not authenticated.' At some point it's gonna say, 'hey, that's a valid cookie.' That's how you brute force the tag."

Key Takeaways

• AES-GCM is only as strong as its tag enforcement. Accepting a tag of arbitrary length neutralizes the integrity guarantee; a one-byte tag is brute-forceable in at most 256 attempts.
• Both Ruby and PHP OpenSSL bindings are affected. The flaw is documented but not enforced — developers are expected to add their own length check, and many do not.
• The correct fix is a strict length check before decryption. Verifying that the received tag is exactly 16 bytes before passing it to OpenSSL eliminates the vulnerability entirely.
• Serialization format matters for exploitability. JSON and delimiter-separated formats expose the tag field directly; concatenation-based formats may still be vulnerable if substring parsing can be manipulated with undersized inputs.
• Format validity oracles enable key-free plaintext recovery. Any application that parses decrypted output with observable error behaviour can be used to recover plaintext byte-by-byte without knowledge of the encryption key.
• Misuse resistance is an API design responsibility. Libraries that accept insecure configurations silently — rather than rejecting or warning loudly at runtime — shift the burden of correctness onto developers who are unlikely to read fine-print documentation warnings.