Lost in Translation: Exploiting Unicode Normalization

Lost in Translation: Exploiting Unicode Normalization

Speakers: John Barnett (with Isabella Barnett / "Angel Hacker")

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=ETB2w-f3pM4

Reading time: ~9 minutes

Type: Briefing

TL;DR

John Barnett and his daughter Isabella ("Angel Hacker"), a cybersecurity engineering student, present a systematic taxonomy of Unicode normalization vulnerabilities that let attackers bypass security filters by submitting characters that look benign to the security layer but are transformed into dangerous content by the application layer. The talk covers four attack classes — multi-byte decoding errors, overlong encoding, byte truncation, and combining/confusable characters — each tied to a real-world case study, with updates released to ActiveScan++ and Recollapse to detect them.

Introduction

The telephone game is a perfect analogy: a message whispered from person to person transforms unpredictably by the time it reaches the end of the line. John Barnett mapped that dynamic to web application security pipelines — a request travels through a CDN security layer, an application firewall, and then into the application itself, where each component may interpret the same bytes differently.

The root cause is CWE-180: "Incorrect Behavior Order: Validate Before Canonicalize." A security check is applied to input in one form; then the application transforms the input into another form; and the transformed data bypasses the check. The pattern recurs across all four attack classes presented. "If there's one thing you take away from this talk," Barnett said, "it's CWE-180."

Attack Class 1: Multi-Byte Decoding Errors (CWE-172)

Unicode can represent characters using multiple bytes; ASCII is limited to single bytes with a leading zero bit. Multi-byte Unicode characters use continuation bytes with a leading 1 and 0 bit pattern, signaling that multiple bytes form a single character.

▶ Watch: Multi-Byte Awareness (04:00)

When a decoder treats multi-byte sequences as independent single bytes — dropping the leading continuation markers — the result is mojibake: "character mutation" in Japanese. The bytes produce unrelated characters that share no semantic connection with the original input.

A practical example: Burp Suite's built-in decoder is not multi-byte-aware. Submitting the UTF-8 encoding of a < (left angle bracket, a multi-byte character) returns three unrelated Unicode characters. A security filter scanning for < will not find it — the check passes. But if the downstream application later decodes the raw bytes correctly, the < reappears. The fix for testing: install the "Decoder Improved" extension from the BApp Store, which correctly handles multi-byte sequences.

The team released updates to ActiveScan++ (also available from the BApp Store) to detect this class of vulnerability automatically.

Attack Class 2: Overlong Encoding (KPEK-80)

Overlong encoding splits a normally single-byte character into a multi-byte representation that is technically invalid but that some decoders accept. The analogy Barnett uses: receiving an Amazon package containing only another Amazon package inside — "needless packaging."

▶ Watch: Overlong Encoding and NIMDA (10:00)

The . (dot), / (forward slash), and \ (backslash) characters can be overlong-encoded into two-byte UTF-8 sequences. A security filter scanning for ../ as a path traversal indicator will not recognize the overlong-encoded form. The application, however, may normalize the sequence back to the ASCII characters before processing the path — and the traversal succeeds.

This is not a new technique: the NIMDA worm of 2001 exploited exactly this flaw to bypass path-traversal filters on IIS web servers, which checked for ../ but did not normalize overlong encodings before the check. Barnett noted he was defending government websites at the time and recognized the pattern immediately.

A live demo using the Regex101 website illustrated the pipeline failure: the same overlong-encoded SQL fragment (SELECT ... FROM) successfully bypasses a regex filter because the overlong-encoded space character does not match the pattern; the application, meanwhile, normalizes it back to a space before executing the query.

Attack Class 3: Byte Truncation

Some server-side processing stores only a single byte from a multi-byte Unicode character — specifically the least significant byte in little-endian memory layouts. The higher bytes are silently discarded.

▶ Watch: Byte Truncation Case Study (12:00)

A real-world case study: a bug hunter who found CRLF injection in a Microsoft application. The carriage return (\r, 0x0D) and line feed (\n, 0x0A) were filtered by the security layer. However, a specific Chinese character — three bytes in UTF-8 — has 0x0D as its least significant byte. When stored in a single-byte buffer on the server, byte truncation leaves only 0x0D, which is the carriage return. The security layer never saw a carriage return; the application produced one.

The tool Shazzer (by Gareth Hayes) automates the inverse lookup: given any target byte value, Shazzer returns all Unicode characters that would truncate to that byte, enabling systematic testing. ActiveScan++ was also updated to include a truncation probe: a Hangul symbol whose least significant byte is { (0x7B) — if the response contains a {, the application may be vulnerable to server-side template injection.

Attack Class 4: Confusables and Normalization Forms (CWE-180)

Unicode defines four normalization forms: NFC, NFD, NFKC, and NFKD. The "K" forms apply compatibility normalization, mapping visually similar ("confusable") characters to their ASCII equivalents. "Best-fit mapping" is a related Windows-specific mechanism that maps Unicode characters to the nearest ASCII equivalent for a given code page.

▶ Watch: Confusables and Best-Fit Mapping (16:00)

Case study – .NET Nuke path traversal (Searchlight Cyber, July): An attacker submitted a file upload with a filename containing Unicode characters that visually resemble backslashes and dots. The security filter ran a regex against the filename and found no match — because the Unicode confusables are not the ASCII backslash or dot characters. The application then applied code page 1251 best-fit mapping, which silently converted the Unicode full-stop to an ASCII . and the Unicode characters resembling backslashes to ASCII \. The result was a UNC path (\\elastify...), enabling SSRF via an outbound connection to a Burp Collaborator-style server.

SSRF via enclosed alphanumerics: Circled digit characters (e.g., ① ② ③) are commonly used to bypass domain-filter checks. Under IDNA processing, these normalize to their base ASCII digit. Demo at the command line using curl confirmed resolution of a circled-numeral domain to the loopback IP.

▶ Watch: Case Studies — Casing and XSS (24:01)

Case study – casing and XSS: The "Latin small letter dotless i" (ı, U+0131) is not an ASCII i. A security filter scanning for <script> will not find <scrıpt>. But in Turkish locale or certain JavaScript engines, applying .toUpperCase() to dotless i yields I, and the normalized string <SCRIPT> is recognized by browsers. The team added casing-based tests to the Recollapse tool (updated in coordination with its creator on the day of the talk).

Case study – combining characters and XSS: Normalization forms that perform decomposition separate characters into their base plus combining diacritic. A "combining long solidus overlay" concatenated with > produces ≯ (not-greater-than) under composition. Placing this combining character at the start of a parameter in a text-area injection context causes the application to transform > into ≯, breaking the text-area close tag. An injected onfocus=alert(1) event handler then executes as cross-site scripting. ActiveScan++ was updated to detect this combining-character variant automatically.

Case study – account takeover via combining characters: In a password reset flow, an attacker submits an email address containing a Unicode character with a combining diacritic — for example, a + combining grave accent. If the database collation is accent-insensitive (case and accent insensitive is a common SQL Server default), the database treats à as equivalent to a and matches the attacker's email to the victim's stored address. The password reset token is then sent to the attacker's email, not the victim's. This is a zero-click account takeover.

▶ Watch: Account Takeover Demo (26:01)

Tooling Updates

All four classes have corresponding detection improvements released alongside this talk:

| Tool | Update |

|---|---|

| ActiveScan++ (BApp Store) | Multi-byte probe, truncation probe (Hangul → { and }), normalization/Kelvin-sign probe, combining-character XSS probe |

| Decoder Improved (BApp Store) | Multi-byte-aware decoding for correct display in Burp |

| Recollapse | New --casing flag to enumerate Unicode characters that map to a target ASCII character under case folding |

| Shazzer (Gareth Hayes) | Inverse lookup: Unicode chars that truncate to a given byte |

The Kelvin sign (K, U+212A) is the recommended probe character for normalization detection: it is one of the few Unicode characters that maps to an ASCII character (capital K) under any normalization form, making it an efficient single-probe test.

Notable Quotes

"If there's one thing you take away from this talk, it's CWE-180. A security check, later on data changes — recipe for disaster."

— John Barnett ▶ 02:00

"This is NIMDA. I was working at the time protecting some government websites. I saw this coming in. I'm like, 'What the heck is going on here?'"

— John Barnett ▶ 10:00

"Anytime you're using uppercase, lowercase, check that data."

— John Barnett ▶ 26:01

"The problem is after those security checks, there's something inside these applications that's actually processing and changing and manipulating data, turning it from something that was not an attack into something that actually works."

— John Barnett ▶ 02:00

Key Takeaways

• CWE-180 is the universal root cause: Applying a security check before canonicalization is the structural mistake underlying all four attack classes. Normalize first, check second.
• Multi-byte awareness is a prerequisite for correct security logic: Any component that processes Unicode character by character without accounting for multi-byte sequences is potentially bypassable with mojibake payloads.
• Overlong encoding bypasses are historical but still present: NIMDA used this technique in 2001; modern applications with regex-based input validation that lack proper URL decoding prior to pattern matching remain vulnerable.
• Database collation settings create account-takeover risk: Accent-insensitive and case-insensitive collations (common SQL Server defaults) treat Unicode characters as equivalent to their ASCII base, enabling password-reset hijacking.
• Third-party libraries introduce hidden normalization: Applications may perform normalization they are unaware of via standard library calls (.toUpperCase(), framework-level encoding) — audit these paths explicitly.

Slides PDF not available for this talk.