ProtocolGuard: Detecting Protocol Non-compliance Bugs via LLM-guided Static Analysis and Dynamic Verification

ProtocolGuard: Detecting Protocol Non-compliance Bugs via LLM-guided Static Analysis and Dynamic Verification

Speakers: Xiangpu Song

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=V3sXujp3snQ

Overview

This talk presents ProtocolGuard, a hybrid framework that detects protocol non-compliance bugs -- semantic errors where implementations deviate from protocol specifications (RFC documents). Unlike memory safety bugs that trigger crashes, these bugs are silent: they don't produce explicit error signals but can cause serious consequences including service disruption, client impersonation, and denial-of-service attacks.

ProtocolGuard combines LLM-guided static analysis to extract protocol rules and identify relevant code, with fuzzing-based dynamic verification using LLM-generated assertion statements as bug oracles. Applied to 11 open-source protocol implementations covering six protocols (MQTT, TLS, DHCP, FTP, and others), the framework extracted 420 rules from specification documents and confirmed 158 unique non-compliance bugs, including previously unknown vulnerabilities with assigned CVEs. The key innovation is using LLMs to generate assertion statements that transform silent non-compliance bugs into observable assertion failures that fuzzing can detect.

Background

▶ Watch: Protocol non-compliance bugs: widespread, silent, and dangerous (0:00)

Network protocols underpin critical infrastructure including IoT, 5G, and enterprise networks. Protocol implementations frequently contain non-compliance bugs -- deviations from the specification that arise because RFC documents are written in natural language, which developers can misinterpret or partially implement. These bugs differ fundamentally from memory safety issues: they don't crash programs, they don't trigger sanitizers, and they often produce correct-looking responses while silently violating protocol semantics.

Existing detection methods struggle with this bug class. Fuzzing relies on explicit signals (crashes, sanitizer reports) that non-compliance bugs don't produce. Differential testing compares responses across implementations, but both vulnerable and correct implementations may return identical responses for non-compliance issues. Traditional heuristic-based static detection uses predefined rules extracted from specifications but lacks the semantic understanding needed to assess whether implementation logic truly preserves protocol invariants.

The motivating example illustrates the challenge: an MQTT broker copies the client identifier field into a fixed-size buffer without validating its length. The broker follows the specification by supporting long IDs, but misses an implicit requirement that client IDs must remain unique and complete. This allows attackers to create client ID collisions and impersonate legitimate clients, causing DoS attacks -- yet the bug produces no crashes, no sanitizer signals, and returns normal-looking responses.

Key Findings

▶ Watch: LLM-guided program slicing: extracting rule-relevant code (4:00)

158 unique non-compliance bugs confirmed across 11 implementations of six protocols, including previously unknown vulnerabilities with assigned CVEs.

420 protocol rules extracted from official specification documents using LLM-guided modal keyword identification (must, must not, shall, etc.).

High precision detection: ProtocolGuard achieved the best precision and recall compared to state-of-the-art AI editors (Cursor configured with Claude and DeepSeek R1), despite using the same underlying models.

100% syntactic correctness for LLM-generated assertion statements -- all generated statements compiled successfully without introducing errors.

Bug categories: The majority of detected bugs related to message parsing, followed by state management, error handling, and access control.

Silent bugs made observable: The assertion-based approach successfully transformed silent non-compliance bugs into observable assertion failures, achieving a relatively high crash triggering rate within 24 hours of fuzzing.

Technical Deep Dive

▶ Watch: Handler function identification and semantic pruning (8:00)

ProtocolGuard consists of four modules:

Module 1 -- Protocol Rule Extraction: Processes specification documents to extract formal rules. The system identifies sentences containing modal keywords (must, must not, shall) that carry constraining semantics. Output is structured JSON including original rule descriptions, request/response types, and fields constrained by the rules. A second LLM pass polishes incomplete rules that span multiple sentences, ensuring each rule captures complete conditions and actions.

Module 2 -- LLM-Guided Program Slicing: Extracts rule-relevant code from the full codebase through three steps:

1. Handler function identification: Based on the insight that message handling logic resides in dedicated handler functions. The LLM traverses the program's call graph to identify entry functions for each message type (e.g., do_tls_client_hello for TLS ClientHello)
2. Forward slicing: Starting from entry functions, the LLM maps specification fields to code variables, establishing slicing criteria. Data dependency analysis extracts all rule-relevant code logic
3. Semantic pruning: Since data dependency analysis only checks data flow, the LLM performs semantic analysis to exclude functions that are data-connected but semantically unrelated to the rule description

Module 3 -- LLM-Based Inconsistency Detection: Analyzes extracted rules and code slices to generate inconsistency reports, identifying where implementations deviate from specification requirements. Uses DeepSeek for program slicing and Claude for code generation.

Module 4 -- Fuzzing-Based Dynamic Verification: The critical innovation -- generating assertion statements as bug oracles through four steps:

1. Extract expected validation constraints from inconsistency reports and protocol rule descriptions
2. Identify missing or incorrect validation logic in the implementation
3. Use an LLM agent to iteratively synthesize and instrument assertion statements at identified locations
4. Compile, test, and refine assertions until correct

For the MQTT motivating example, the LLM generates an assertion that validates client ID field length against the buffer size. When the fuzzer generates an oversized client ID, the assertion terminates the program, exposing the previously silent bug as a detectable crash.

Demo / Proof of Concept

▶ Watch: LLM-generated assertion statements as bug oracles for fuzzing (10:00)

The evaluation covered 11 open-source C-language protocol implementations across six protocols spanning IoT (MQTT v3.1, v5), secure transport (TLS 1.3), file transfer (FTP), and network services (DHCP). Using DeepSeek for program slicing and Claude for code analysis, ProtocolGuard:

• Extracted 420 rules from specification documents
• Detected 181 inconsistencies with high precision
• Confirmed 158 unique non-compliance bugs after excluding intentional specification deviations
• Outperformed Cursor AI editor configured with the same Claude and DeepSeek R1 models
• Achieved 100% syntactic correctness on generated assertion statements
• Demonstrated effective crash triggering within 24 hours of directed fuzzing

Defensive Implications

▶ Watch: Q&A: applicability to different specification types and languages (14:00)

ProtocolGuard addresses a critical blind spot in protocol security testing. Most security teams focus on memory safety bugs because they produce detectable signals, while non-compliance bugs -- which can be equally dangerous -- remain undetected for years.

For protocol implementers: The framework can be integrated into CI/CD pipelines to automatically verify specification compliance as code evolves. The rule extraction module creates a structured representation of protocol requirements that serves as a living compliance checklist.

For security auditors: The LLM-guided program slicing enables focused code review of rule-relevant logic rather than manual review of entire codebases. The inconsistency reports provide prioritized findings tied directly to specification clauses.

For fuzzing teams: The assertion statement generation transforms the fundamental challenge of logic bug detection -- the absence of crash signals -- into a solvable problem by inserting specification-derived oracles directly into the code.

Current limitations: The system currently works best with C-language implementations and protocols with formal specifications using modal keywords (primarily RFC-based protocols). Application to other languages (Java, Python) or less formally specified protocols may require adaptation. The Q&A discussion confirmed the approach is primarily targeted at RFC-based, C-implemented protocols.

Key Takeaways

• Protocol non-compliance bugs are widespread, silent, and dangerous -- 158 unique bugs found across 11 implementations of six protocols
• LLM-guided program slicing extracts rule-relevant code from entire codebases, enabling focused analysis without manual code review
• LLM-generated assertion statements transform silent non-compliance bugs into observable assertion failures detectable by fuzzing
• ProtocolGuard outperforms direct LLM application (Cursor with Claude/DeepSeek R1), demonstrating that structured pipelines matter more than raw model capability
• The approach works best for RFC-based protocols with modal keyword constraints implemented in C
• Not all specification inconsistencies are bugs -- developers sometimes intentionally deviate from specifications for valid reasons

About the Speaker(s)

Xiangpu Song (Shan Pu) presented this research at NDSS. The work demonstrates practical application of LLMs to formal protocol compliance verification, combining natural language understanding of specifications with program analysis of implementations. The research resulted in CVE assignments for previously unknown vulnerabilities discovered across multiple protocol implementations.