Client or Server? Hidden Sword of Damocles in Kafka

Client or Server? The Hidden Sword of Damocles in Kafka

Speakers: Ji'an Zhou, Ying Zhu, ZiYang Li

Conference: DEF CON 33

YouTube: https://www.youtube.com/watch?v=Sa6Onq53TsY

Slides: https://media.defcon.org/DEF%20CON%2033/DEF%20CON%2033%20presentations/Ji%27an%20Zhou%20%26%20Ziyang%20Li%20-%20Client%20or%20Server%20The%20Hidden%20Sword%20of%20Damocles%20in%20Kafka.pdf

Overview

Apache Kafka is the backbone of modern data-intensive architectures. Deployed by thousands of enterprises for real-time data pipelines, event streaming, and critical application integration, a single Kafka cluster may process billions of messages per day across the world's largest organizations. At DEF CON 33, Ji'an Zhou, Ying Zhu, and ZiYang Li — security engineers from Alibaba Cloud — presented a systematic analysis of a class of vulnerabilities that has been lurking in Kafka's architecture since its design: the ambiguity of client vs. server identity in Kafka's protocol, and how that ambiguity enables a chain of exploits ranging from unauthorized data access to full broker compromise in production Kafka ecosystems.

The research goes well beyond known Kafka security issues, introducing novel attack vectors in both the legacy ZooKeeper-based and the modern KRaft control plane modes, and demonstrating exploitation of the Kafka ecosystem that extends into the broader Java deserialization and SSRF landscape.

Background

▶ Watch: Kafka architecture: data plane vs control plane and broker roles (2:19)

What Is Apache Kafka?

Apache Kafka is an open-source distributed event streaming platform. In its core architecture, three types of participants interact:

• Producers: Data sources that write messages to Kafka topics
• Consumers: Applications that read and process messages from topics
• Kafka Cluster: A group of brokers that store and replicate messages, managed by a control plane

The control plane has two implementations:

• ZooKeeper mode (legacy): Cluster metadata (topic configuration, broker registry, partition leadership) is managed by an external Apache ZooKeeper ensemble
• KRaft mode (new): Kafka's own Raft-based consensus implementation manages metadata internally, without ZooKeeper dependency

In the data plane, requests are classified as either produce requests (clients writing to topics) or fetch requests (clients reading from topics). This producer/consumer/broker model handles message routing, replication between brokers, and storage.

The Security Problem: Who Is Really a Client vs. a Server?

Kafka's protocol was designed for high throughput in trusted network environments. The distinctions between a Kafka client (producer or consumer) and a Kafka broker (server) in the protocol are less strict than one might expect. A Kafka broker uses the same wire protocol to communicate with other brokers as clients use to communicate with brokers. Replication, partition leadership negotiation, and metadata operations all flow over the same TCP-based Kafka protocol.

This design creates a class of vulnerabilities rooted in the question: when a connection is established to a Kafka broker, how does the broker determine whether it is talking to a legitimate client, a legitimate peer broker, or an attacker masquerading as one or the other? The researchers term this ambiguity the "Hidden Sword of Damocles" — a threat that hangs over every Kafka deployment, largely unrecognized.

Prior Research Context

Previous Kafka security research has focused primarily on known vulnerability classes: unauthenticated access to clusters deployed without TLS or SASL authentication (a common misconfiguration), specific CVEs in Kafka's deserialization handling, and administrative interface exposure. The research presented by Zhou, Zhu, and Li extends this landscape significantly by examining the protocol-level identity confusion problem and its exploitation in both control plane implementations.

Key Findings

▶ Watch: ZooKeeper attack chain: writing fake broker registration for RCE (6:08)

• Kafka's broker-to-broker replication protocol can be triggered by a malicious client, enabling unauthorized data access by an attacker who can connect to a Kafka cluster without producer/consumer privileges for the target topic.

• The ZooKeeper control plane contains exploitable vulnerabilities that allow an attacker with write access to ZooKeeper (achievable through broker manipulation) to achieve code execution on Kafka brokers.

• The KRaft control plane introduces new attack surfaces, including vulnerabilities in the KRaft metadata quorum protocol handling that were not present in the ZooKeeper mode.

• Java deserialization vulnerabilities remain reachable through Kafka's protocol under specific conditions, enabling remote code execution against broker JVM processes.

• SSRF vulnerabilities exist in Kafka's Connect framework, allowing an attacker to use a Kafka Connect worker as a proxy to reach internal network services.

• Attacks chain across control and data planes: A vulnerability in one plane (e.g., gaining ZooKeeper write access) enables escalation in the other (achieving broker RCE), creating multi-stage exploitation chains.

Technical Deep Dive

▶ Watch: Identity confusion: Kafka cannot distinguish legitimate clients from fake bro... (6:45)

Kafka's Dual-Role Protocol and Broker Identity Confusion

When a Kafka broker receives an incoming connection, it examines the API keys (request types) used by the connecting party to determine what role the connecting party is playing. However, a connecting client can present itself as a peer broker by using the broker-specific API keys — specifically the Fetch Request with an internal broker identifier (replicaId field set to a positive broker ID rather than -1, which is used by consumer clients).

Legitimate Kafka replication uses this mechanism: follower brokers fetch messages from the leader broker using a Fetch Request with their own replicaId. If an attacker sets replicaId to a legitimate broker's ID (or a plausible value), some versions of Kafka will treat the connection as a broker replication request and respond with data the attacker is not authorized to receive as a regular consumer.

This "replication impersonation" attack allows an attacker who cannot consume from a topic (because they lack consumer group authorization) to fetch topic data by masquerading as a replicating broker — completely bypassing the consumer authorization check.

ZooKeeper Control Plane Exploitation Chain

In ZooKeeper-based Kafka clusters, all cluster metadata — broker registrations, topic configurations, partition leadership, and ACLs — is stored in the ZooKeeper tree. If an attacker gains write access to ZooKeeper (which requires network access to the ZooKeeper client port, commonly 2181, which is frequently exposed without authentication in misconfigured deployments), a full cluster compromise chain becomes available:

1. Broker registration manipulation: An attacker writes a fake broker registration entry into ZooKeeper under /brokers/ids/<ID>. Legitimate Kafka brokers read these entries to establish cluster topology.

1. Leader election manipulation: By modifying partition leadership entries in ZooKeeper, the attacker can direct producer/consumer traffic to the fake broker endpoint.

1. Follower replication abuse: Causing a legitimate broker to attempt replication from the attacker's fake broker. When the legitimate broker connects to the fake broker to fetch replicated data, the attacker can respond with crafted protocol messages.

1. Java deserialization via crafted Kafka protocol messages: Older Kafka versions (and certain plugin configurations) deserialize Java objects from specific protocol fields. Responding with a crafted payload containing a known Java deserialization gadget chain achieves RCE on the connecting legitimate broker's JVM.

The full chain takes a ZooKeeper write primitive and escalates it to RCE on multiple Kafka brokers.

KRaft Control Plane Attack Surface

The newer KRaft (Kafka Raft) mode eliminates ZooKeeper but introduces its own attack surface. The KRaft quorum protocol handles metadata log replication among a set of designated controller nodes using a Raft consensus variant. The researchers identified:

• Quorum voter impersonation: Similar to the data plane replication impersonation, an attacker who can reach the KRaft controller port may be able to inject crafted Raft protocol messages, potentially disrupting quorum or triggering controller elections.
• Metadata log poisoning: If a controller node can be manipulated into accepting crafted log entries, these may propagate to all controller nodes and eventually to all brokers as authoritative metadata updates — enabling ACL manipulation, broker deregistration, or topic deletion.

The specific CVE details and patch status for the KRaft vulnerabilities were disclosed to the Apache Kafka security team prior to the talk.

Kafka Connect SSRF

Apache Kafka Connect is a framework for building scalable data pipelines that connect Kafka topics to external systems (databases, cloud storage, other message queues). Kafka Connect workers expose a REST API for managing connector configurations. The researchers identified that certain connector configurations can be manipulated to cause the Connect worker to make HTTP requests to attacker-controlled URLs — a Server-Side Request Forgery (SSRF) condition.

Since Kafka Connect workers typically run in internal network segments with access to multiple internal services (databases, cloud metadata APIs, internal monitoring systems), SSRF via the Connect REST API provides an attacker with a pivot point into the internal network. The cloud metadata endpoint (169.254.169.254 for AWS IMDSv1) is a particularly valuable SSRF target, potentially exposing IAM credentials.

Exploitation in Practice: The Full Attack Chain

The researchers demonstrated a complete multi-stage attack chain:

1. Attacker gains access to a Kafka cluster (perhaps through an exposed unauthenticated listener, or with limited consumer credentials).
2. Replication impersonation is used to exfiltrate topic data from topics the attacker is not authorized to consume.
3. ZooKeeper access (obtained via the broker connection if ZooKeeper is co-located without authentication) is used to register a fake broker.
4. Legitimate broker's replication connection to the fake broker is used to deliver a deserialization payload.
5. RCE on the broker JVM provides full control of the Kafka node, including access to all topics' stored data and the ability to modify cluster configuration.

Demo / Proof of Concept

▶ Watch: Core vulnerability: broker impersonation enables unauthorized data access (11:17)

The DEF CON 33 presentation included live demonstrations:

1. Replication impersonation: A Python client script using the Kafka protocol library directly (bypassing the consumer ACL check) to fetch from a restricted topic by setting replicaId to impersonate a broker.

1. ZooKeeper chain: Against a test Kafka cluster with default ZooKeeper configuration (no authentication), demonstrating the progression from ZooKeeper write access to broker registration injection to triggering replication from a fake broker, ending with a Metasploit-style shell on the target broker JVM.

1. Kafka Connect SSRF: Demonstrating extraction of AWS instance metadata credentials via a crafted connector configuration submitted to the Connect REST API.

All demonstrations used test environments; the researchers followed responsible disclosure processes with the Apache Kafka security team before presenting.

Defensive Implications

▶ Watch: Kafka Connect SSRF: server-side request forgery via connector configuration (33:42)

For Kafka administrators:

• Enable authentication on all Kafka listeners: configure SASL (SCRAM-SHA-256 or SCRAM-SHA-512 preferred, or Kerberos in enterprise environments) on all broker listeners, including inter-broker replication listeners.
• Enable TLS encryption on all listeners. Cleartext Kafka traffic allows passive interception and man-in-the-middle attacks.
• Restrict ZooKeeper access: ZooKeeper client ports (2181) should never be exposed to untrusted networks, and ZooKeeper SASL authentication should be enabled.
• In KRaft mode, ensure the controller port (9093 by default) is not accessible from untrusted network segments.
• Restrict Kafka Connect REST API access to trusted management networks; do not expose it to the internet or untrusted internal segments.
• Upgrade to patched Kafka versions that address the specific CVEs disclosed in this research.
• Enable Kafka ACLs and configure them with least-privilege principles — do not use wildcard allow policies.

For security teams:

• Audit Kafka deployments for exposed unauthenticated listeners using network scanning tools.
• Monitor Kafka broker logs for unexpected replication connections from unknown IP addresses.
• Treat the Kafka ZooKeeper quorum as a critical security boundary — compromise of ZooKeeper is equivalent to compromise of the entire Kafka cluster.

Key Takeaways

1. Kafka's client/server identity ambiguity in the replication protocol enables unauthorized data access via broker impersonation, bypassing consumer authorization controls.
2. ZooKeeper write access enables a full broker RCE chain via fake broker registration and Java deserialization gadget exploitation.
3. The KRaft control plane introduces new Raft protocol-level attack surfaces distinct from ZooKeeper vulnerabilities.
4. Kafka Connect SSRF enables internal network pivoting from Kafka infrastructure to cloud metadata and internal services.
5. Defense requires authentication and encryption on all listeners, ZooKeeper protection, and least-privilege ACL configuration — the majority of exposed Kafka clusters use none of these controls.

About the Speaker(s)

▶ Watch: Live demo: full exploitation chain against a real Kafka cluster (38:14)

Ji'an Zhou, Ying Zhu, and ZiYang Li are security engineers on the Alibaba Cloud security research team. They focus on vulnerabilities in cloud-native infrastructure components, with expertise in distributed systems security, Java ecosystem vulnerabilities, and large-scale cloud platform hardening. Their research at DEF CON 33 represents a systematic security audit of Apache Kafka's protocol and ecosystem from the perspective of attackers who have studied the codebase and protocol specification in depth.