Inside Look at a Chinese Operational Relay Network

Inside Look at a Chinese Operational Relay Network

Speakers: Michael Torres, Zane Hoffman

Conference: DEF CON 33

YouTube: https://www.youtube.com/watch?v=e93V9TWnxJo

Slides: https://media.defcon.org/DEF%20CON%2033/DEF%20CON%2033%20presentations/Michael%20Torres%20Zane%20Hoffman%20-%20Inside%20Look%20at%20a%20Chinese%20Operational%20Relay%20Network.pdf

Overview

Michael Torres (MTU) and Zane Hoffman (Earl) present a months-long investigation that began with a simple research question — "what secrets are people accidentally publishing in Docker Hub container images?" — and ended with the discovery and full technical reconstruction of a PRC-based operational relay box network they named Meeting Mac. This network appears to serve as proxy infrastructure for anonymous browsing, social media account management, and potentially more malicious activities, and has been in active development since 2022.

The investigation was methodologically novel: rather than starting from threat intelligence or incident response data, Torres and Hoffman wrote their own Docker Hub scanning tool inspired by a DEF CON 32 talk on container image exploitation, accumulated 80,000+ container image pulls over three months, and discovered a Docker Hub user who was using container images as free cloud storage for network infrastructure backups. Those backups, combined with whited-out (deleted) files still present in container image layers, exposed the entire operational infrastructure of the relay network: SSH keys and configs, API credentials across 10+ cloud providers, database backups containing private messages and operational notes, password templates, email addresses, social media accounts, and even LLM interaction logs showing the operators using AI assistance to build the system.

The researchers stopped short of exploiting the infrastructure (they had SSH keys and root access available) and instead documented everything for presentation, noting the ethical line between researching an adversary's infrastructure and actively compromising it.

Background

▶ Watch: ORB infrastructure: compromised routers and IoT devices (1:03)

The Docker Hub exploitation research ecosystem began expanding at DEF CON 32 with a talk demonstrating a Docker exploitation framework, including the ability to identify whited-out files — files that appear deleted from a container image but remain present in the underlying overlay filesystem layer as zero-length replacement files, with the original content still accessible in earlier layers of the image tarball. This is a common mistake made by developers who docker build with COPY . (which includes all hidden files and directories) and then attempt to delete sensitive files in a later layer.

Torres and Hoffman's scanning approach: pull container images from Docker Hub, extract and analyze all layers, search for whited-out files and exposed secrets (SSH keys, API keys, passwords, database credentials) using a tool they describe as "the world's worst version of TruffleHog." They scaled to pulling 5,000 containers per day, which required upgrading to a Docker Hub premium account.

The broader context: Docker's own "How to containerize an application" documentation recommends git clone followed by COPY . . in the Dockerfile. The .git folder (and any private repository structure it contains, including CI/CD tokens stored as personal access tokens) is included in the COPY . . by default unless explicitly excluded via .dockerignore. This is a systemic misconfiguration at scale: out of 15,000 container images containing .git metadata in the researchers' dataset, 1,500 pointed to non-public (private) git repositories.

Key Findings

▶ Watch: Attribution methodology: linking ORB activity to Chinese APTs (12:29)

Private git repos in public Docker images at scale: Of 80,000+ container images pulled, approximately 15,000 contained .git folder metadata. Of those, 1,500 referenced private or non-public git repositories. Notable examples included: an enterprise branch of an Apache open-source project exposing their private repository, a Russian dashcam manufacturer's private repo, and a security vendor's private detection rules. None responded to responsible disclosure attempts.

CI/CD personal access token exposure enables code injection: When developers use personal access tokens (rather than short-lived CI/CD tokens) for Docker Hub push jobs, those tokens — which grant write access to the repository — are embedded in the .git/config inside the container image. An attacker can extract the token, push to the private repository, and trigger a new CI/CD build that includes their malicious code — which then gets published back to the Docker Hub image, completing a supply-chain attack loop. The researchers demonstrated this with a write to a repository README.

Meeting Mac: Docker as cloud storage for covert network infrastructure: One Docker Hub user was pushing multiple containers per day, each containing data labeled as "backups." These weren't application containers — they were backups of an entire operational relay infrastructure: configuration files, database dumps, SSH keys, credentials, and personal notes. Three additional related Docker Hub accounts were identified, spanning 38 total repositories with hundreds of image tags per repository.

Full SSH root access available: The infrastructure backups included a complete ~/.ssh/config file specifying the path to SSH private keys for accessing their nodes — and those SSH private keys were also present in the container images. The researchers chose not to exercise this access.

Password template and credential reuse: All operator passwords followed a template: AB[characters]@[letter corresponding to username]. For temp0230, the password ended in T. For coral86, it ended in CO. The same template was used for email accounts, domain management panels, and social media — representing a complete credential compromise across their entire infrastructure given discovery of the template and any one password.

API keys and cloud credentials: Plain-text API keys for Heroku, GitHub, Grok (xAI), and OpenAI were present throughout the container images and notes. The Heroku keys were used to run exit nodes for the relay network; GitHub keys backed up infrastructure configurations. The xAI and OpenAI keys were used — via the relay network itself, to bypass PRC internet restrictions — for AI-assisted development of the infrastructure.

BT Panel vulnerabilities: The operators used BT Panel (a Chinese-native cPanel equivalent, built on Python Flask) to manage their servers. Hoffman performed vulnerability research on BT Panel and found: (1) a patched vulnerability where the panel downloaded a PAM module over plaintext HTTP during an install routine, allowing MITM to deliver a malicious PAM module executed as root at the next login; (2) an unpatched vulnerability where the update/install script downloads RPMs and tarballs over HTTP and runs as root; (3) a universal TOTP bypass where all users share the same TOTP secret key (anyone who registers gets the codes for every user); (4) a static SSH one-time password that never expires.

Malicious PyPI package (PTMPL): The infrastructure included a malicious Python package uploaded to PyPI under the name PTMPL. A VirusTotal link was confirmed. The package contained a post-install hook that downloaded and executed a script from one of the operators' domains — a classic malicious PyPI package technique. The package has since been removed from PyPI but remains analyzable on VirusTotal.

Cobalt Strike listener log: A Cobalt Strike listener log was found in the backup data, consistent with this infrastructure having been used or prepared to support offensive operations, though the researchers noted it appeared to be test/development usage rather than confirmed operational deployment.

Technical Deep Dive

▶ Watch: Tactics, techniques, and procedures used by Chinese ORBs (17:29)

Docker Image Scanning Architecture

Torres and Hoffman's scanning pipeline:

1. Container enumeration: Iterating Docker Hub's public API to identify all public images, filtered by criteria such as update frequency (the Meeting Mac account was pushing multiple times daily) and presence of interesting metadata.
2. Layer analysis: Pulling each image as a tarball (docker save) and extracting the overlay filesystem layers. Each layer is a .tar.gz file containing the filesystem diff at that build step.
3. Whiteout detection: Scanning each layer for .wh. prefix files (Docker's overlay filesystem convention for marking deleted files) and then checking earlier layers for the original file content.
4. Secret scanning: Running pattern-matching (TruffleHog-style regex patterns for SSH keys, API keys, JWT tokens, passwords, database connection strings) against all file contents across all layers.
5. Git metadata extraction: For images containing .git folders, extracting the .git/config file to identify remote repository URLs and check whether those repositories are publicly accessible.

The elastic search index for their findings totaled 240,000 entries — acknowledging a high false-positive rate from example keys and common development patterns.

Meeting Mac Infrastructure Architecture

Based on the reconstructed infrastructure from the backup data:

Data storage layer (multiple redundant free-tier providers):

• Docker Hub (primary — how the researchers found it)
• Mega (file hosting)
• GitHub repositories (code and config backups)
• Hugging Face Spaces (compute and storage)
• Backblaze buckets
• OneDrive (one instance; decoded JWT from OneDrive auth token revealed a .ac.cn university email and a China Telecom home IP)
• TeraCloud (webdav share, largest by volume)

Compute layer:

• Traditional VMs on Heroku (exit nodes), various cloud providers
• GitHub Actions (serverless compute for automation tasks)
• Hugging Face Spaces (always assigned to the Chrome work group)

Hub domains (defanged in presentation):

• Three primary hub domains for internal C2 connectivity
• A GitHub repository access domain allowing browsing of their private repos via a custom domain
• File hosting domain for tooling and setup scripts

Node numbering scheme: Each node had a 3-digit ID. Port numbers for services followed the pattern [node-id][service-port]: e.g., port 12322 = SSH to node 123. Services included: SSH (22), VNC (5900), Visual Studio Code web server, TTYD (web terminal), headless Chrome, a scraper service, and port 90 (which returned Great Firewall blocks — an unidentified internal service).

Node type differentiation by cloud provider: Hugging Face nodes (4xx numbering) were always assigned the Chrome work group for headless browser automation. Other providers had different work group assignments.

Social management tooling: Tooling to create and manage social media accounts on Twitter/X and Facebook, including: browser fingerprint persistence (cookies + full browser fingerprints stored in the credential API, not just session cookies), headless Chrome automation, social media account seeding, and account creation automation for Gmail and Outlook.

Reverse port forwarding: The central hub server managed connectivity to all nodes via reverse port forwards — each node initiated an outbound SSH tunnel to the hub, making the nodes accessible from the hub without inbound firewall rules on the node side.

Credential API

The infrastructure included an HTTP API endpoint that served credentials on demand — either for internal use by automation scripts or potentially for external users of the relay service. The API served: SSH keys, API tokens, browser cookies, and full browser fingerprints. The fingerprint persistence ensured that automated browser sessions could evade bot detection systems that check for fingerprint consistency across sessions (a technique used by session hijacking and account takeover attacks).

AI-Assisted Development in PRC

The backup data included logs of LLM interactions (with OpenAI models and Grok) used to assist in building the infrastructure. The operators used their own relay network to access these APIs — bypassing PRC internet restrictions that block access to major AI platforms. This created a self-referential dependency: the tools used to build the censorship circumvention network were accessed via the censorship circumvention network itself.

BT Panel Vulnerability Details

BT Panel vulnerability research (published in a blog post by Hoffman with SSD Disclosure):

1. Patched PAM module download over HTTP: During setup, BT Panel downloads a PAM (Pluggable Authentication Module) over unencrypted HTTP. A MITM attacker serving a malicious PAM module achieves root code execution the next time any user logs into the system, since PAM modules run during authentication with elevated privileges.

1. Unpatched update/install HTTP download: The update and install scripts download RPMs and tarballs over HTTP and execute them as root. Still unpatched as of the talk date.

1. Shared TOTP secret: All BT Panel users on a given installation share the same TOTP (Time-based One-Time Password) base secret. Registering any account with the panel gives you the TOTP codes for all other users on that installation — including the admin account.

1. Static SSH OTP: The one-time password feature for SSH access is implemented as a static credential that never expires or rotates.

Demo / Proof of Concept

▶ Watch: Case study: specific ORB campaigns and targeted organizations (27:29)

Due to technical difficulties with the presentation display, the demos were partially described rather than fully shown. The researchers demonstrated:

• The .git/config-based CI job token extraction and write access demo (writing to a repository README via an extracted personal access token).
• Screen grabs of the Meeting Mac infrastructure backup data: SSH config showing paths to keys, database backups with operational notes, credential files with the password template, and domain lists.
• The BT Panel vulnerability demos (described; some shown via blog post reference).
• A VirusTotal link for the PTMPL malicious PyPI package.

Defensive Implications

▶ Watch: Geopolitical context: ORBs as nation-state evasion infrastructure (32:29)

For developers and DevOps engineers:

• Add .gitignore and .dockerignore entries for .git, .env, and all credential files before building container images. Using COPY . in Dockerfiles without a comprehensive .dockerignore is the root cause of the entire class of secrets-in-containers vulnerabilities.
• Use short-lived CI/CD tokens (e.g., GitHub Actions GITHUB_TOKEN) for Docker push operations rather than personal access tokens with long-term write access.
• Audit existing public Docker Hub images for .git metadata using tools like dive (Docker image layer explorer) before they are used in production.

For security researchers and blue teams:

• Docker Hub scanning is a viable threat intelligence source: consistent container fingerprints can identify clusters of infrastructure belonging to the same operator, and backup-as-image patterns leave detailed operational artifacts.
• BT Panel deployments should be hardened immediately: disable the HTTP-based update endpoint, rotate the TOTP secret per-user, and replace the static SSH OTP with a proper rotating credential mechanism.

For cloud and platform providers:

• Docker Hub's free-tier offering (including unlimited public image storage) is being abused as covert backup infrastructure. Anomaly detection on users with high push frequency and non-application-like content could identify abuse patterns.

Key Takeaways

• A single Docker Hub scanning project seeded from a DEF CON talk led to the discovery of a full PRC-based operational relay network, demonstrating that threat intelligence can come from unexpected sources when researchers are methodologically creative.
• Developer misconfiguration (COPY . . without .dockerignore, use of personal access tokens in CI/CD) creates systematic secrets exposure in container images at scale — 1,500 of 80,000 scanned containers pointed to private git repositories.
• The Meeting Mac operators used Docker Hub, GitHub, Mega, Hugging Face, Backblaze, and OneDrive as distributed free-tier storage for covert infrastructure backups, demonstrating creative abuse of legitimate cloud services.
• The credential template, SSH keys, and API keys found in the backups would have allowed complete compromise of the relay network from multiple angles; the researchers declined to exercise this access, choosing documentation over exploitation.
• BT Panel, widely deployed for server management in PRC-aligned infrastructure, has multiple unpatched vulnerabilities including an unpatched HTTP-based update mechanism allowing root code execution via MITM and a universal TOTP bypass.

About the Speaker(s)

▶ Watch: Defensive recommendations against ORB-based intrusions (42:29)

Michael Torres (MTU) is a security researcher with a broad background in network security and offensive tooling. He describes himself as "just another dude" and came to this research through practical curiosity about what secrets might be hiding in publicly available container images.

Zane Hoffman (Earl) is similarly a security researcher who contributed the Docker scanning methodology, infrastructure analysis, and BT Panel vulnerability research. Hoffman published the BT Panel findings in collaboration with SSD Disclosure. Both researchers approach their work from a practical, methodology-driven perspective, noting their research workflow runs on Discord, a Google Doc, and Notepad rather than sophisticated threat intelligence platforms.