These Are NOT the Vulnerabilities You Are Looking For: Hiding Vulnerabilities in Containers

These Are NOT the Vulnerabilities You Are Looking For: Hiding Vulnerabilities in Containers

Speakers: Kyle Quest, Creator of DockerSlim / MinToolkit

Conference: BSides Seattle

YouTube: https://www.youtube.com/watch?v=IPywCnTSeDk

Overview

Kyle Quest, creator of the popular open-source tool DockerSlim (now called MinToolkit), demonstrates how container vulnerability scanners can be trivially deceived by removing or mutating the metadata they depend on -- reducing a container with 9,000 vulnerabilities to zero reported findings while leaving every vulnerability fully exploitable. The talk exposes a fundamental weakness in how the entire container security scanning ecosystem operates.

The implications are serious: organizations that rely on third-party container images and trust vulnerability scanner output as a security gate are exposed to supply chain attacks where malicious actors deliberately strip scanner-visible metadata from images. The scanners report a clean bill of health while command injection vulnerabilities, outdated libraries, and other exploitable flaws remain fully functional inside the container.

This is not a theoretical concern. Quest demonstrates the attack live against five different commercial and open-source scanners, proving that the technique works universally across the container security tooling ecosystem.

Background

▶ Watch: Audience poll: who uses container security scanners? (0:30)

Container vulnerability scanning has become a standard practice in modern DevSecOps pipelines. Tools like Grype, Trivy, Snyk, Docker Scout, and Google's OCI Scanner analyze container images to identify known vulnerabilities in both operating system packages and application dependencies. These scanners are frequently used as quality gates in CI/CD pipelines, blocking deployment of images that exceed vulnerability thresholds.

The scanners work by examining metadata within the container image rather than testing the actual running behavior of the software. For OS-level vulnerabilities, they read distribution information from /etc/os-release (or its symlink at /usr/lib/os-release) and package metadata from /var/lib/dpkg/status on Debian-based systems. For application vulnerabilities, they parse dependency lock files like package-lock.json for Node.js applications and extract version information from package.json.

This metadata-dependent approach creates an inherent vulnerability: if the metadata is removed or mutated, the scanners have no fallback mechanism to detect vulnerabilities through behavioral analysis or binary inspection.

Key Findings

▶ Watch: Running MinToolkit OB1 obfuscation mode on the container (6:15)

Quest's primary finding is that removing three categories of metadata from a container image renders all tested vulnerability scanners completely blind:

1. OS distribution information (/etc/os-release and /usr/lib/os-release): Removing these files alone kills Google's OCI Scanner entirely -- it cannot identify any vulnerabilities without knowing the base distribution. Other scanners are severely degraded.

1. OS package metadata (/var/lib/dpkg/status on Debian): Removing the dpkg status database eliminates all OS-level package vulnerability detection across every scanner tested.

1. Application dependency metadata (lock files and version numbers): Removing package-lock.json and mutating version strings in package.json to nonsensical values eliminates application-level vulnerability detection.

The demo application -- a simple Node.js app in a Debian container -- went from approximately 9,000 reported vulnerabilities to zero across Grype, Trivy, Snyk, Docker Scout, and OCI Scanner after these modifications. Yet the application remained fully functional, and Quest demonstrated live exploitation of a known command injection CVE in the obfuscated image.

An additional finding is that SBOM (Software Bill of Materials) generators like Syft are equally affected, since they rely on the same metadata sources to enumerate components.

Technical Deep Dive

▶ Watch: Exploring the obfuscated image: missing OS release and dpkg files (11:45)

The obfuscation technique targets the specific files and data structures that vulnerability scanners parse. MinToolkit automates this process through its OB1 (obfuscation) mode, which performs both dynamic and static analysis of the container image, then reassembles it with metadata stripped or mutated.

OS-level obfuscation: The /etc/os-release file on the original Debian container is a symlink to /usr/lib/os-release containing distribution name and version. In the obfuscated image, both files are simply removed. This is the single most impactful change -- Google's OCI Scanner is completely neutralized by this alone because it cannot determine which vulnerability database to query without knowing the distribution.

Package metadata obfuscation: The /var/lib/dpkg/status file contains metadata for all installed Debian packages including names and versions. Removing this file eliminates the scanners' ability to enumerate installed OS packages and match them against vulnerability databases.

Application-level obfuscation: The tool removes package-lock.json entirely and mutates the version strings in package.json to values that do not correspond to any known package versions. Additionally, it creates symlinks in the application directory that add another layer of confusion for current scanner implementations, though Quest notes these symlinks do not significantly impact current scanners.

The critical insight is that the application itself remains fully functional after all metadata removal. The Node.js runtime does not need package-lock.json to run, the OS does not need /etc/os-release for normal operation, and the dpkg status database is only used by the package manager, not by the installed software. The obfuscation has zero impact on the container's runtime behavior.

Demo / Proof of Concept

▶ Watch: Mutated package.json versions and removed lock files (15:40)

Quest ran the entire demonstration live, starting by scanning the original demo container image with Grype (approximately 9,000 vulnerabilities), Trivy, Snyk, Docker Scout, and OCI Scanner. He then ran MinToolkit's OB1 obfuscation mode against the image, which performed analysis and produced an obfuscated variant with a modified tag name.

Rescanning the obfuscated image with Grype returned zero vulnerabilities. Quest then explored the obfuscated image filesystem, showing the missing /etc/os-release files, the absent /var/lib/dpkg/status, the removed lock file, and the mutated version strings in package.json.

The most compelling part of the demo was running the obfuscated container and exploiting the command injection vulnerability that scanners could no longer detect. The application's ping endpoint accepted user input that could be used for command injection, and Quest successfully executed arbitrary commands inside the container -- proving that the vulnerabilities were fully present and exploitable despite every scanner reporting a clean image.

Defensive Implications

▶ Watch: Supply chain implications: attackers can hide vulns in third-party images (20:00)

This research has immediate implications for any organization using container vulnerability scanning as a trust gate for third-party images. If a supply chain attacker distributes a malicious container image with stripped metadata, standard scanning tools will fail to detect the embedded vulnerabilities.

Defenders should consider several countermeasures. First, never rely on vulnerability scanning as the sole security gate for container images. Runtime security monitoring, behavioral analysis, and application-level testing (DAST, IAST) provide defense-in-depth that is not dependent on metadata.

Second, use distroless or minimal base images where possible -- these reduce the actual attack surface rather than just hiding it. MinToolkit's original purpose (slimming containers) actually removes unused packages rather than just hiding their metadata.

Third, implement policies that flag containers missing expected metadata files. A container image without /etc/os-release or /var/lib/dpkg/status is inherently suspicious and should trigger additional review rather than being passed as "clean."

Fourth, have explicit conversations with development teams about the limitations of vulnerability scanners. Developers who understand that scanners rely on metadata rather than behavioral analysis are less likely to develop false confidence in scan results.

Key Takeaways

• Container vulnerability scanners (Grype, Trivy, Snyk, Docker Scout, OCI Scanner) rely entirely on metadata and can be trivially blinded by removing OS release info, package databases, and dependency lock files
• A container with 9,000 known vulnerabilities can report zero findings across all major scanners while remaining fully exploitable
• SBOM generators like Syft are equally vulnerable since they depend on the same metadata sources
• Removing just /etc/os-release is enough to completely neutralize some scanners like Google's OCI Scanner
• Organizations trusting third-party container images based solely on scan results are exposed to supply chain attacks using this technique
• Defense requires layered approaches: runtime monitoring, behavioral analysis, and policies flagging containers with missing metadata

About the Speaker(s)

Kyle Quest is the creator of DockerSlim (now MinToolkit), a widely-used open-source tool for optimizing and slimming container images. He describes himself as "the Docker Slim guy." The tool was originally built to reduce container image size by removing unnecessary components, but Quest added obfuscation capabilities to demonstrate the fragility of the vulnerability scanning ecosystem. He presented this research as an AI-free talk, noting with some humor that the demonstration contains "no AI, AI free."