MalwareDB: An Open-Source Bookkeeping System for Malicious and Benign Files

MalwareDB: An Open-Source Bookkeeping System for Malicious and Benign Files

Speakers: Richard Zak, Malware and Machine Learning Researcher

Conference: ShmooCon

YouTube: https://www.youtube.com/watch?v=wXbnUm89IJw

Overview

Richard Zak's ShmooCon talk introduced MalwareDB, an ambitious open-source project designed to address a pervasive challenge faced by malware analysts, incident responders, and machine learning researchers: the efficient storage, management, and contextualization of vast collections of malicious and benign files. Born out of the speaker's personal experience dealing with "obscene amounts of malware" – often terabytes across millions of files – MalwareDB aims to provide a robust, flexible, and interoperable solution for what Zak terms "bookkeeping" in the realm of cybersecurity samples. The project, initiated during the pandemic, fills a critical gap in the existing tool landscape, offering a centralized system to track not just the files themselves, but also their rich metadata, extracted features, and relationships.

The core motivation behind MalwareDB stems from the limitations of simply storing files by their cryptographic hashes. While essential for unique identification, this approach offers little insight into file similarity, origin, or potential connections to broader campaigns or threat actors. Zak highlighted that a single file, in isolation, provides minimal context. MalwareDB seeks to remedy this by integrating advanced analytical capabilities, including various hashing algorithms, automated feature extraction, and optional ties to external threat intelligence sources like VirusTotal. By doing so, it empowers security teams to move beyond basic file storage towards a more intelligent, collaborative, and context-rich understanding of the digital threats they encounter.

Ultimately, MalwareDB is presented as a foundational tool for enhancing threat intelligence, improving incident response workflows, and accelerating the development of machine learning models for malware detection and clustering. Its open-source nature, coupled with a design philosophy emphasizing interoperability and extensibility, positions it as a valuable asset for the security community. Zak's presentation at ShmooCon marked a significant public unveiling of this project, inviting contributions and collaboration to further evolve its capabilities and impact.

Background

▶ Watch: Introduction to MalwareDB and problem statement (0:20)

The genesis of MalwareDB lies in a fundamental problem encountered by security professionals: the sheer volume and complexity of managing malware samples. As organizations and researchers collect "terabytes, millions of files" of both malicious and benign executables, documents, and other artifacts, the challenge of effective "bookkeeping" becomes immense. Traditional methods often involve simple file system storage, perhaps indexed by a cryptographic hash like SHA-256. While this allows for unique identification and retrieval, it falls short when attempting to derive deeper insights, understand relationships between samples, or efficiently contextualize new threats against a historical repository.

Richard Zak, drawing from his experience as a malware and machine learning researcher, identified a significant void in the open-source landscape for tools that could comprehensively manage these datasets. He noted that while projects like VXCage offered a starting point – allowing files to be stored in MongoDB and retrieved by their SHA-256 hash – they lacked the depth required for advanced analysis. VXCage, a Python script, demonstrated the utility of a database-backed approach but didn't extend to storing additional metadata, parsing file features, or enabling similarity searches. This limitation means that analysts often have to re-analyze samples or manually cross-reference information, leading to inefficiencies and potential missed connections.

Furthermore, the environment in which malware research occurs presents unique challenges. Collecting and storing malicious files can be precarious; endpoint security software, designed to protect systems, often inadvertently deletes these critical samples, disrupting research efforts. This necessitates secure storage mechanisms that can safeguard samples while still making them accessible for analysis. The need for context is also paramount: knowing the origin of a sample (e.g., public source like VirusShare, or a private client engagement) dictates sharing policies and helps assess its relevance. Without a system to track this, sensitive data might be inadvertently exposed, or valuable public intelligence might not be fully leveraged. MalwareDB was conceived to directly address these multifaceted problems, building upon the basic premise of VXCage but significantly expanding its capabilities to offer a holistic solution for malware data management.

Key Findings

▶ Watch: Leveraging fuzzy hashes for file similarity (3:30)

MalwareDB stands out as a significant contribution to the open-source security community by offering a comprehensive and extensible platform for malware and goodware management. Its key findings and contributions can be summarized as follows:

1. Unified Sample Management: MalwareDB provides a centralized client-server system for storing and retrieving both malicious and benign files. This moves beyond simple file system storage, enabling a structured approach to managing vast datasets of samples, which can number in the millions and span terabytes.

1. Advanced Hashing and Similarity Analysis: A core contribution is the integration of multiple hashing algorithms, including standard cryptographic hashes (SHA-256, MD5, SHA-1) alongside fuzzy hashes or similarity hashes. These include ssdeep and Lochard's Jaccard Distance (LZJD). Fuzzy hashes are crucial because they allow for the comparison of two files to determine their degree of similarity, even if they are not byte-for-byte identical. This is invaluable for identifying variants, related malware families, or code reuse without requiring deep manual analysis. Unlike services like VirusTotal, which may display these hashes but not allow searching on them, MalwareDB enables direct query capabilities based on similarity metrics.

1. Automated Feature Extraction: The system incorporates parsers for various file types, including executables and documents. These parsers automatically extract relevant features such as the number and names of sections in an executable, or the title and page count of a document. This automated feature extraction is vital for building rich metadata profiles for each sample, which can then be used for more advanced analysis, classification, and machine learning model training.

1. Contextual Data Enrichment: MalwareDB allows users to store critical contextual information alongside samples. This includes the source or origin of the file (e.g., VirusShare, VirusTotal, specific client engagements, or campaign data), and labels (e.g., AV detections, inferred malware family information, potentially derived from tools like Clarify). This distinction between public and private data sources is crucial for managing sharing policies and ensuring compliance. The ability to tie into external services like VirusTotal optionally further enriches the contextual data, providing insights into a file's detection status and prevalent labels.

1. Robust and Optimized Backend: The project leverages PostgreSQL as its backend database. Zak emphasized PostgreSQL's speed, open-source nature, and, critically, its extensibility. This extensibility is exploited to implement database extensions that optimize the calculation and comparison of similarity metrics directly within the database. This approach significantly enhances performance, allowing the database to efficiently handle similarity searches across large datasets without requiring all data to be loaded into memory or processed externally.

1. Secure Sample Storage: An innovative optional feature is the encryption of stored samples. This addresses the common problem of endpoint security software automatically deleting malicious files, which can severely hinder research and analysis efforts. By encrypting samples and managing the keys, MalwareDB ensures that files are protected from accidental deletion while remaining accessible to analysts via the system's retrieval mechanism.

1. Interoperability and Open-Source Philosophy: MalwareDB is designed as a client-server application with a standard HTTP interface, making it highly interoperable. This means it can easily integrate with existing security tools and workflows, serving as an additional data store or an alternative file storage mechanism rather than a siloed solution. Its entirely open-source nature, written in Rust for performance and memory safety, encourages community contributions and adoption.

Technical Deep Dive

▶ Watch: Optional encryption to protect samples from AV software (6:00)

MalwareDB's architecture is a testament to modern, robust software engineering principles, designed for both performance and extensibility in the demanding field of malware analysis. At its core, it operates as a client-server application, adhering to a standard HTTP interface. This design choice is critical for its stated goal of interoperability, allowing other systems and tools to easily interact with MalwareDB, either by contributing samples and metadata or by querying its extensive repository. The server component, running on a designated machine, orchestrates all data storage, retrieval, and processing tasks.

The choice of PostgreSQL as the backend database is a deliberate and strategic decision. Richard Zak highlighted several advantages: its inherent speed, its open-source nature, and, most importantly, its robust extensibility. This extensibility is a cornerstone of MalwareDB's efficiency, particularly for handling similarity searches. Traditionally, comparing fuzzy hashes across a large dataset would involve retrieving all relevant hashes and performing comparisons in application logic, which can be computationally intensive and slow. However, with PostgreSQL, custom extensions can be developed and integrated directly into the database engine. These extensions can then perform the complex similarity calculations (e.g., comparing ssdeep hashes or LZJD values) directly at the database level, significantly optimizing query performance by pushing computation closer to the data. This "database doing the work" approach ensures that similarity searches remain fast and scalable even with millions of samples.

Data stored within MalwareDB goes beyond just the raw file. Each sample is associated with a rich set of metadata:

• Cryptographic Hashes: Standard identifiers like SHA-256, MD5, and SHA-1 are calculated and stored for precise identification.
• Fuzzy Hashes: ssdeep (context triggered piecewise hash) and LZJD (Lochard's Jaccard Distance) are generated to enable similarity comparisons. These hashes capture structural or content similarities, allowing analysts to find variants or related samples even if minor modifications have been made.
• Extracted Features: MalwareDB includes rudimentary parsers for common file types, such as executables and documents. For executables, features like the number of sections, their names, and potentially imports/exports can be extracted. For documents, metadata like title, author, and page count are relevant. While described as "not quite production-ready" yet, these parsers lay the groundwork for automated, deep feature extraction, which is crucial for machine learning applications.
• Contextual Metadata: This includes the origin of the sample (e.g., VirusShare, a specific client, or an internal campaign), acquisition timestamp, and labels. Labels can be derived from external sources like VirusTotal (indicating AV detections) or processed by tools like Clarify to infer malware family names. This contextual data is vital for understanding the provenance and significance of each sample, and for managing data sharing policies (e.g., differentiating between public and private data).

An innovative technical feature is the optional encryption of stored samples. This directly addresses the practical problem of endpoint security software deleting malware samples, which are crucial for research. MalwareDB implements a mechanism where the raw sample files can be encrypted at rest. When a user requests a file, MalwareDB handles the decryption process, managing the encryption keys internally. This ensures that the samples are protected from automated deletion by host-based security solutions, preserving the integrity of the research dataset.

The entire MalwareDB system, including both client and server components, is written in Rust. Richard Zak expressed his strong preference for Rust, citing its performance characteristics, memory safety guarantees, and his personal experience of increased productivity with the language. Rust's ability to provide C-like performance without the typical memory safety pitfalls (like buffer overflows or use-after-free errors) makes it an excellent choice for a security-sensitive application that needs to handle large amounts of data efficiently. The project is fully open source and available on GitHub, with installation facilitated via cargo install malwaredb for Rust developers.

Looking ahead, the roadmap includes significant technical enhancements. One key area is the pre-loading of data from public sources like VirusShare. The idea is that upon initial setup, users could opt to populate their MalwareDB instance with a foundational dataset of known malware, providing immediate value for comparison and analysis. The most ambitious future technical development involves integrating machine learning capabilities. This would leverage the extracted features and contextual metadata to:

• Cluster samples: Identify previously unknown relationships between malware samples, potentially revealing new families or threat actor connections.
• Train custom classification models: Allow users to train their own "goodware or badware" models based on their specific datasets, continually improving accuracy as more data is added.
• Identify related threat actors: By analyzing patterns in clustered samples and their metadata, AI could help infer connections between different campaigns or actors.

While the human remains responsible for the final determination of maliciousness, the ML component is envisioned as a powerful aid for pattern recognition and insight generation, making the vast datasets more actionable.

Demo / Proof of Concept

▶ Watch: Key benefits: tracking, similarity, and team collaboration (6:30)

While the talk did not feature a live demonstration or a detailed proof-of-concept walkthrough of MalwareDB in action, Richard Zak clearly articulated the intended functionality and user interaction paradigms. The presentation focused on the architectural design, core features, and the problem MalwareDB aims to solve, rather than a step-by-step product demo.

However, the speaker described how a user would interact with the system to achieve its primary goals:

1. Data Ingestion: The initial step for any user would be to load samples into the database. This process would involve submitting files, which MalwareDB would then process to generate cryptographic hashes, fuzzy hashes (ssdeep, LZJD), extract features using its parsers, and store any user-provided contextual metadata (e.g., origin, labels).
2. Querying by Similarity: Once data is loaded, a key interaction would be to query the database using a similarity hash from a new or known sample. For instance, an analyst encountering a new suspicious file could submit its ssdeep hash to MalwareDB and ask, "What else in my database is similar to this?" The system, leveraging its PostgreSQL extensions for optimized fuzzy hash comparisons, would then return a list of related samples, along with their associated metadata, features, and origins. This capability is central to identifying variants or related threats quickly.
3. Future Machine Learning Interaction: Although not yet implemented, Zak outlined future interactions with the planned machine learning components. Once activated, users would be able to leverage the stored data to train custom models for tasks like clustering samples to identify hidden relationships or classifying files as goodware or malware based on their organizational context and data. This would allow for more automated and intelligent insights into the evolving threat landscape.

Zak emphasized that the system is "mostly feature complete" in its current iteration, indicating that the core storage, hashing, and metadata management capabilities are functional, even if not explicitly demonstrated live during the talk. He also mentioned having "some machines in the basement" set up for running the system and working on integrating the ML components, suggesting active development and practical application of the project.

Defensive Implications

▶ Watch: Using Postgres extensions for efficient similarity metrics (8:10)

MalwareDB presents several significant defensive implications for security teams, researchers, and organizations grappling with the constant influx of cyber threats:

1. Enhanced Threat Intelligence and Context: Defenders can build a richer, more actionable threat intelligence repository. By storing not just the malware itself, but also its origin (e.g., client engagement vs. public source like VirusShare), acquisition date, and associated labels (AV detections, inferred malware families), teams gain crucial context. This allows for better risk assessment, understanding the scope of campaigns, and determining if a newly discovered threat is related to past incidents or specific threat actors. The ability to differentiate between public and private data also helps manage intelligence sharing responsibly.

1. Accelerated Incident Response and Triage: During an incident, time is critical. MalwareDB's ability to perform similarity searches using fuzzy hashes (ssdeep, LZJD) allows responders to quickly determine if a newly observed malicious file is a known variant or shares significant code with previously analyzed samples. This can drastically reduce analysis time, allowing teams to leverage prior knowledge, existing detection rules, and remediation strategies for related threats, rather than starting from scratch. "Have I seen this before, or something similar?" becomes a rapidly answerable question.

1. Improved Malware Analysis Workflow: For malware analysts, MalwareDB centralizes and contextualizes samples, reducing redundant effort. Instead of re-analyzing every new variant, analysts can quickly pull up existing metadata, extracted features, and even collaborative notes associated with similar samples. The automated extraction of features (e.g., PE section names, document titles) provides immediate insights without manual reverse engineering, streamlining the initial triage phase.

1. Facilitated Team Collaboration: MalwareDB functions as a shared knowledge base for security teams. Instead of individual analysts maintaining disparate collections of samples and notes, a centralized MalwareDB instance allows teammates to access and benefit from each other's analysis. This fosters a more collaborative environment, ensures consistency in findings, and prevents "reinventing the wheel" when similar threats emerge.

1. Foundation for Advanced Machine Learning Defense: The structured storage of samples, features, and labels provides an ideal dataset for developing and training custom machine learning models. Defenders can use MalwareDB to:

• Train goodware/badware classifiers: Tailor detection models to their specific environment and threat landscape, improving accuracy over generic solutions.
• Cluster malware: Identify new or unknown malware families, potentially uncovering sophisticated threat actor campaigns by grouping related samples that might not be detected by traditional signatures.
• Identify relationships: Uncover connections between seemingly disparate samples, threat actors, or attack campaigns, leading to a more holistic understanding of the adversary.

1. Data Integrity and Sample Preservation: The optional encryption feature directly addresses a common pain point: the accidental deletion of critical malware samples by endpoint security software. By managing keys and decrypting on demand, MalwareDB ensures that valuable research and incident response data remains intact and accessible, preventing loss of intelligence due to automated security measures.

1. Interoperability with Existing Security Stacks: Designed with an HTTP interface, MalwareDB is not a siloed solution. It can be integrated into existing security pipelines, feeding data to or receiving data from other tools like SIEMs, threat intelligence platforms, or automated analysis sandboxes. This flexibility allows organizations to augment their current defensive capabilities without requiring a complete overhaul of their infrastructure.

In essence, MalwareDB empowers defenders by transforming raw, overwhelming volumes of files into an organized, searchable, and context-rich repository. It enables proactive threat hunting, faster incident response, and more intelligent security operations through better data management and the groundwork for AI-driven insights.

Key Takeaways

• Comprehensive Malware Management: MalwareDB is an open-source client-server system designed to efficiently store, manage, and contextualize vast collections of malicious and benign files, addressing a critical gap in existing security tools.
• Advanced Similarity Analysis: It leverages both cryptographic and fuzzy hashes (e.g., ssdeep, LZJD) to enable rapid similarity searches, allowing defenders to quickly identify variants, related malware, and shared code without manual re-analysis.
• Robust & Performant Backend: Utilizing PostgreSQL with custom extensions, MalwareDB optimizes database-level calculations for similarity metrics, ensuring high performance and scalability even with millions of samples.
• Contextual Data Enrichment: The system stores extensive metadata, including file origins, labels (e.g., AV detections), and automatically extracted features, providing crucial context for threat intelligence and incident response.
• Secure & Collaborative: MalwareDB offers optional file encryption to protect samples from accidental deletion by endpoint security software and is designed for team collaboration, serving as a centralized repository for shared analysis.
• Foundation for AI/ML: It lays the groundwork for future machine learning integration, enabling automated clustering of samples, identification of threat actor relationships, and training of custom goodware/malware classification models based on an organization's unique data.

About the Speaker(s)

Richard Zak is a dedicated programmer and a researcher specializing in malware and machine learning. He is the creator and primary developer behind MalwareDB, an open-source project he initiated during the pandemic as a personal endeavor. This talk at ShmooCon marked his first presentation at the conference, though not his first public speaking engagement overall. Zak's motivation for developing MalwareDB stems directly from his professional experience, where he encountered significant challenges in managing "obscene amounts" of malware samples—often terabytes across millions of files—for analysis and machine learning model training. He is a passionate advocate for open source, a long-time Linux user, and a proponent of the Rust programming language, which he utilized for MalwareDB due to its performance and memory safety benefits. He envisions MalwareDB as a tool to empower the security community, bridging the gap for effective malware data bookkeeping.