Eiffel: A Tool to Oversee Incident Response From the Heights

Eiffel: A Tool to Oversee Incident Response From the Heights

Speakers: Óscar Salvador (Lead Analyst, Cybersecurity Agency of Catalonia); Juan Gonzalez (Head of CERT, Cybersecurity Agency of Catalonia)

Conference: FIRSTCON

YouTube: https://www.youtube.com/watch?v=jSQppCu7fK4

Overview

In an era of rapidly evolving cyber threats and increasingly complex digital infrastructures, the Cybersecurity Agency of Catalonia (ACSC) has developed "Eiffel," a sophisticated platform designed to revolutionize incident response (IR) and threat hunting across a highly heterogeneous and geographically dispersed constituency. Presented by Juan Gonzalez, Head of the Catalonia CERT, and Óscar Salvador, Lead Analyst, this talk at FIRSTCON unveiled Eiffel as a critical tool for gaining a holistic, top-down view of security incidents, moving beyond traditional artifact-centric analysis to an event-focused approach.

Eiffel addresses the significant challenges faced by regional CERTs, which often oversee a multitude of disparate entities—from government departments and universities to hospitals and municipalities—each with unique technological stacks and security postures. The platform integrates diverse data sources into a normalized data model, leverages advanced analytics and AI, and incorporates tools like Velociraptor to provide a comprehensive, actionable understanding of security events. This initiative is particularly vital given the industrialization of cyberattacks and the continuous pressure on defenders to scale their capabilities.

The core premise of Eiffel is to provide "a view from the top of the Eiffel Tower," allowing analysts to identify critical security events and affected assets across an entire region, rather than being bogged down in individual machine forensics. By consolidating and analyzing vast volumes of data, Eiffel aims to serve as a force multiplier for incident response teams, enabling quicker detection, more efficient investigation, and proactive threat hunting, ultimately enhancing the cybersecurity resilience of Catalonia's public sector.

Background

▶ Watch: Catalonia CERT's scope and challenges (2:00)

The Cybersecurity Agency of Catalonia (ACSC) provides cybersecurity services to the regional government, encompassing a broad and diverse constituency. This includes a vast array of government information systems, 60 hospitals, 10 public universities, and over 900 municipalities. This sprawling and highly heterogeneous scope—where each entity possesses its own unique technology, policies, and operational decisions—presents substantial challenges for unified cybersecurity management and incident response.

Prior to 2018, the Catalonia CERT primarily engaged in physical incident response, involving on-site visits to clone disks, dump memory, and conduct forensic analysis in a laboratory setting. This approach, while thorough, was inherently slow and unscalable. The landscape dramatically shifted with the onset of ransomware attacks, notably Ryuk, starting in 2018. The COVID-19 pandemic in 2020 further exacerbated this situation, forcing a rapid transition to remote work environments. This opened "plenty of doors" without adequate controls, creating a "perfect storm" that ransomware capitalized on, making traditional physical IR impractical and inefficient.

In response to these evolving threats and operational constraints, the CERT recognized the urgent need to handle incidents remotely. This necessitated the adoption of new technology stacks, standardized procedures, and orchestration mechanisms. This initial effort, a "self-contained security incident management platform," was internally dubbed "Paris"—a playful acronym for its Catalan translation. While "Paris" laid the groundwork for remote operations and industrialized evidence processing, it soon became apparent that it still focused heavily on individual assets. Analysts could efficiently process data from single machines, but lacked a global overview of incidents impacting multiple entities or the entire region. This dependency on deep human expertise for correlating disparate findings from heterogeneous outputs remained a significant bottleneck. The challenge was clear: how to elevate the perspective from street-level forensic detail to a panoramic view of the entire cyber landscape, much like observing Paris from the Eiffel Tower.

Key Findings

▶ Watch: Velociraptor integration benefits for incident response (12:30)

The development of Eiffel represents a critical evolution in the Catalonia CERT's incident response capabilities, driven by several key findings and strategic shifts. The initial move to remote incident handling, while necessary, highlighted the limitations of a purely asset-centric approach. While an evidence gateway and automatic processing mechanisms successfully industrialized the acquisition and preliminary analysis of remote evidence, generating processed information for analysts, this still left a critical gap: the absence of a global overview of incidents. Analysts could efficiently work on hundreds of machines, but correlating findings across diverse systems remained a manual, human-intensive effort.

This realization led to the core innovation of Eiffel: establishing a top analytic layer to gain a holistic perspective. By ingesting all processed information into a data lake and normalizing it, Eiffel unlocks new capabilities. This includes the ability to query data across the entire constituency, build custom visualizations that quickly highlight relevant activity, and abstract from specific artifact formats. This normalization, based on models like Elastic Common Schema (ECS), ensures that data from various sources—whether from disk images, specialized tools, or Velociraptor—can be consistently understood and analyzed.

Furthermore, Eiffel integrates advanced detection capabilities, moving beyond simple artifact analysis to identify "non-normal activity" through alerts and monitors. This involves leveraging community-sourced Sigma rules (currently almost 3,000) and developing custom rules (65 in recent months), alongside matching extracted indicators (IPs, URLs) against internal and public threat intelligence. This proactive detection transforms DFIR engagement findings into future detections, effectively operationalizing CTI. The platform also began integrating Velociraptor for both incident response and threat hunting, providing a powerful, standardized, and self-sufficient means of data collection and initial triage, offering a transversal view beyond individual assets. Finally, the project recognized the potential of Artificial Intelligence (AI) as a future force multiplier, initiating efforts to develop a forensic assistant capable of summarizing incidents and guiding investigations, while also acknowledging the challenges of model size, cost, and hallucinations.

Technical Deep Dive

▶ Watch: Early steps using AI for forensic assistance (18:00)

Eiffel is an integrated platform built upon several technical pillars designed to centralize, normalize, and analyze security telemetry from a diverse and distributed environment.

At its foundation, the system begins with an evidence gateway. This mechanism is crucial for the remote incident handling paradigm, allowing entities across Catalonia to securely transmit acquired forensic data back to the CERT. Beneath this gateway lies an automatic mechanism responsible for downloading and processing these evidences. This industrializes the initial phase of IR, ensuring that raw data is consistently prepared for analysis and made available to multiple analysts simultaneously. The output of this stage, while processed, still presented a heterogeneous format, necessitating further layers.

The next critical component is the data lake, into which all processed evidence is ingested. To overcome the heterogeneity of source data, Eiffel employs a normalization strategy, primarily leveraging the Elastic Common Schema (ECS). This standardizes data fields and event types, regardless of their origin (e.g., from a Cape image, a specific forensic tool, or Velociraptor). This normalization is fundamental for unlocking advanced analytics, allowing analysts to query and correlate data across the entire constituency. The data lake then feeds a layer of custom visualizations, enabling analysts to quickly identify anomalies and trends without deep dives into raw logs. The ingestion pipelines, particularly those built with Logstash, are substantial, currently encompassing 17,000 lines of code, highlighting the complexity and the challenges of debugging.

A key analytic layer within Eiffel focuses on alerting for non-normal activity. This is achieved through two primary mechanisms:

1. Sigma Rules with Elastalert: Eiffel continuously queries the data lake using Sigma rules, a generic signature format for SIEM systems. The team actively uses almost 3,000 community-sourced Sigma rules and has developed approximately 65 custom rules tailored to specific threats or their environment. These rules are converted to Elastalert format for real-time detection. A signature management process is in place to whitelist effective rules and fine-tune noisy ones, tagging them for better identification.
2. Monitors: These components match extracted indicators of compromise (IOCs) such as IPs, server names, and URLs against both internal Cyber Threat Intelligence (CTI) and public sources. This combination helps identify "needles in the haystack" – critical events that might otherwise be overlooked.

The integration of Velociraptor has been a transformative step. A dedicated Velociraptor server is deployed, with agents deployed across the infrastructure of participating entities. This allows for transversal views of the entire infrastructure, moving beyond individual asset focus. Velociraptor's output, being in JSON format, greatly simplifies the ingestion and normalization process into Eiffel's data lake. It facilitates cross-data analysis from IR engagements and threat hunting efforts. The team also utilizes an offline version of Velociraptor to expedite triage. While highly beneficial, the deployment of Velociraptor introduced new challenges, including the need for additional technology and the creation of a single point of failure with the dedicated server. A notable incident involved a scare when a new data store compression setting in Velociraptor initially appeared to be a catastrophic failure but was resolved by adjusting parameters.

Beyond immediate incident response, Eiffel is designed to collect context information to build a baseline of normal operations. This includes data on hostnames, machine types (DC, workstation, server), IPs, user activity, and processes. This baseline is crucial for identifying deviations during an incident. The platform also measures readiness by assessing factors like audit policy configurations and data retention periods, comparing them against three different baselines tailored for domain controllers, servers, or workstations. This helps predict the ease of response in a given environment.

For threat hunting, Eiffel employs a unique approach given the heterogeneous environment and lack of fine-grained telemetry in many entities. Instead of continuous monitoring, they conduct one-shot Velociraptor hunts either yearly or every six months. This generates a technical baseline that can be leveraged during an incident by comparing current state to the historical baseline to pinpoint relevant events.

Looking to the future, Eiffel is actively integrating Artificial Intelligence (AI). The initial focus is on developing a forensic assistant. Currently, data from Eiffel is being fed into cloud-based AI models, demonstrating preliminary success with limited data. However, concerns about model size limits, computational cost, and AI hallucinations are being addressed. To mitigate these, a pre-processing phase is under development: a Python application that programmatically identifies patterns (peaks, valleys, prevalence anomalies) in the data, generating "signals" that guide the AI, effectively reducing the volume of data presented to the large language models. The platform is also working on integrating Sysmon telemetry into its data model, further enriching the event data. This is being tested in an internal "Mirror Dimension" environment, allowing the CERT to simulate attacks and refine detection capabilities within Eiffel.

Demo / Proof of Concept

▶ Watch: Future plans: open-sourcing Eiffel, Sysmon integration (25:00)

During the presentation, Óscar Salvador provided a live demonstration of a specialized version of Eiffel, tailored to showcase its key functionalities without exposing sensitive operational data. The demo illustrated how Eiffel provides a high-level overview and allows for deep dives into specific incidents.

The interface presented various dashboards and visualizations, categorized by data type, tactic, or other relevant criteria. A core aspect of the demo was the display of context information, derived from the acquired data. This included metrics such as the total number of hosts, users, and events, along with details on machine roles (e.g., Domain Controller, workstation, server). Crucially, the demo highlighted the readiness measurement feature, showing simple visualizations of event channel retention across different systems, indicating how prepared an entity might be for an incident.

The "alerts briefing" dashboard was a central element, distinguishing between alerts from verified rules (65-76 alerts) and those from stable community Sigma rules (9 alerts). It also showed a staggering 24,000 detections from "other Sigmas," underscoring the challenge of noise and the importance of rule fine-tuning.

A pivotal visualization displayed events over time, with clear spots or peaks of relevant activity marked. Óscar demonstrated how to identify a significant peak of activity associated with numerous alerts. By drilling down, the demo revealed that two specific hosts were generating the majority of verified and stable alerts on a particular day. This capability allows analysts to quickly pivot from a high-level anomaly to the precise machines involved.

Further investigation into these hosts revealed that the alerts spanned many different MITRE ATT&CK tactics, strongly suggesting an active and multi-stage compromise. A detailed timeline of alerts for these specific hosts was then presented, visually mapping the progression of the incident. This timeline quickly identified the use of several known malicious tools and malware:

• Rubius (likely a reference to RedLine Stealer or similar)
• SharpHound (part of BloodHound, used for active directory enumeration)
• Rclone (a legitimate tool often abused for data exfiltration)
• Mimikatz (credential dumping tool)
• PowerView (part of PowerSploit, used for Active Directory reconnaissance)

The demonstration effectively showcased Eiffel's ability to transform vast amounts of raw data into actionable intelligence, enabling analysts to rapidly pinpoint affected systems, understand the scope of an attack, and identify the tools and techniques employed by adversaries.

Defensive Implications

▶ Watch: Eiffel as a crucial force multiplier for analysts (28:00)

Eiffel's design and implementation offer several crucial defensive implications for organizations, particularly those managing large, heterogeneous environments like regional CERTs or large enterprises.

Firstly, the emphasis on data normalization using standards like Elastic Common Schema (ECS) is paramount. Defenders should prioritize adopting a consistent data model across all their telemetry sources. This foundational step is critical for enabling cross-source querying, correlation, and the development of robust analytics, ultimately moving beyond siloed data views.

Secondly, the platform highlights the power of leveraging open-source tools and community-driven intelligence. Integrating Velociraptor for data collection and Sigma rules with Elastalert for detection demonstrates how powerful, flexible, and cost-effective solutions can be built. Defenders should actively explore and integrate such tools, contributing back to the community where possible, to enhance their defensive posture. The experience with 3,000 community Sigma rules versus 65 custom ones also underscores the need for a rigorous signature management process, including whitelisting and fine-tuning, to reduce noise and increase detection fidelity.

Thirdly, the concept of a "top analytic layer" providing a global overview is a significant paradigm shift. Instead of focusing solely on individual artifact analysis, defenders should strive to build systems that aggregate and contextualize events across their entire infrastructure. This allows for quicker identification of widespread campaigns, lateral movement, and coordinated attacks, which are often missed when focusing on isolated incidents. This also means moving towards an event-focused analysis rather than solely evidence-focused.

Fourthly, the proactive approach to threat hunting with one-shot Velociraptor hunts to establish technical baselines is highly effective for environments lacking continuous, fine-grained telemetry. Organizations can adopt similar strategies to periodically collect critical system state information, which can then be used for comparative analysis during an incident, drastically reducing investigation time. Coupled with this, measuring readiness through audit policy and retention assessments provides a tangible way to improve incident response preparedness.

Finally, the early integration of Artificial Intelligence (AI), particularly for developing a forensic assistant and employing pre-processing for signal generation, offers a glimpse into the future of IR. Defenders should begin exploring how AI can augment their teams, acting as a force multiplier rather than a replacement for human analysts. However, it's crucial to acknowledge and plan for challenges like data volume, cost, and the potential for AI hallucinations, building in safeguards and human oversight. The plan to integrate Sysmon telemetry and test detections in a "Mirror Dimension" environment also exemplifies a proactive, continuous improvement cycle for detection engineering.

Key Takeaways

• Shift to a Top-Down, Event-Focused IR: Eiffel moves from individual artifact analysis to a holistic, event-driven view across the entire constituency, enabling faster detection and broader understanding of incidents.
• Data Normalization is Critical: Leveraging standards like Elastic Common Schema (ECS) for data ingestion and normalization is fundamental for effective cross-source querying and analytics in heterogeneous environments.
• Open-Source Tools as Force Multipliers: Integration of Velociraptor for data collection/triage and Sigma rules with Elastalert for detection provides powerful, flexible, and cost-effective capabilities.
• Proactive Threat Hunting and Readiness: Implementing one-shot threat hunts to establish technical baselines and measuring readiness (e.g., audit policies, retention) significantly enhance an organization's defensive posture.
• AI for Augmentation, Not Replacement: Early exploration of AI as a forensic assistant, with strategies for pre-processing data and addressing limitations like cost and hallucinations, points towards future efficiencies in IR.
• Continuous Improvement and Detection Engineering: The ongoing development of custom Sigma rules, rigorous signature management, and testing in isolated environments like "Mirror Dimension" are essential for adapting to evolving threats.

About the Speaker(s)

Juan Gonzalez serves as the Head of the CERT for the Cybersecurity Agency of Catalonia. With a background in technology, threat analysis, and incident response, Juan brings extensive experience to his leadership role. He candidly shared that for the past eight years, his primary focus has been raising his three children (ages eight, seven, and four), but he is now enthusiastically re-engaging with the cybersecurity field. His leadership has been instrumental in guiding the Catalonia CERT through significant transformations, including the shift to remote incident handling and the development of innovative platforms like Eiffel.

Óscar Salvador is the Lead Analyst at the Cybersecurity Agency of Catalonia. As the primary architect and implementer of complex incident response solutions, Óscar is responsible for handling the most challenging incidents and ensuring the technical efficacy of the CERT's operations. His deep technical expertise is evident in the detailed explanation of Eiffel's architecture, from data ingestion pipelines to the integration of Velociraptor and the development of advanced analytics. Óscar is the hands-on expert making the sophisticated systems work and driving the technical evolution of the CERT's capabilities.