Speeding Up Vulnerability Triage: Automating Context Retrieval with AI Agents

Speeding Up Vulnerability Triage: Automating Context Retrieval with AI Agents

Speakers: Jorge Gimenez (Kraken)

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=gUEjbWGH1Ro

Overview

In the contemporary cybersecurity landscape, vulnerability management teams are frequently overwhelmed by a deluge of alerts from various scanning tools, leading to significant challenges in prioritization and remediation. Jorge Gimenez, a vulnerability management expert at Kraken, addressed this critical issue in his VulnCon talk, "Speeding Up Vulnerability Triage: Automating Context Retrieval with AI Agents." The presentation meticulously dissects the inherent flaws in traditional vulnerability triage processes, arguing that the core problem is not detection, but rather the efficient aggregation of context necessary to make informed decisions.

Gimenez introduces an innovative approach centered on the strategic deployment of AI agents to automate and enhance the vulnerability triage workflow. His research demonstrates how a sophisticated architecture, leveraging an orchestrator with specialized sub-agents, can dramatically improve the accuracy and speed of assessing vulnerability impact. By shifting the focus from simply identifying vulnerabilities to understanding their true exploitability and business relevance, this methodology aims to empower security teams to concentrate on high-impact issues, thereby significantly bolstering an organization's security posture.

This talk is particularly pertinent for security professionals grappling with the scale and complexity of modern cloud and container environments. It provides a practical, data-driven perspective on how advanced Large Language Models (LLMs) can be harnessed to cut through the noise, reduce manual effort, and transform vulnerability management from a reactive, resource-intensive task into a proactive, intelligent process.

Background

▶ Watch: Why vulnerability triage is currently broken (1:06)

Jorge Gimenez began his journey in vulnerability management from an offensive security background, quickly realizing that the skills honed in red teaming did not directly translate to effective vulnerability prioritization. He describes the initial experience as being "overwhelmed by the data," likening the continuous influx of alerts to a "TikTok effect" where focus becomes elusive. The fundamental challenge, he posits, is that while scanners are adept at identifying what is vulnerable, they conspicuously fail to articulate if it matters in a given operational context.

This problem is exacerbated by several factors:

• Data Overload: Organizations are inundated with alerts from diverse sources like cloud scanners, container scanners, and application security tools. Each data source, while providing crucial context, also adds to the sheer volume, making it nearly impossible for human analysts to process manually.
• Lack of Context: A CVE alert in a virtual machine doesn't reveal if the VM is internet-exposed, if the vulnerable application code is actually in use, or if it's shielded behind a VPN. Similarly, container scanning often generates immense noise, making it difficult to discern critical issues from benign findings.
• High False Positive Rates: A significant portion of alerts turn out to be false positives, forcing security engineers to spend considerable time manually separating real threats from noise. This repetitive, manual workflow does not scale with the increasing number of data sources and alerts.
• Manual Triage Inefficiency: A typical manual triage process for a container image CVE involves analyzing the CVE, checking its Exploit Prediction Scoring System (EPSS) score, determining if it's exploitable in the specific environment (e.g., for containers vs. phones), checking if the image is deployed, and verifying if the vulnerable dependency is actively used at runtime. This multi-step process for each finding is time-consuming and unsustainable.

Gimenez's core assumption underpinning his research is that vulnerability triage is fundamentally a context aggregation problem, not a detection problem. The data exists, but the ability to effectively gather, synthesize, and interpret it at scale is "broken." This realization prompted the exploration of AI agents as a potential solution to scale this complex, context-dependent workflow.

Key Findings

▶ Watch: Initial approach: Single LLM agent and its pitfalls (11:00)

The initial exploration into automating vulnerability triage with AI agents revealed critical limitations of a simplistic, single-agent approach:

1. Context Rot and Overflow: The primary failure point of the initial single LLM agent architecture was its inability to manage the vast amounts of raw data retrieved from multiple sources. As each tool sequentially dumped its output into the agent's context window, the window rapidly overflowed. This phenomenon, described by Anthropic as context rot, leads to a decrease in recall accuracy as tokens increase. Earlier, crucial evidence was silently dropped, and the model became biased towards the most recent data, effectively losing sight of its original goal.
2. Hallucination and Incorrect Conclusions: With context being lost or diluted, the AI agent frequently hallucinated or fabricated evidence, leading to confidently wrong conclusions. The lack of a clear reasoning trace made it impossible to debug where these mistakes originated.
3. Inflexibility with Complex Workflows: The initial workflow-based approach, which called tools sequentially, proved too rigid for the unpredictable and complex nature of vulnerability triage. It required anticipating and defining an unmanageable number of edge cases, which is impractical for resource-constrained security teams.
4. The Solution: Orchestrator with Specialized Sub-Agents: The most significant finding was that a different architectural abstraction was required. The successful approach involved an orchestrator LLM, operating on the ReAct framework (Reason, Act, Observe), which dynamically decides what tool to call next based on its current understanding of the problem. Crucially, each "tool" is itself a specialized sub-agent, designed for a specific task (e.g., cloud posture, runtime security, source code analysis).
5. Independent Context for Sub-Agents: A key innovation was giving each sub-agent its independent context window. This means sub-agents start from scratch, process specific data, and return synthesized, relevant findings to the orchestrator, rather than raw, voluminous output. This significantly reduces the orchestrator's context burden and prevents overflow.
6. Stack-Aware Design: For optimal effectiveness, the sub-agents must be stack-aware, meaning they are tailored to the organization's specific environment, naming conventions, data usage patterns, and inter-source mappings (e.g., how container images map to source code repositories). A generic agent would incur higher costs and yield less accurate results due to a lack of environmental understanding.
7. Auditability and Observability: To build trust and allow for continuous improvement, the system was designed with robust auditability. It stores the full reasoning trace of both the orchestrator and sub-agents, including all tool calls and their outputs. This allows analysts to validate decisions, identify failure points, and demonstrate compliance.
8. Prompt Engineering Evolution: The speaker noted a shift in prompt engineering strategy. Initially, prompts were highly specific and lengthy. However, better results were achieved by treating prompts as general guidelines, allowing the LLM's inherent reasoning capabilities to guide its actions rather than rigidly dictating them.

These findings highlight that while AI offers immense potential for automating security tasks, its effective deployment requires careful architectural design, context management, and environmental customization.

Technical Deep Dive

▶ Watch: Real-world example: Automated subdomain takeover triage (25:30)

The technical journey presented by Jorge Gimenez details an evolution from a naive, yet intuitive, application of LLMs to a sophisticated, multi-agent architecture.

The Initial, Flawed Approach: Single LLM Agent

The first attempt involved a single, powerful LLM agent equipped with a comprehensive set of tools, each designed to interact with a specific data source (e.g., cloud posture, runtime security, source code). The workflow was envisioned as a sequential process:

1. Ingest Raw Alert: The LLM received unstructured alert data, leveraging its ability to extract relevant information.
2. Sequential Tool Calls: The agent would then call tools one after another:

• Retrieve cloud posture information about the asset.
• Gather runtime security data.
• Extract source code details.

1. Context Accumulation: Each tool's raw output (often large JSON objects or extensive logs) was directly appended to the LLM's context window for the current session.
2. Decision Making: The expectation was that the LLM, with all this aggregated context, would make an informed triage decision.

This approach quickly encountered severe limitations:

• Context Window Overflow: The sheer volume of raw data from multiple sources, especially from verbose outputs like those from security configuration posture (MCP) tools, rapidly filled and exceeded the LLM's finite context window.
• Context Rot: As the context window filled, earlier, potentially critical information was effectively "forgotten" or pushed out of the model's effective attention span. Anthropic's observation that "as tokens increase, recall accuracy decreases" was directly experienced. The model began to suffer from a bias towards the end of the context, focusing only on the last few pieces of information.
• Hallucination: With incomplete or skewed context, the LLM frequently generated confident but incorrect conclusions, fabricating evidence to support its reasoning. This made the output unreliable and untrustworthy.
• Lack of Flexibility: The sequential, workflow-like execution was too rigid. Real-world triage often requires dynamic decision-making, where the next step depends on the outcome of the previous one. The single agent couldn't adapt to complex logical branches or unforeseen scenarios without extensive, unmaintainable edge-case definitions.

The Refined Architecture: Orchestrator with Specialized Sub-Agents

Recognizing the shortcomings of the single-agent model, Gimenez's team pivoted to a more modular and flexible architecture: an orchestrator LLM coordinating multiple, specialized sub-agents.

1. The Orchestrator:

• At its core, the orchestrator is an LLM powered by the ReAct framework: Reason, Act, and Observe. This means:
• Reason: The orchestrator analyzes the current state, the alert, and the information gathered so far to decide the next logical step. It doesn't follow a predefined sequence.
• Act: It selects and invokes the most appropriate sub-agent (which it treats as a "tool") for the current task.
• Observe: It receives and evaluates the synthesized output from the sub-agent, updating its understanding of the problem before reasoning again.
• This dynamic, iterative process allows the orchestrator to adapt to the evolving context of the triage, mimicking how a human analyst would investigate.

1. Specialized Sub-Agents:

• Each "tool" in this architecture is no longer just an API wrapper; it is itself a fully capable LLM-powered sub-agent.
• Each sub-agent is designed for a specific task (e.g., CloudPostureAgent, RuntimeSecurityAgent, SourceCodeAgent, CVEEnrichmentAgent).
• Crucially, each sub-agent operates with its independent context window. When the orchestrator calls a sub-agent, that sub-agent starts with a clean slate, processes the specific data relevant to its task, and performs its own ReAct cycle.
• Instead of returning raw, verbose data, the sub-agent synthesizes findings and returns only the most relevant, concise information to the orchestrator. This dramatically reduces the data volume passed back to the orchestrator, preventing its context window from overflowing.
• Context Compaction is still performed within sub-agents if needed, especially when dealing with a large number of assets (e.g., 30-40 assets for investigation).

1. Interoperability and Stack-Awareness:

• While sub-agents do not directly communicate, they interoperate through the orchestrator. Information gathered by one sub-agent (e.g., an asset ID from the Cloud Posture agent) can be passed by the orchestrator as input to another sub-agent (e.g., the Source Code agent) to continue the investigation.
• A critical design principle is that sub-agents must be stack-aware. This means they are explicitly configured and trained to understand the organization's specific environment, including:
• Naming conventions.
• How different data sources map to each other (e.g., linking a container image registry entry to its corresponding Git repository).
• Specific ways teams use and interact with various data sources (e.g., GitHub for shift-left security or code storage).
• This customization ensures higher accuracy and relevance, as a generic agent would make assumptions not applicable to a unique enterprise environment.

1. Auditability and Observability:

• To ensure trust and facilitate debugging, the system meticulously logs the entire reasoning trace. This includes:
• The orchestrator's decisions.
• All tool calls made by both the orchestrator and sub-agents.
• The raw and synthesized outputs from each call.
• This comprehensive logging allows security engineers to review the AI's logic, identify where errors occurred (e.g., an agent attempting to access non-existent network logs for port 23), and continuously improve the system. It also provides essential traceability for audit and compliance purposes, validating risk acceptance decisions.

1. Prompt Engineering:

• Gimenez emphasized a shift in prompt engineering strategy. Instead of providing overly specific, long prompts, the agents are given more general guidelines. The prompts define the goal, the context of the environment, and the desired outcome, but they allow the LLM to exercise its reasoning capabilities to determine the exact steps. This leverages the LLM's strengths and prevents over-constraining its problem-solving abilities.

The orchestrator framework used by Kraken is LangGraph, which facilitates the creation of such dynamic, reactive agentic systems. While the solution is admittedly more expensive in terms of LLM token usage due to multiple calls and investigations, the value derived from drastically reduced manual effort and improved prioritization far outweighs the cost, especially when considering the expense of human analyst time. The system's design anticipates future improvements in LLM capabilities, including larger context windows and enhanced reasoning, which will only further boost its effectiveness.

Demo / Proof of Concept

▶ Watch: Key lessons: Tailored, scoped, and observable agents (36:00)

Jorge Gimenez presented two compelling real-world examples to illustrate the effectiveness of the orchestrator-sub-agent architecture in automating vulnerability triage.

Example 1: Subdomain Takeover

The Problem: A raw alert indicates a subdomain takeover vulnerability. Such an alert typically states that a DNS record points to a third-party resource, but the organization no longer hosts content there, potentially allowing an attacker to claim the resource and host malicious data. The initial alert is sparse, lacking crucial details like:

• Is the DNS record still active?
• What was the exact output of the scanner that triggered this?
• How was the record provisioned? Can it be traced back to infrastructure as code (IaC)?

The Automated Triage Process:

1. Ingestion: The raw subdomain takeover alert is fed into the orchestrator in natural language.
2. Initial Reasoning: The orchestrator identifies the immediate need: determine if the DNS record is active and if the cloud resource is still alive. It knows the alert came from a Cloud Posture scanner.
3. Cloud Posture Sub-Agent Call: The orchestrator invokes the CloudPostureAgent to investigate. This sub-agent checks the DNS record, its pointing destination, and analyzes the full output of the scanner to confirm the vulnerability's current state.
4. Orchestrator Update: The CloudPostureAgent returns synthesized findings to the orchestrator: "Domain is still active, but there's nothing exposed in the third-party provider." This updates the orchestrator's understanding.
5. Source Code Sub-Agent Call (IaC Tracing): Knowing the domain is active, the orchestrator's next logical step, based on the company's established practices (stack-awareness), is to trace its provisioning. It calls the SourceCodeAgent to identify which Git repositories reference this domain, looking for IaC definitions.
6. Final Evidence & Decision: The SourceCodeAgent returns the affected source code path. The orchestrator then reasons: "Vulnerability confirmed, it's exploitable, and the affected code path is identified."

Outcome: From a vague alert, the system produces a highly actionable remediation. The alert is marked as exploitable, the full traceability of the investigation is logged, and the exact reference in the source code (e.g., a Terraform file) needing remediation is attached as evidence. This process, which would typically consume 1 to 1.5 hours of an analyst's time, is now automated, significantly scaling the team's capabilities.

Example 2: New CVE (Unauthenticated RCE in Container Dependency)

The Problem: A new critical CVE (e.g., an unauthenticated RCE) is announced, affecting a dependency found in container images. The initial alert provides only the CVE number, with no information on affected assets or their deployment status. A colleague might raise an alarm, but without concrete data. Key questions that need answers:

• Which container images/assets are affected?
• Are these images actually deployed in the environment? (Many images might be scanned but never deployed).
• Is the vulnerable dependency actually used at runtime within the deployed images? (Often, dependencies are present but not invoked).
• Of the deployed images, how many are internet-facing, and what is their business criticality?

The Automated Triage Process:

1. Ingestion: The CVE alert is sent to the orchestrator.
2. Initial Reasoning & Inventory Retrieval: The orchestrator first extracts the CVE details and then determines the need to find all potentially affected inventory. It calls various sub-agents (e.g., ContainerScanningAgent, AssetInventoryAgent) to gather a comprehensive list of assets potentially containing the CVE.
3. Deployment Status & Exposure Assessment: With the list of potentially affected assets, the orchestrator prioritizes by checking deployment status and exposure.

• It calls the CloudPostureAgent and RuntimeSecurityAgent to determine which container images are actually deployed.
• It also ascertains their exposure (internet-facing vs. internal) and business criticality (production vs. staging).
• Result: Out of 10 potentially affected containers, only 5 are deployed. Of these, 2 are production systems, and only 1 is internet-facing. This drastically reduces the scope.

1. Dependency Usage Verification: The orchestrator then needs to confirm if the vulnerable dependency is actively used.

• It invokes the SourceCodeAgent, passing the identified deployed container images. Due to the stack-aware design, the agent knows how to map container images to their respective source code repositories.
• Result: The SourceCodeAgent reveals that 3 containers actively use the malicious dependency. One container is a mirror image, for which the source code is unavailable, requiring manual investigation.

1. Final Prioritization & Action Plan: The orchestrator synthesizes all gathered information into a clear, prioritized action plan:

• Immediate Action: Prioritize remediation of the production asset that is exposed to the internet.
• High Priority: Address the production and staging containers that are internally exposed.
• Manual Investigation: Investigate the mirror container that is internally exposed but lacks traceable source code.

Outcome: Instead of manually sifting through 10 containers, the security team is directed to investigate only one (the mirror image) and is provided with three distinct, prioritized remediation actions. This transforms a daunting, manual effort into a focused, impactful response, demonstrating the power of automated context aggregation in scaling vulnerability management.

Defensive Implications

▶ Watch: Q&A: Ensuring agent accuracy and reducing hallucinations (43:00)

The adoption of an AI-driven orchestrator with specialized sub-agents for vulnerability triage carries profound implications for defensive security operations:

1. Enhanced Prioritization and Reduced Alert Fatigue: The most significant benefit is the ability to move beyond raw scanner alerts to a context-rich understanding of vulnerabilities. By automatically correlating CVE data with deployment status, runtime usage, exposure, and business criticality, security teams can prioritize remediation efforts on true threats with the highest business impact, rather than getting bogged down by noise or false positives. This directly combats alert fatigue.
2. Significant Reduction in Manual Effort: Automating the laborious process of gathering context from disparate data sources (cloud posture, runtime security, source code, asset inventory) frees up security engineers' time. Tasks that previously took hours of manual investigation can now be completed in minutes, allowing analysts to focus on higher-value activities like complex threat hunting, architectural reviews, or strategic security improvements.
3. Improved Accuracy and Consistency: By providing comprehensive, synthesized context, the AI agents help ensure more accurate vulnerability assessments. The structured, auditable reasoning process reduces human error and ensures a consistent approach to triage across the organization.
4. Faster Time to Remediation: With clear, actionable remediation paths (e.g., direct links to affected source code, prioritized lists of critical assets), development and operations teams can address vulnerabilities more quickly and efficiently, reducing the window of exposure.
5. Increased Auditability and Compliance: The system's inherent design for logging the full reasoning trace, tool calls, and outputs provides an invaluable audit trail. This transparency is crucial for demonstrating due diligence to auditors, validating risk acceptance decisions, and continuously improving the security program.
6. Tailored Security Operations: The emphasis on stack-aware sub-agents means that the AI solution is not a generic, off-the-shelf tool but one deeply integrated with an organization's unique environment, naming conventions, and data usage patterns. This customization ensures maximum relevance and effectiveness, adapting to the specific nuances of an enterprise's infrastructure and development practices.
7. Proactive Security Posture: By quickly identifying and prioritizing exploitable vulnerabilities, security teams can shift from a reactive "whack-a-mole" approach to a more proactive stance, addressing critical risks before they can be exploited by adversaries.
8. Scalability for Growing Environments: As cloud environments become more dynamic and complex, and the volume of assets and data sources continues to grow, human-centric triage becomes increasingly unsustainable. AI agents provide the necessary scalability to keep pace with modern infrastructure, ensuring that security operations do not become a bottleneck.

In essence, this agentic system transforms vulnerability triage from a labor-intensive, often overwhelming task into an intelligent, efficient, and highly effective component of an organization's overall defensive strategy.

Key Takeaways

• Vulnerability triage is a context aggregation problem, not a detection problem. The challenge lies in synthesizing disparate data sources to understand true risk, not merely identifying vulnerabilities.
• Single LLM agents struggle with context management. Attempts to aggregate all raw data into a single agent's context window lead to "context rot," hallucination, and unreliable conclusions.
• Orchestrator with specialized sub-agents is the effective architecture. A dynamic orchestrator (using ReAct framework) that coordinates specialized sub-agents, each with independent context and the ability to synthesize findings, is crucial for scaling complex triage workflows.
• Tailoring AI solutions to the environment is paramount. For maximum effectiveness, AI agents must be "stack-aware," understanding an organization's unique naming conventions, data mappings, and operational processes. Generic solutions often fail or incur higher costs.
• Auditability and observability are non-negotiable. Logging the full reasoning trace, tool calls, and outputs of AI agents is essential for building trust, debugging errors, continuous improvement, and demonstrating compliance to auditors.
• Prompt engineering should guide, not dictate. Instead of overly specific prompts, providing general guidelines allows LLMs to leverage their reasoning capabilities more effectively, leading to more robust and adaptable solutions.

About the Speaker(s)

Jorge Gimenez is a security professional working in the vulnerability management team at Kraken. With a background rooted in offensive security, he brings approximately five years of experience in red teaming and security consulting. Around two years prior to this talk, he transitioned into vulnerability management, where he has been actively involved in automating processes he finds repetitive or "boring." His work at Kraken focuses on leveraging AI to streamline security operations and enable teams to concentrate on the most impactful aspects of their business.