AI Beyond Triage and Hunting
Chris Botelho (LimaCharlie)
Overview
In his compelling talk at Wild West Hackin' Fest @ Mile High, Chris Botelho, Senior Solutions Engineer at LimaCharlie, presented a deep dive into overcoming the inherent limitations of Large Language Models (LLMs) for practical, operational cybersecurity. Titled "AI Beyond Triage and Hunting," the presentation moves past the common narrative of AI generating images or text, focusing instead on how LLMs can be engineered to perform complex, multi-step security operations at scale. Botelho, a self-professed AI skeptic, shared LimaCharlie's journey in building an open-source framework that enables LLMs to intelligently interact with external tools, manage vast amounts of data, and orchestrate sophisticated defensive actions.

Key moments
- 0:00 Introduction and why LLMs struggle with operations
- 6:20 High token burn: The MCP 'startup tax' problem
- 12:00 Limitations of large context windows: 'middle problem' and cost
- 20:30 Efficiently processing large log files using cloud storage
- 24:00 Sub-agents: Using specialized LLMs for different tasks
- 28:00 The Meta Skill: Dynamic tool loading and token optimization
- 33:00 Demo: Automated malware analysis, rule creation, and verification
- 40:00 AI's role: Augmenting SOC analysts, not replacing them
AI Beyond Triage and Hunting
Speakers: Chris Botelho, Senior Solutions Engineer, LimaCharlie
Conference: Wild West Hackin' Fest @ Mile High
YouTube: https://www.youtube.com/watch?v=9L_qv4V_o-4
Overview
In his compelling talk at Wild West Hackin' Fest @ Mile High, Chris Botelho, Senior Solutions Engineer at LimaCharlie, presented a deep dive into overcoming the inherent limitations of Large Language Models (LLMs) for practical, operational cybersecurity. Titled "AI Beyond Triage and Hunting," the presentation moves past the common narrative of AI generating images or text, focusing instead on how LLMs can be engineered to perform complex, multi-step security operations at scale. Botelho, a self-professed AI skeptic, shared LimaCharlie's journey in building an open-source framework that enables LLMs to intelligently interact with external tools, manage vast amounts of data, and orchestrate sophisticated defensive actions.
The core problem addressed is the "point in time" nature of LLMs – their knowledge is fixed at their training cutoff, rendering them blind to real-time operational data. This talk unveils a suite of innovative techniques, including progressive disclosure, external data handling, and specialized sub-agents, designed to transform LLMs from static knowledge bases into dynamic, actionable security assistants. By open-sourcing their entire solution, LimaCharlie aims to empower the security community to build robust, scalable AI-driven defenses, emphasizing augmentation of human analysts rather than their replacement.
This article dissects Botelho's presentation, exploring the technical challenges encountered, the solutions implemented, and the profound implications for the future of security operations. It highlights how these advancements move AI beyond mere summarization or basic hunting, enabling it to execute complex workflows, reverse engineer malware, and autonomously create and validate detection rules.
Background
▶ Watch: Introduction and why LLMs struggle with operations (0:00)
The fundamental challenge with current LLMs, as Botelho eloquently put it, is their "stuck in the past" nature. An LLM's knowledge base is limited to the data it was trained on, up to a specific point in time. This makes them excellent for generating content, writing code, or summarizing existing information, but woefully inadequate for dynamic, real-time operational tasks common in cybersecurity. Security operations, by definition, deal with constantly evolving threats, live network data, and real-time system events – information LLMs simply do not possess inherently.
To bridge this operational gap, Anthropic introduced the Model Context Protocol (MCP), a mechanism allowing LLMs to interact with external tools and APIs. The MCP defines how an AI can communicate with a tool, specifying error handling, input/output formats, and the tool's capabilities. This enables an LLM to, for instance, query databases, execute commands, or interact with external services. The protocol is structured around three components: the host (the user's interface, like a cloud code editor), the client (maintains the connection to the MCP server), and the MCP server (which informs the AI about available tools and their usage). Theoretically, this opens up a world of possibilities, from managing cloud resources to interacting with security information and event management (SIEM) platforms.
However, LimaCharlie's early experiments with MCP quickly exposed significant bottlenecks, primarily related to the LLM's context window. The context window is essentially the LLM's working memory, where it stores all ongoing conversation, prompts, tool uses, and their outputs. As this window fills, the LLM becomes slower, less accurate, and prone to making mistakes. When LimaCharlie attempted to expose its extensive API surface (over 140 distinct APIs) to an MCP server, they encountered a massive "startup tax," burning an estimated 50,000 tokens just to initialize the LLM's understanding of available tools. Considering that typical commercial LLMs like Opus 4.5/4.6 for regular users have a context window of 200,000 tokens, this initial burn represents a substantial portion of available memory before any actual work even begins.
Further exacerbating the problem is the "startup tax" incurred by the LLM itself. Botelho demonstrated that a plain Claude instance, with no MCP servers or custom tools loaded, already consumed 31,000 tokens on startup. Anthropic's own research indicated that up to 33% of a developer's context window could be burned just on initial MCP server loading. This token inefficiency makes large-scale, dynamic operational use cases economically unfeasible and technically unreliable. Simply increasing the context window (e.g., to 1 million tokens for enterprise versions) isn't a panacea; it incurs higher costs, increases latency, and introduces a phenomenon akin to "middle blindness," where the LLM struggles to retain information in the middle of a very long context. Research also suggests that LLM performance degrades significantly once the context window exceeds 50% capacity, making a 200,000-token window effectively only 100,000 usable tokens for reliable operations.
Moreover, direct processing of large data within the context window proved impossible. A simple lookup list of Tor nodes could consume 200,000 tokens, and a 10-megabyte log file was estimated to require an astonishing 2.6 to 3.5 million tokens – far exceeding any commercially available LLM's capacity. These challenges highlighted the critical need for novel approaches to context management and data handling to unlock the true potential of AI in operational security.
Key Findings
▶ Watch: Limitations of large context windows: 'middle problem' and cost (12:00)
LimaCharlie's exploration into operationalizing LLMs for cybersecurity led to several critical findings regarding the practical limitations and necessary architectural shifts:
- Context Window as a Primary Bottleneck: The limited size and performance degradation of the LLM's context window proved to be the most significant hurdle. Initial attempts to expose all 143 LimaCharlie APIs to an MCP server consumed 50,000 tokens on startup, severely limiting operational headroom. Even large context windows (e.g., 1 million tokens) suffer from "middle blindness" and performance issues beyond 50% utilization, making them less effective than their raw capacity suggests.
- Prohibitive "Startup Tax": A substantial portion of the context window (e.g., 31,000 tokens for plain Claude, or 33% of developer context according to Anthropic) is consumed simply by loading the LLM and its tool definitions before any user interaction, making efficient use challenging from the outset.
- Ineffective for Large Data Processing: Directly feeding large datasets like log files (e.g., 10MB file consuming 2.6-3.5 million tokens) or extensive lookup lists (200,000 tokens for Tor nodes) into the LLM's context window is technically and economically unfeasible with current LLM capabilities.
- Inaccurate and Incompatible Tool Search: Anthropic's native "tool search tool" (relying on regex or best patching 25) was found to be insufficiently accurate, with observed success rates of only around 50%. Furthermore, these native solutions are often LLM-specific (e.g., Claude-specific), hindering cross-platform interoperability, which was a key goal for LimaCharlie.
- Hallucinations in Cheaper Models: While faster and more cost-effective LLMs like Haiku exist, they exhibited a high propensity for hallucinations, fabricating data and leading to unreliable results. This rendered them unsuitable for production security environments where accuracy is paramount.
- The Need for Specialized Agents: A single, general-purpose LLM is inefficient for all tasks. High-level reasoning requires powerful, expensive models, while mundane data parsing or script execution can be handled by faster, cheaper, but less "intelligent" models. This highlighted the necessity for a multi-agent architecture.
These findings underscored that simply throwing more powerful LLMs or larger context windows at the problem was not a viable solution. Instead, a fundamental re-architecture of how LLMs interact with tools, manage data, and specialize in tasks was required to enable their effective use in dynamic security operations.
Technical Deep Dive
▶ Watch: Sub-agents: Using specialized LLMs for different tasks (24:00)
LimaCharlie's solution to these challenges involved a multi-faceted approach, fundamentally rethinking how LLMs interact with their environment and manage information. The core tenets are context window optimization, external data handling, and specialized sub-agents.
Context Window Optimization through Progressive Disclosure
To combat the massive token burn from loading all API definitions, LimaCharlie implemented a concept akin to progressive disclosure or lazy loading. They created a single, overarching meta-skill called the "LimaCharlie call skill." When the LLM starts, it only knows about this one meta-skill. This skill acts as a gateway to the entire API surface. Instead of presenting all 143 APIs at once, the meta-skill provides a brief overview of about 10 core capabilities.
When the LLM needs to perform a specific task (e.g., "query data for the past seven days"), it queries the meta-skill. The meta-skill then dynamically loads the relevant Claude skills (which are essentially markdown files defining how to interact with specific tools or API endpoints) on demand. This approach dramatically reduces the initial token load from 50,000 down to approximately 3,000 tokens on startup. It ensures the LLM's context window only contains the information immediately relevant to the current task, allowing it to "reach back" for more details as needed without being overwhelmed. This design also inherently supports cross-LLM compatibility, as the core logic isn't tied to a specific LLM's proprietary tool search mechanism.
External Data Handling for Large Datasets
Addressing the impossibility of loading large log files or lookup lists directly into the LLM's context window, LimaCharlie developed a mechanism to process data on disk. When the LLM requests a large dataset (e.g., "all systems connected to an outbound port 22"), the MCP server doesn't return the raw data. Instead, it temporarily stores the data in a secure cloud storage location and returns a signed, ephemeral URL to the LLM.
The LLM is then instructed to download the data to its local disk using standard tools like curl. Once on disk, the LLM can leverage local Linux utilities such as jq (JSON query), grep, bash, and strings to parse and analyze the data. This keeps the massive data volume out of the LLM's precious context window. Only the relevant, parsed, and highly summarized information is then fed back to the LLM for high-level reasoning. This approach is deterministic: small data goes directly to the LLM, while large data triggers the URL-based, on-disk processing workflow.
Specialized Sub-Agents and Model Orchestration
Recognizing that different LLM models excel at different tasks, LimaCharlie implemented a tiered architecture using sub-agents. The main, high-level reasoning model (e.g., Opus) is reserved for complex decision-making, task orchestration, and synthesizing information. This model is powerful but expensive and slower.
For the "grunt work" – tasks like file acquisition, data parsing, or generating simple scripts – LimaCharlie employs specialized sub-agents powered by a faster, cheaper, mid-tier LLM model like Sonnet. Initially, they experimented with Haiku due to its speed and cost-effectiveness, but quickly abandoned it because of rampant hallucinations – the model would fabricate information, leading to unreliable and dangerous outcomes in a security context. Sonnet, while slightly more expensive than Haiku, offered a crucial balance of speed, cost, and accuracy, making it suitable for production use.
Each sub-agent operates with its own independent context window, preventing the main agent's context from being polluted or overloaded. These sub-agents are tasked with specific jobs (e.g., "Go grab this file," "Parse this log," "Write a shell script to extract X"). They perform their task, and then return only the concise, relevant results back to the main Opus agent. This "right tool for the job" approach optimizes token usage, reduces latency, and significantly improves the overall efficiency and reliability of complex workflows.
Self-Correction and Open Source Commitment
A notable capability demonstrated is the LLM's ability to self-correct and adapt. If the LLM encounters a task for which it doesn't have a pre-defined skill, it can leverage its inherent code-writing capabilities (e.g., through Claude Code) to read API documentation on the fly and generate a Python or Bash script to accomplish the task. This makes the system incredibly flexible and extensible.
Furthermore, LimaCharlie's commitment to open source is central to their philosophy. The entire framework, including the MCP server implementation, the meta-skill, and related tools like LCRE (LimaCharlie Reverse Engineering CLI – a wrapper for headless Ghidra), is publicly available. This encourages community contribution, transparency, and interoperability, aligning with the dynamic nature of cybersecurity.
Demo / Proof of Concept
▶ Watch: The Meta Skill: Dynamic tool loading and token optimization (28:00)
Chris Botelho presented a compelling video demonstration of their AI-powered security pipeline, showcasing an end-to-end automated malware analysis, rule creation, and validation workflow. The demonstration, while pre-recorded due to conference Wi-Fi issues, highlighted the system's ability to handle complex, multi-stage tasks autonomously.
The initial prompt given to the main LLM (Opus) was highly sophisticated: "Analyze this malware. I want you to not only analyze it and figure out everything that's going on, I want you to reverse engineer it. I want you to go create new rules within our system. Then I want you to verify it by actually going and spinning up an Azure VM, running the malware on it, stick our agent on there so we can view it, and then make sure it works. And if it doesn't, go back and fix it."
The LLM, acting as the orchestrator, immediately broke this down into a series of parallel and sequential tasks. It intelligently identified dependencies, ensuring tasks like VM provisioning or agent installation completed before malware execution.
Key steps and technical aspects of the demo included:
- Orchestration and Context Management: The "LC Call Tool" (the meta-skill) was actively shown reducing token usage from the initial 50,000 down to approximately 3,000 tokens as the LLM focused on specific tasks. The system displayed task dependencies, with some tasks "blocked by #2," indicating intelligent workflow management.
- Reverse Engineering: A sub-agent was spawned to perform reverse engineering. It utilized LCRE, LimaCharlie's open-source wrapper for Ghidra headless, to decompile the provided malware sample. The demo showed the LLM identifying the malware as a .NET binary. Crucially, it was able to detect and decrypt an encrypted configuration within the malware, even brute-forcing a weak encryption key in a separate test case – all without human intervention.
- Dynamic Environment Provisioning: Another sub-agent interacted with the local Azure CLI tool (not an MCP server, demonstrating flexibility in tool integration) to provision a new, isolated Azure Virtual Machine. This process happened in parallel with the malware analysis.
- Agent Deployment and Malware Execution: Once the VM was ready, the system installed the LimaCharlie agent on it to monitor activity. It then uploaded the malware to the VM (using cloud storage for larger samples if needed) and executed it. The LLM intelligently handled situations where the malware might be too large for direct base64 encoding, opting for cloud storage and PowerShell downloads.
- Automated Rule Verification: After malware execution, the system queried LimaCharlie to confirm that the newly generated detection rules had fired as expected. If a rule didn't fire, the system could provide a reason (e.g., "C2 server is down, but the rule would have fired"). The demo proudly displayed "zero false positives" for the generated rules.
- Automated Cleanup: Finally, the system demonstrated responsible resource management by automatically deleting the Azure resource group and VM, ensuring no artifacts or costs were left behind.
The entire complex workflow, from initial prompt to full cleanup, was completed in 23 minutes. Botelho emphasized that the goal is not to replace SOC analysts but to augment them, automating repetitive tasks that an analyst would typically perform manually. This allows analysts to focus on higher-level reasoning, novel threats, and tasks requiring human intuition, making them more efficient, less bored, and more effective. The system even generated a comprehensive report at the end, which could be formatted and sent to management, further streamlining operational processes.
Defensive Implications
▶ Watch: AI's role: Augmenting SOC analysts, not replacing them (40:00)
The technical advancements presented by Chris Botelho have profound implications for defensive cybersecurity operations, offering a glimpse into a future where AI acts as a highly capable force multiplier rather than a simple alert triager.
- Automated Threat Lifecycle Management: The most significant implication is the ability to automate the entire lifecycle from threat intelligence to detection and response. An LLM can now ingest a new malware sample, reverse engineer it, automatically generate specific detection rules for a platform like LimaCharlie, deploy those rules, and then validate their efficacy in a live, isolated environment. This drastically reduces the time from threat discovery to active defense, moving beyond reactive measures.
- Scalable and Consistent Operations: By overcoming context window limitations and enabling efficient data handling, this framework allows AI to operate at an unprecedented scale. Security teams managing thousands of endpoints or multiple customer environments can leverage the AI to apply and verify detection logic consistently and rapidly, without being bogged down by manual processes.
- Reduced Analyst Burnout and Enhanced Focus: Automating repetitive, labor-intensive tasks such as setting up test environments, basic malware analysis, or validating rule logic frees up human SOC analysts. This allows them to focus on complex investigations, threat hunting for novel attacks, and tasks that require critical thinking and intuition – areas where AI currently falls short. The goal is to make analysts "less bored" and more engaged with high-value work.
- Proactive Rule Validation and Quality Assurance: The ability to automatically spin up a test environment, execute malware, and verify rule effectiveness is a game-changer for quality assurance. Defenders can ensure their detection rules work as intended before deploying them to production, significantly reducing the likelihood of false positives or missed detections. The system's capacity to explain why a rule might not have fired provides invaluable feedback for continuous improvement.
- Intelligent Data Analysis at Scale: The methodology of offloading large datasets to cloud storage and processing them on disk with local tools enables LLMs to effectively "read" and analyze massive log files, network captures, or endpoint telemetry that would otherwise overwhelm their context window. This unlocks new possibilities for automated behavioral analysis and anomaly detection.
- Flexible and Future-Proof Tooling: The open-source nature of LimaCharlie's solution and its design for cross-LLM compatibility ensures that defenders are not locked into proprietary ecosystems. This fosters innovation, allows organizations to choose the best LLM for their needs, and facilitates community-driven improvements to the framework. The LLM's ability to self-generate scripts for new APIs also makes the system highly adaptable to evolving toolsets.
- "AI as an Intelligent Scripting Engine": The framework effectively turns the LLM into an intelligent orchestrator capable of writing, executing, and debugging complex shell scripts and Python code to interact with existing security infrastructure (e.g., Ghidra, Azure CLI, LimaCharlie's platform). This bridges the gap between high-level reasoning and low-level system interaction.
In essence, these advancements transform AI from a passive knowledge base into an active, autonomous operator within the security stack, capable of performing complex, multi-step defensive actions with speed and precision, ultimately making human defenders more effective.
Key Takeaways
- LLMs require sophisticated context management for operational security. Direct use of LLMs for real-time security operations is hampered by severe context window limitations, high token costs (startup tax), and performance degradation with large inputs. Solutions like progressive disclosure (lazy loading of tools) and external data handling (processing large files on disk) are crucial.
- Specialized LLM sub-agents optimize efficiency and accuracy. Using a tiered approach with different LLM models for specific tasks (e.g., Opus for high-level reasoning, Sonnet for scripting and data parsing) significantly improves speed, cost-effectiveness, and reliability, while mitigating issues like hallucinations prevalent in cheaper models like Haiku.
- AI can automate complex, multi-step security workflows. The demonstration of automated malware analysis, reverse engineering, rule generation, and live validation in an isolated environment showcases AI's potential to orchestrate entire defensive pipelines, moving beyond simple triage.
- AI augments, not replaces, human analysts. The ultimate goal is to empower security professionals by automating repetitive, time-consuming tasks, freeing them to focus on high-value, intuitive, and novel threat investigations where human expertise remains indispensable.
- Open-source development is vital for AI in cybersecurity. LimaCharlie's commitment to open-sourcing their framework fosters transparency, collaboration, and rapid innovation, preventing vendor lock-in and allowing the community to collectively advance practical AI applications in defense.
- LLMs can act as intelligent orchestrators and code generators. The ability of LLMs to dynamically write and execute scripts (Python, Bash) to interact with existing tools and APIs, even for undefined tasks, highlights their potential as highly adaptable automation engines.
About the Speaker(s)
Chris Botelho is a Senior Solutions Engineer at LimaCharlie, bringing a wealth of experience from various roles within the cybersecurity industry. His background includes serving as a CISO, working as a consultant, and even a stint as a penetration tester, where he earned several CVEs. Despite his past in offensive security, Botelho expresses a strong preference for blue teaming, focusing on catching the "bad guys." He is deeply passionate about privacy and smart home automation, maintaining an entirely off-grid smart home with no cloud connections for his numerous devices. Initially an AI skeptic, Botelho's work at LimaCharlie has transformed his perspective, as he discovered practical applications for LLMs beyond content generation, leading him to develop the innovative solutions discussed in this talk. He identifies as a "nerd first," emphasizing his technical curiosity and dedication to open-source principles.
Reviews
Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT
This isn't your typical 'AI will save the SOC' marketing garbage. Botelho actually tackles the engineering nightmare of LLM context limits, demonstrating how to chain specialized agents and local CLI tools to make models do real work without bankrupting your API budget.
Heather Calloway (CISO) — SOLID
A rare, sober look at why plugging an LLM directly into your SOC data pipeline is an expensive way to generate hallucinations. Botelho provides a credible, engineered blueprint for using specialized AI sub-agents to orchestrate malware analysis and rule validation. If your team is asking for budget to build AI-driven workflows, make them watch this first to understand the token economics and context limitations.
→ Top-rated talks at Wild West Hackin' Fest @ Mile High 2026