Detection & Deception Engineering in the Matrix

Detection & Deception Engineering in the Matrix

Speakers: Bob Rudis, VP Data Science, GreyNoise Labs; Glenn Thorpe, Sr. Director, Security Research, GreyNoise Intelligence

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 3–4, 2026, San Francisco

YouTube: Watch the full talk

Reading time: ~10 minutes

TL;DR

GreyNoise's adversary engineering team built Orby — an AI-powered threat intelligence analyst that operates over a planetary-scale sensor network generating 22 terabytes of packet captures and 20 million Elasticsearch documents every 90 days. By building a "skill" system that loads domain expertise into the model at runtime, Orby can triage thousands of detection tags, profile novel threat actor infrastructure, and validate its own findings — and it demonstrated all of this live during the talk.

Introduction

The last talk of [un]prompted 2026's second day landed with an appropriate sense of both exhaustion and triumph. Bob Rudis, VP of Data Science, and Glenn Thorpe, Sr. Director of Security Research at GreyNoise Labs, delivered one of the conference's most theatrical and technically ambitious sessions — a live demonstration of Orby, their internal AI-powered threat intelligence analyst, completing a real threat analysis workflow in real time while the duo talked around it.

GreyNoise Labs is the adversary engineering and R&D arm of GreyNoise Intelligence. Their work involves crafting detection rules (called "tags"), deploying honeypots across a global sensor network, and hunting for threat actors and zero-days in the resulting data. The scale of the problem is what makes Orby's existence necessary: three thousand sensors across ninety-plus countries, 3,500 detection tags, 200 attacker profiles, and over the past ninety days alone — 22 terabytes of packet captures, 32 terabytes of session information, and 20 million Elasticsearch documents. One small team is responsible for finding meaningful signal in all of it.

"We are tired," Thorpe said, with the weary clarity of someone who had been doing exactly that for years. "And here's why."

▶ Watch: Introduction and Why Orby Exists (00:00)

The Origin Story: React-to-Shell and the Breaking Point

Orby was born out of necessity, not ambition. Rudis described it as a reverse origin story: normally the hero creates villains; here, a villain created the hero.

That villain was React-to-Shell, a vulnerability that hit in December 2025. While not as bad as Log4j in overall impact, it was severe enough to trigger an onslaught of exploitation attempts that continued through the time of the conference. By January 6, 2026, GreyNoise had observed:

• 8,200+ unique IPs attacking React-to-Shell targets
• From 1,000+ Autonomous System Numbers (ASNs)
• Slinging 70,000 completely unique payloads
• Many of those payloads were AI-generated — specifically, with help from Claude — exhibiting massively diverse network fingerprints

If Rudis and Thorpe had tried to manually analyze that volume of unique payloads and fingerprints, they would have failed to keep up. "And if Glen or I had tried to do [this manually], we would have..." Rudis let the implication hang. That pressure is what forced Orby into existence.

▶ Watch: The React-to-Shell Origin Story (02:00)

Orby's Architecture: The Skill System

Orby runs inside Claude Code — or at least, it started there. The team acknowledged they've hit the limits of Claude Code and are planning to build their own agent framework in Go. But the architectural pattern they've developed is what matters, and it's transferable.

The core insight is what Rudis calls the "skill system." Every LLM starts with an empty context and — for their specialized threat intelligence work — it's about as useful as, in Rudis's memorable phrase, "a thirteen-year-old with access to both a flamethrower and Wikipedia." The solution is loading the model with deep domain knowledge at runtime through structured skill files.

Each skill is a text file containing:

• A structured system prompt defining a specific workflow phase
• Associated resources (reference data, field definitions, intelligence databases)
• Bound tools (MCP servers and other integrations the skill can call)
• Mandatory constraints on behavior
• Anti-pattern definitions (what it must not do)
• Output specifications for precise, reproducible deliverables

The Orby workflow demonstrated live during the talk comprised approximately six phases. The team ran a live analysis — starting it before the talk began, checking in mid-talk — and it reached Phase 4 (final report generation) before they'd finished describing the architecture.

▶ Watch: The Skill System Architecture (06:00)

The Tool Stack: MCP Servers and Specialized Intelligence

What makes Orby functionally powerful rather than just architecturally interesting is its tool integration. The team built or adapted MCP servers for every data source and capability they rely on:

• Arkime (formerly Moloch) — for PCAP access and session data from the sensor fleet
• GreyNoise API — the primary ground truth for threat intelligence data
• DuckDB — all analytics happen locally through DuckDB. "Friends don't let friends let LLMs do their own analytics 'cause they suck at everything if you don't give them tools," Rudis explained.
• JA4 MCP server — encodes everything the team knows about JA4 network fingerprints, enabling infrastructure correlation across threat actor tooling
• OST domain MCP server — built in collaboration with a partner at Sandia National Laboratories, baking all knowledge about out-of-band security testing domains into a command-line and MCP tool, made public at the conference
• TShark MCP server — wraps the TShark packet analysis tool with thirty years of combined expert PCAP analysis knowledge
• Suricata — integrated similarly for rule-based detection
• VirusTotal — for enrichment and cross-referencing
• Censys — for internet-scale asset intelligence

The Censys integration received special praise: "It leveled up our analyses and their MCP servers are awesome," Rudis said, though he acknowledged they still had to write a custom skill to use it effectively.

▶ Watch: The Tool Stack (14:01)

Real-World Intelligence Production: Proxy Network Profiling

To show what Orby actually produces, Rudis and Thorpe described a recent intelligence operation. Over the past seventy days, they had tracked a previously unidentified malicious proxy network. With Orby's assistance, they identified the infrastructure, and then discovered that the threat actors had inadvertently left their Go profiling endpoint exposed — allowing Orby to profile the entire proxy network and map its behavior in detail.

"Orbi and I managed to dig into it, profile the whole thing, and we could see exactly what it would go do," Rudis said, with a mix of professional admiration and adversarial satisfaction. The findings were shared with the Canadian Centre for Cyber Security (CCCS), who were working to take down the infrastructure.

This kind of intelligence — finding and profiling novel threat actor infrastructure — is exactly what Orby was built for, and exactly what would have taken weeks of manual effort before. The team shared that all of this intelligence work is available through the GreyNoise platform.

Another Orby use case demonstrated during the talk: analyzing internet traffic shifts related to geopolitical events. "A little thing happened on Saturday that everyone probably realizes, which changes things up from an internet perspective," Rudis noted obliquely. Orby was configured with a specialized preference set to analyze conflict-related traffic, separating it from routine threat data — allowing the team to keep monitoring ransomware activity (which was spiking while attention was elsewhere) while Orby handled the geopolitical traffic surge.

▶ Watch: Intelligence Production Demos (10:00)

Controlling Agent Behavior: Guard Rails and Validation

Orby operates on real threat intelligence data, and errors can have real consequences — wrong attribution, bad IoCs, or hallucinated vulnerability data. The team developed several mechanisms to keep it in check:

Guard hooks — Shell commands that terminate unwanted behaviors immediately rather than asking the model politely not to repeat them. If a model "will do it anyway," you need a hard stop, not a soft reminder.

Decision trees — Enforced sequencing to ensure workflow phases happen in the correct order. "You can't have dessert before dinner."

Bulletproof patterns — Rigid templates for operations with multiple valid approaches but only one acceptable output format. Timestamp conversions were the example given: many ways to do it, but "don't think about it, just do it this way and move on."

Checklists — AI is "very, very good at following checklists," Rudis noted.

The Validator — A separate model that extracts every factual claim from a completed report and independently re-verifies each one against the source data. Early reports produced startling results: "Seventy-eight claims extracted, less than half were verified." In one case, the model had been summarizing GreyNoise tag names — which gradually shifted over a long run until the final report contained entirely invented tag names. The validator caught this and the fix was straightforward: add an explicit instruction never to summarize or truncate tag names.

The validator also caught one case where Orby appeared to have added something to the CISA Known Exploited Vulnerabilities (KEV) list that wasn't actually there yet. "I can't explain that one," Rudis said. "I should have actually checked to see if it got added by now."

▶ Watch: Guard Rails and Validation (16:02)

The Live Demo: Watching Orby Work

The talk's most memorable element was that Orby was actually running a threat analysis during the session itself — analyzing a real IP address through the full six-phase workflow without human intervention. When the team checked in mid-talk, it had already reached Phase 4, having:

1. Retrieved the target IP address
2. Looked it up in Censys
3. Cross-referenced with VirusTotal
4. Combined all enrichment data
5. Retrieved and analyzed JA4 fingerprints
6. Generated preliminary findings

The final report — available at a URL shared during the presentation — showed a complete threat intelligence assessment including infrastructure profiling, behavioral analysis, and validated claims. The validator ran concurrently, checking each claim against source data.

The live demonstration made a philosophical point as much as a technical one: Orby isn't a demo tool. It's a production system doing real work right now.

▶ Watch: Live Orby Check-In and Report (12:01)

The Manifesto: Designing LLM Behavior

Closing the talk, Rudis presented what he called the team's "manifesto" for how to design what an LLM will do. He framed it simply: "Assume these things are gonna do something really stupid and plan around it. Give it data, tell it what to do, retell it what to do, tell it what not to do."

He also shared a cost optimization note that reflects their operational approach: "We do the real hard work on Opus 4.6, and we do all the lightweight work on either Haiku or Sonnet to take things a little bit cheaper."

On tooling choices, Rudis offered candid guidance for the audience: "For anyone that's trying to do this in-house, start with Claude Code. It's the best thing out there. You're gonna outgrow it, and you should really consider moving to something else." GreyNoise is planning to build their own threat hunting agent framework in Go, potentially with LangGraph or LangChain as the orchestration layer.

▶ Watch: The Manifesto and Closing (22:02)

Notable Quotes

"We are tired. And here's why." — Glenn Thorpe (04:00)

"For our threat hunting context, it's actually by default about as smart and helpful as a thirteen-year-old with access to both a flamethrower and Wikipedia." — Bob Rudis, on LLMs without skills (06:00)

"Friends don't let friends let LLMs do their own analytics 'cause they suck at everything if you don't give them tools." — Bob Rudis (14:01)

"Assume these things are gonna do something really stupid and plan around it. Give it data, tell it what to do, retell it what to do, tell it what not to do." — Bob Rudis (22:02)

Key Takeaways

• Skills are the load-bearing structure. LLMs start empty; structured skill files that inject domain knowledge, workflow phases, constraints, and anti-patterns are what make them useful for specialized threat intelligence work.
• Give the model specialized tools, not raw data. DuckDB for analytics, JA4 MCP for fingerprints, TShark for PCAP — the expertise goes in the tools, not just the prompt.
• Validate every output. A separate validator model re-checking all claims against source data caught hallucinated tag names, summarized data, and more. For high-stakes security intelligence, this is essential.
• Tiered model selection controls cost. Opus 4.6 for complex reasoning, Haiku or Sonnet for lightweight tasks — matching model capability to task complexity dramatically reduces inference cost.
• Name your agent. Rudis made this point genuinely and with some humor: giving the AI system a name changes the relationship and helps the team reason about its behavior consistently.

Slides Reference

No slides PDF is listed for this presentation. The full live demonstration of Orby — including the completed threat intelligence report — was shared via URL during the talk and is available through the GreyNoise platform. The OST domain knowledge MCP server referenced during the talk was made publicly available to conference attendees.