Zeal of the Convert: Taming Shai-Hulud with AI

Zeal of the Convert: Taming Shai-Hulud with AI

Speaker: Rami McCarthy, Principal Security Researcher, Wiz

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 3–4, 2026, San Francisco

Watch on YouTube: https://www.youtube.com/watch?v=6P77Zbo2TA4

Reading time: ~12 minutes

TL;DR

When a massive NPM supply chain attack campaign called Shai-Hulud leaked data from tens of thousands of compromised machines across GitHub, Wiz security researcher Rami McCarthy used AI to do in two days what two weeks of manual work could not: identify over 2,400 victim companies, including 37 of the Fortune 100. His talk is a detailed, honest post-mortem on what AI actually did and did not do well — and a practical guide to building AI workflows that compound in value over time.

Introduction

The Dune reference in the title is intentional. Shai-Hulud is the great sandworm — vast, destructive, and seemingly untameable. Rami McCarthy, a principal security researcher at Wiz, used it as the name for a series of interconnected supply chain attacks that crossed NPM and GitHub over the past year, alongside related campaigns like Singularity and the NPM Chalk attack. These attacks targeted the NPM ecosystem, and in many cases beginning in August, resulted in large amounts of stolen data being leaked across thousands of GitHub repositories.

McCarthy's challenge: capture that data before it disappears (GitHub and victims were cleaning it up), analyze 250,000 files across 30 gigabytes, and figure out who the data belonged to — so that victims could be notified and could respond. All of this on a compressed timeline, with ephemeral data and a problem space too large for any individual to navigate manually.

His talk at [un]prompted 2026 was not primarily about supply chain attacks. It was about how he actually used AI throughout this investigation, what worked, what failed, and what conceptual frameworks he developed for doing better work with AI over time. McCarthy is a self-described "convert" — someone who came to AI workflows through necessity and emerged with strong opinions about how to do it right.

▶ Watch: McCarthy's Introduction — The Year of the Worm (00:00)

Goal 1: Collecting the Leaked Data — Why Bash Beat APIs

The first problem was collecting the leaked data before it disappeared. McCarthy gave what he called the "naive example" — asking an AI to use the GitHub CLI to grab all repositories by name. It works. But it misses everything that matters: caching for idempotency, search segmentation to bypass API rate limits, parallelization, backoff handling, and performant downloads.

His slide deck made the point bluntly: Bash > APIs. The functional requirements for a real data collection pipeline include idempotency (so you can restart without re-downloading), pivoting for data completion, and creative approaches to working around search limits. The non-functional requirements — caching, parallelism, rate limiting, backoff — are the difference between a toy script and something that actually works at scale.

"Ask a stupid question, get a stupid answer," reads one of his slides. This was the first lesson of the investigation: the quality of AI output is bounded by the quality of the prompt and the thoughtfulness of the problem decomposition.

McCarthy introduced what he calls the RPI loop — Research, Plan, Implement. Before asking AI to write code, ask it to research the problem domain. Then ask it to plan the approach. Only then move to implementation. "Don't just jump to 'please write this script.'"

He also advocates for building composable utilities rather than throwaway scripts. Each tool you build represents effort to get there. As the cost of generating code drops, the value of being able to recall and reuse past utilities grows. He referenced a Simon Willison post called "Hoard Things You Know How to Do" as capturing this idea well.

▶ Watch: RPI Loop and Composable Utilities (02:00)

Goal 2: Analyzing the Data — 250,000 Files, 30 Gigabytes, One Laptop

The result of the collection phase was what McCarthy called a "poor man's data lake" — about 250,000 flat files, 30 gigabytes, spread across 30,000 repositories. Not big data by enterprise standards, but enough that linear scanning became IO-bound.

The first major AI contribution to analysis was fingerprinting for deduplication. The dataset appeared to contain 30,000 victim machines, but CI/CD systems produce duplicate runs. McCarthy asked Claude to derive a fingerprint that would identify unique machines. Claude one-shotted a working fingerprint. After deduplication, the actual count was 13,000 unique machines — and 77% of them were CI/CD runners, not developer laptops.

This fingerprinting result fundamentally changed what questions were worth asking. Knowing the shape of the data — CI/CD-heavy, not user-device-heavy — changed the analysis strategy entirely.

AI also excelled at signal extraction and pattern matching: identifying what programming languages were in use on compromised machines, what package ecosystems were present, whether systems were containerized, and which packages appeared to be the root cause of infection. "The ability to jump from a human trying to identify these patterns to Claude kicking out 25 different environment signals was significant," McCarthy said.

However, AI's relationship with data at scale revealed a consistent failure mode: AI is lazy about coverage. Ask it to analyze "all" of a large dataset and it will silently sample. Ask it to find "every company" and it will return the top ten. This is not a bug that can be prompted away — it requires architectural solutions at the harness level.

▶ Watch: Fingerprinting and Signal Extraction (06:01)

Goal 3: Attribution — Identifying 2,400 Victims with AI

The hardest problem was attribution: given 13,000 repositories worth of stolen data, figure out which companies owned it.

McCarthy worked through attribution in increasing order of sophistication:

Reasoning models for bulk processing. He fed every GitHub repository name from the dataset into Gemini — without even adding newlines between them, so they were merged into a single string. He got back useful signal anyway: "Find me the ten major companies" produced genuinely accurate answers. Reasoning models have changed the game for processing bulk, unclean data.

Secret scanning and enrichment. The dataset contained secrets — API keys, tokens, credentials. These could be validated, and the validation response could reveal the owning organization. TruffleHog has nearly 800 detectors for this purpose, but McCarthy found it had undesirable characteristics: it is commonly fingerprinted (triggering security alerts at targets), validation has a high false-negative rate, and it is relatively slow. His solution: feed TruffleHog's rules to an LLM and ask it to port the trivial ones — rules where validation is a simple GET or POST request — to a different engine. Thirty minutes later, he had 58 new rules in a completely different scanning engine.

This moment illustrated a broader point McCarthy made several times: content rules and detectors are becoming fungible. The value is no longer in which tool you pick — it is in the engine's architecture and your ability to compose it with everything else.

The creativity-credulity tradeoff. AI produced genuinely surprising attribution signals. It could identify that an encoded JWT likely contained claims worth extracting — something McCarthy would not have thought to try. It correctly surfaced attribution signals from obscure data patterns.

But the downside was consistent credulity. Specific failure modes he documented:

• The string "nucleus" appeared in some data → AI confidently connected it to "a specific company that uses a platform named Nucleus" (an extremely common platform name)
• Azure DevOps credentials → AI concluded the victim must work at Microsoft
• Consumer Microsoft service logins → attributed to Microsoft as a victim organization

"The double-edged sword: creativity and credulity," McCarthy summarized.

▶ Watch: Attribution — Reasoning Models and the Creativity-Credulity Tradeoff (08:01)

Building Feedback Loops and Injecting Skepticism

The key architectural insight McCarthy developed over the course of the investigation was the value of feedback loops between AI analysis and deterministic rules.

The pattern: use AI to analyze a sample of data and identify attribution signals. Take those signals, distill them into deterministic rules, run the rules at scale, and measure uplift. Then repeat — each cycle improving the rule set.

One concrete example: AI identified a new environment variable that hosted self-hosted Git instances as an attribution signal. After running that single new rule, the scan surfaced a major Russian e-commerce company, a Portuguese company, a Thai fintech, and a US Fortune 500 — all from one signal derived in one AI analysis session.

The counterpoint to AI credulity is injecting skepticism. In its simplest form, this means pausing during a coding session and asking: "Let's slow down. Let's check if these are true positives." At a more sophisticated level, it means building a "skeptical persona" into the agent harness — running a mix of agents that actively balance each other's conclusions.

The human-in-the-loop pattern was also critical. McCarthy could spot short terms and domain-specific signals that AI consistently missed. His solution was just-in-time programs — rather than manually examining data, spending 10 minutes building a throwaway tool with a clean UI to make human review faster. "If you're about to sit down and do something for an hour, think about whether 10 minutes of that would be better served by making a clean UX."

▶ Watch: Feedback Loops and Injecting Skepticism (14:01)

Results and the Live Demo

The headline numbers: the Shai-Hulud 2.0 incident concretely impacted at least 37 of the Fortune 100, all manually confirmed. The agentic analysis found over 2,400 impacted companies total — after just two days of building the analysis engine, heavily leveraging AI.

The comparison: the two weeks prior, doing this manually, McCarthy had found 200 impacted companies. The AI-powered approach found 12 times more in one-seventh the time.

McCarthy ran a live demo of two skills he published in a repository:

1. Victim attribution skill — Given data about a victim, it performs multi-stage enrichment: checking Truffle secrets validation, identifying internal tooling indicators, resolving domains through obscure APIs. For an Alibaba example, it correctly identified that one of the leaked secrets came from a documentation file used by an internal Alibaba AI developer tool.

1. Security contact skill — Given a company name, it checks security.txt, bug bounty platforms, security contact pages, and cert contacts in parallel. For 2,400 victims, this automates what would otherwise be 30 minutes of LinkedIn stalking per company.

"There's a repository there that has those two skills. Take 'em, use 'em, mix 'em up."

▶ Watch: Results and Live Demo of Attribution Skills (18:01)

Notable Quotes

"Ask a stupid question, get a stupid answer. You can't just assume you can give a dumb prompt and AI will pop out magic. You have to be thoughtful." — 02:00

"Product content rules, detectors — they're becoming fungible. The engines are really what matters." — 12:01

"The double-edged sword: creativity and credulity. The baseline alignment feels a little credulous for a lot of security purposes." — 10:01

"Two weeks of manual work: 200 companies. Two days of building the agentic engine: 2,400 companies." — 18:01

Key Takeaways

• Use the RPI loop (Research, Plan, Implement) — never jump straight to asking AI to write code. The quality of the output is bounded by the quality of the problem decomposition.
• AI excels at signal extraction, pattern matching, and reasoning over messy bulk data — but it is systematically lazy about coverage. Build harnesses that force it to be thorough rather than trusting it to be comprehensive.
• Inject skepticism deliberately — either as a manual check during analysis sessions or as an architectural element with agents that validate each other's conclusions. AI's credulity is a significant failure mode in security use cases.
• Use AI to derive deterministic rules, not to replace them. The best pattern is AI-assisted analysis → distillation into concrete rules → scale testing → iteration. This approach compounds over time.
• Detectors and content rules are becoming fungible across tools. Invest in understanding the underlying engine and composing it with other systems, rather than picking a tool for its built-in rule library.

Slides Reference

Slides from this talk are available as 2026-04-04-D1-S1-15-36-Zeal-of-the-Convert-Taming-Shai-Hulud-.pdf. Key slides include: "The Year of the Worm" (setting the attack context), "Goal 1: Scraping the leaked data" with the "Bash > APIs" principle, the data swamp visualization, "Goal 2: Analyzing the leaked data / Victimology," the "Creativity / Credulity" dual slide, "Enrichment: Porting Secrets Rules," "Agentic Analysis," "Kick off Feedback Loops," "Injecting Skepticism," "Bootstrapping Human-in-the-Loop," and the results slide: "37% of the Fortune 100."