200 Bugs/Week/Engineer: How We Rebuilt Trail of Bits Around AI

200 Bugs/Week/Engineer: How Trail of Bits Rebuilt Around AI

Speaker: Dan Guido, CEO and Co-founder, Trail of Bits

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 4, 2026, San Francisco

Watch on YouTube: https://www.youtube.com/watch?v=kgwvAyF7qsA

Reading time: ~9 minutes

TL;DR: Dan Guido walked through how Trail of Bits transformed from an AI-assisted firm to a genuinely AI-native one — hitting 200 bugs found per week per engineer, up from 15 — by treating AI not as a tool but as a structural operating system. The journey required defeating deep psychological resistance inside the company before any of the technology could matter.

The Solow Paradox Is Happening Again

▶ Watch: The productivity paradox and AI-native vs AI-assisted (2:00)

Dan Guido opened by citing a survey published just two weeks before the conference: thousands of CEOs reported that AI had produced no measurable impact on employment or productivity. Economists, he said, are calling this the Solow Paradox — a callback to Robert Solow's 1987 observation that "you can see the computer age everywhere but in the productivity statistics."

"I don't think AI doesn't work," Guido said. "I think most companies are doing it wrong. They give people tools without changing the system. Everyone gets a ChatGPT license, leadership waits for the productivity numbers to move, and they don't."

That gap between expectation and outcome, he argued, is the difference between AI-assisted and AI-native. One is a tool. The other is an operating system. The rest of the talk was about building the operating system at Trail of Bits.

Three Levels of AI Maturity

▶ Watch: Defining AI-native (2:30)

Guido defined three distinct tiers that organizations move through:

• Level 1 — AI-Assisted: Everyone gets ChatGPT. They draft emails and generate boilerplate. The organization doesn't change; work just happens slightly faster.
• Level 2 — AI-Augmented: Workflows are incrementally redesigned. Agents enter the loop. Maybe AI does the first pass on code review and a human does the second. The process is different, not just faster.
• Level 3 — AI-Native: A structural shift. The organization is designed from the ground up assuming AI is a co-participant in the work — not a tool, a teammate. Knowledge management, delivery models, and domain expertise are all designed to be consumed and amplified by agents.

At Trail of Bits, Level 3 means that 14 years of audit knowledge compounds with code. "Every engagement we do, every skill and workflow we build makes the next engagement faster. Every engineer operates with an arsenal of specialized agents built from our audit knowledge. That's not 'we use AI.' That's 'AI is on the team.'"

95% of the Company Was Against It

▶ Watch: Internal resistance and psychological barriers (4:00)

Guido was candid about what it cost to get there. "When I first launched this initiative inside Trail of Bits, the pushback was extraordinary. About 5% of the company was with me. 95% were against. At least 50% were passively resisting. A good quarter were actively resisting. I'm sure people thought about quitting. Cards on the table."

He attributed this to four well-documented psychological barriers that any organization attempting AI transformation will face:

1. Self-enhancing bias. Experts overestimate the quality of their own judgment. Research by Paul Meehl and Robin Dawes shows that crude linear models built from variables an expert claims to use will outperform that expert almost always — not because the model is smarter, but because it's consistent. "Humans don't operate the same every day. Some of you are very hungover right now."

2. Identity threat. "I spent 20 years doing this — a machine can't replicate that." Stanford ran a cooking study with the same device advertised two ways: "it does the cooking for you" versus "it helps you cook better." Cooks rejected the first framing. Guido made the parallel explicit: security auditing is symbolic work, and framing AI as replacing the auditor kills adoption. The correct frame is that AI makes you a more dangerous auditor.

3. Opacity. People feel they understand human judgment but not AI judgment — even though, as Guido noted, they don't really understand either. The fix is making the model's reasoning visible, showing why something is flagged rather than simply producing a result.

4. Intolerance for imperfection. In academic studies, participants watch an algorithm outperform a human forecaster, then switch back to preferring the human the moment the algorithm makes a single error. The fix: give people perceived control over the system.

▶ Watch: Solutions to the psychological barriers (8:00)

The Five-Part System

▶ Watch: The five-part AI-native system (12:01)

Trail of Bits built a five-component system, each element addressing one of the barriers above. Guido was explicit: you need all five for them to compound.

Part 1: Standardize. Everyone at Trail of Bits uses Claude Code, treated like any other enterprise tool — supported configs, no fragmented defaults, a clear "this is how we do it here." Skipping this step means 40 different workflows and zero organizational leverage.

Part 2: AI Handbook. Not a guide to prompting. A document that removes ambiguity about what tools are approved for what use cases, especially around sensitive client data. Which tools can't be used on client code. Where meeting recorders are disallowed by engagement type. "When a sensitive client asks what we're running on their code, I can give a clear answer that everyone knows."

Part 3: AI Maturity Matrix. This makes AI proficiency a first-class professional capability — the same as knowing Git, writing tests, or auditing C. There are clear levels, clear expectations for advancing, and consequences for stagnation. "If someone is at Level 0 and not engaging with this process, there may be a transition out of the company." The matrix shifts the identity frame: Level 3 isn't the person who uses AI the most, it's the person who invents new ways and builds new tools. "The identity shifts from 'I don't need AI' to 'I'm the one who makes AI dangerous.'"

Part 4: Hackathons. Two-to-three-day focused sprints with a single objective. For the most recent autonomy-focused hackathon, the constraint was that everyone had to run Claude Code in bypass permissions mode — intentional, to force people to learn sandboxing and guardrails. People worked in pairs for quality control. Success was measured on activity: issues filed and fixed, PRs reviewed and merged.

Part 5: Skills. A skill is a reusable structured workflow, usually with examples, constraints, and output verification. Trail of Bits maintains three repos: two public (an external community repo and a curated marketplace), and one internal containing company-specific workflows and the most sensitive vulnerability research skills. As of the conference, the internal system had 94 separate plugins, 201 skills, 84 specialized agents, and 400 reference files encoding 14 years of domain expertise.

▶ Watch: Results — 200 bugs per week (22:03)

The Infrastructure: Sandboxes, MDM, and MCP

▶ Watch: Toolchain, sandboxes, and the package cooldown policy (18:02)

Guido detailed the supporting infrastructure. For sandboxing, Trail of Bits gives engineers options rather than mandates: a dev container option, a DigitalOcean single-use virtual machine option, and documented native macOS sandboxing. The goal isn't uniformity — it's ensuring every engineer has a safe path.

One standout policy: a mandatory seven-day package cooldown. No one at Trail of Bits installs a package less than seven days old, for any package manager, enforced via MDM. "That's the easiest way to reduce risk and make the safe path the default path."

Trail of Bits is also open sourcing their Jamf MDM configuration stack for securing AI agents. And for MCP integration, they've wrapped tools like Slither (a static analyzer for Ethereum smart contracts) inside an MCP server, letting agents call internal tools reliably under organizational governance.

For prompt injection, the current "blunt instrument" is simple: for certain clients, internet access is denied entirely. They're evaluating more surgical solutions, including agent-native shells like NoNoShell and Agent.sh that can enforce protection at the kernel syscall level rather than relying on guidance in a CLAUDE.md file.

The Numbers: 200 Bugs Per Week

The results are striking. On engagements where the code base and scope allow it, Trail of Bits went from finding 15 bugs per week to 200 bugs per week per engineer. That's not a faster human — it's "an auditor running a fleet of specialized agents that do targeted analysis across the code base in parallel and then validate the results."

Across most engagements, 20% of all bugs reported to clients were initially discovered by AI.

On the business side, the sales team averages $8 million in revenue per representative — roughly double the industry average for security consulting — with AI assisting on proposal drafting, competitive positioning, conference prep, and lead enrichment. Same repositories, same compounding effect.

Guido acknowledged this is roughly one year into building the system seriously: "The models get better every month, the skills grow every week, and a year from now these numbers are going to look very different."

▶ Watch: Open questions — billing models and the future (26:03)

Open Questions and the Road Ahead

Guido closed by acknowledging what Trail of Bits hasn't solved. Confidential inference — running queries against frontier models without sending client code to a third party — remains an open problem. They've been evaluating Tinfoil.sh alongside other private inference servers. Policy enforcement and continuous learning are also unsolved: there's a wealth of data on each engineer's device about how they're actually using AI, and no good mechanism yet to collect and synthesize it.

Most provocatively, Guido raised the billing model question: "If some people can outperform others by 200x, I think you have to start billing by expertise and by results." The nature of how Trail of Bits offers services to the world will likely change in the next six to twelve months.

Notable Quotes

"That's the gap between AI-assisted and AI-native. One is a tool, the other is an operating system." (2:00)

"The identity shifts from 'I don't need AI' to 'I'm the one who makes AI dangerous.'" (10:01)

"That's not a faster human. That's an auditor running a fleet of specialized agents that do targeted analysis across the code base in parallel." (22:03)

"If some people can outperform others by 200x, I think you have to start billing by expertise and by results." (26:03)

Key Takeaways

• Giving people AI tools without changing organizational structure produces the Solow Paradox — activity without productivity gains.
• The four psychological barriers to AI adoption (self-enhancing bias, identity threat, opacity, intolerance for imperfection) are well-documented and require deliberate structural countermeasures, not just enthusiasm.
• All five system components must be implemented together — standardization, handbook, maturity matrix, hackathons, and skills — for the compounding effect to activate.
• A mandatory package cooldown policy (no packages less than 7 days old, enforced via MDM) is one of the highest-leverage low-cost security controls for AI-native orgs.
• The consulting billing model — hours times people — may be fundamentally broken when individual engineers can outperform teams by 200x.

Slides reference: No slides PDF was listed for this session. The transcript PDF (2026-04-04-Day2-Stage1-Guido-txt.pdf) provided detailed source content used throughout this article.