Guardrails beyond Vibes: Shipping Security Agents in Production
Jeffrey Zhang, Siddh Shah
[un]prompted 2026 — AI Security Practitioner Conference · Day 1 · 1
Stripe's security engineering team replaced ad hoc "vibe checks" with a rigorous engineering discipline for deploying AI security agents in production. By combining modular multi-agent architectures with a golden-standard evaluation pipeline using LLM-as-judge, they shipped a threat modeling agent and a security routing agent with measurable accuracy and the confidence to iterate rapidly. Their core lesson: every meaningful quality improvement came from systematic evaluation, not intuition. ---
AI review
Stripe shipped real agents and measured them with real evals, and the honest accounting of what that required — including AlphaEvolve failing on their use case and a JSON formatting prompt causing a 10% undetected accuracy regression — is more valuable than another success-story talk. The eval pipeline design is the star.