Cyber Policy Shark Tank

Beau Woods (Host · I Am the Cavalry), Jake Williams, Aditi Narasimhan, Daria Bahrami, Emily Park, Matthew Yetter, Alan McQuinn, Nicholas Leiserson

Critical Effect DC 2025 · Day 2 · Track I: Critical Mass

Overview

The "Cyber Policy Shark Tank" session at Critical Effect DC 2025 offered a unique and dynamic platform to bridge the gap between technical cybersecurity and artificial intelligence (AI) experts and legislative policymakers. Hosted by Beau Woods of I Am the Cavalry, this event, inspired by the popular TV show Shark Tank, challenged participants to present actionable policy proposals to a panel of influential congressional staffers and White House officials. The core objective was to transform theoretical discussions into concrete strategies, fostering practical outcomes that address pressing cyber and AI security challenges.

Watch on YouTube

Visual summary for Cyber Policy Shark Tank by Beau Woods, Jake Williams, Aditi Narasimhan, Daria Bahrami, Emily Park, Matthew Yetter, Alan McQuinn, Nicholas Leiserson
Visual summary for Cyber Policy Shark Tank by Beau Woods, Jake Williams, Aditi Narasimhan, Daria Bahrami, Emily Park, Matthew Yetter, Alan McQuinn, Nicholas Leiserson

Key moments

  1. 0:00 Introduction to Cyber Policy Shark Tank format
  2. 4:40 Meet the 'Sharks': influential policy makers
  3. 9:40 Aditi's pitch: Standardized AI model cards and CARE
  4. 25:00 Jake's pitch: Secure signal archiving for government
  5. 40:00 Daria's pitch: Coordinated national AI security strategy
  6. 53:00 Sharks reflect on the policy pitch experience
  7. 57:00 Announcement of Cyber Policy Shark Tank winners

Cyber Policy Shark Tank

Speakers: Beau Woods (Host, I Am the Cavalry); Jake Williams; Aditi Narasimhan; Daria Bahrami; Emily Park; Matthew Yetter; Alan McQuinn; Nicholas Leiserson

Conference: Critical Effect DC 2025

YouTube: https://www.youtube.com/watch?v=gC0UeaEYm8Y

Overview

The "Cyber Policy Shark Tank" session at Critical Effect DC 2025 offered a unique and dynamic platform to bridge the gap between technical cybersecurity and artificial intelligence (AI) experts and legislative policymakers. Hosted by Beau Woods of I Am the Cavalry, this event, inspired by the popular TV show Shark Tank, challenged participants to present actionable policy proposals to a panel of influential congressional staffers and White House officials. The core objective was to transform theoretical discussions into concrete strategies, fostering practical outcomes that address pressing cyber and AI security challenges.

This innovative format provided a critical avenue for "cyber policy entrepreneurs" to not only articulate complex problems but also to propose implementable solutions that align with the legislative and regulatory levers of power available to Congress and federal agencies. By engaging directly with those who shape policy, the session aimed to fast-track the adoption of critical security measures, moving beyond mere "talk into action" to achieve tangible progress in national cybersecurity and AI resilience. The "sharks," representing various key committees and offices, offered their time, resources, and network as "prizes," emphasizing the real-world impact of successful proposals.

The importance of such an initiative cannot be overstated in an era where technological advancements, particularly in AI, are rapidly outpacing policy development. Events like the Cyber Policy Shark Tank are essential for ensuring that legislative bodies are informed by ground-level expertise, leading to more effective, forward-looking, and technically sound policies. It serves as a vital feedback loop, enabling policymakers to understand the practical implications and potential solutions for the complex threats facing the digital landscape.

Background

▶ Watch: Introduction to Cyber Policy Shark Tank format (0:00)

The Cyber Policy Shark Tank is an evolution of the "hackers on the hill" phenomenon, often associated with conferences like DEF CON, which aims to facilitate direct engagement between the cybersecurity community and government officials. Recognizing the need for more structured and outcome-oriented interactions, I Am the Cavalry, led by Beau Woods, developed this "spicier, different, or more adventurous" approach. The pilot program, initially an off-the-record event, proved highly successful, leading to its public instantiation at Critical Effect DC 2025.

The format mirrors the popular entrepreneurial pitching shows, but instead of financial investment, the "sharks"—comprising senior policy advisors from the White House Office of the National Cyber Director (ONCD), the Senate Homeland Security and Governmental Affairs Committee, the Senate Armed Services Subcommittee on Cybersecurity, and the House Science Committee—offered their expertise, connections, and commitment to advance promising policy ideas. This shift from capital to collaboration underscores the session's focus on practical implementation within the policy sphere.

To ensure proposals were actionable and relevant, a specific scoring rubric was established:

  • Leveraging Correct Levers of Power (4 points): Assessed whether proposals utilized appropriate legislative tools such as legislation, letter writing, or congressional hearings, rather than suggesting actions outside the legislative branch's purview. This served as a practical civics lesson for participants.
  • Addressing the Problem (3 points): Evaluated if the policy idea directly and effectively solved the stated problem.
  • Implementability (3 points): Judged the feasibility and practicality of putting the proposed policy into practice.
  • Creativity (1 point): Rewarded innovative thinking in addressing the problem.

This 11-point system (going "to 11") guided the contestants in crafting well-rounded proposals and provided the sharks with a clear framework for evaluation. The session aimed not just to identify good ideas, but to refine them into viable policy initiatives capable of making a real difference.

Key Findings

▶ Watch: Aditi's pitch: Standardized AI model cards and CARE (9:40)

The Cyber Policy Shark Tank featured three distinct pitches, each addressing critical aspects of cybersecurity and AI policy.

Pitch 1: Standardized AI Model Cards and Continuous AI Reporting (CARE)

Aditi Narasimhan from the AI Village presented a proposal focused on improving the evaluation and flaw disclosure processes for AI models. She highlighted the shortcomings of current methods like private red teaming (often opaque and unactionable) and bug bounties (resource-intensive and difficult to scale). Her solution, Continuous AI Reporting (CARE), is a two-part approach:

  1. Crowdsourced AI Evaluations: Building a comprehensive suite of evaluations, similar to the CVE program for software vulnerabilities, but tailored for the unique complexities of AI. These evaluations are functions designed to expose specific AI flaws (e.g., copyright infringement, hidden biases, hallucinations).
  2. Standardized Public Model Cards: Requiring AI model vendors to publish standardized documents that detail the model's intent, scope, and, critically, explicitly list the specific types of flaws the model is designed to resist. This creates a "clear and testable contract" between model claims and public evaluations.

Aditi's asks included congressional endorsement of these standards, their adoption in federal AI procurement (requiring standardized model cards and CARE evaluations in government contracts), and the establishment of an FFRDC (Federally Funded Research and Development Center), such as a "Center for AI Standards and Innovation," to manage AI flaw categories, akin to MITRE's role with CVEs. The proposed timeline for the open-source tech stack for CARE was 1.5-2 years for refinement, with the FFRDC established within 5 years.

Pitch 2: Oversignal – Secure Archiving for Federal Communications

Jake Williams addressed the pervasive problem of federal agencies struggling with secure and compliant records retention for messaging applications. He highlighted instances where agencies purchased non-FedRAMP accredited, demonstrably insecure solutions (like the now-defunct Telemessage) or resorted to SMS, leaving sensitive discussions vulnerable to foreign adversaries like Salt Typhoon.

His proposal, Oversignal, is a conceptual Signal application clone for iOS and Android, coupled with a security-hardened Linux server image. Agencies would run their own hardened archive servers, either on-premise or in FedRAMP-accredited cloud environments, ensuring secure archiving of Signal messages. This approach aims to provide a government-owned, self-service solution that addresses the high cost and complexity of existing commercial offerings, many of which lack on-premise options or FedRAMP accreditation.

Jake requested appropriations funding for the development of Oversignal. He also suggested that Federal Records Act amendments could reinforce the need for specific messaging applications, framing the initiative as a critical infrastructure protection issue under various cybersecurity statutes. He emphasized the significant cost savings for the federal government and the improved national security posture.

Pitch 3: Securing America's AI Future – A National Strategy for Data Integrity and Behavioral Anomaly Defense

Daria Bahrami, Head of Policy at Dreadnode, presented a strategy for securing America's AI future, focusing on the need to move beyond manual security processes against automated AI attacks. She acknowledged the ongoing Test AI Act but proposed a refined, coordinated execution model.

Her solution involves a proven three-agency coordination model:

  1. NIST and the Chief Digital and AI Office (CDAO) would collaborate to establish AI evaluation standards through two complementary working groups.
  • NIST's working group would focus on data quality assurance, crafting evaluations for dataset contamination and hidden biases, convening developers and data scientists.
  • CDAO's working group, in coordination with DIU (Defense Innovation Unit), would lead red teaming efforts, potentially partnering with national labs, to expose AI failure modes like hallucinations or capability degradation in real-world adversarial conditions.
  1. The Office of Management and Budget (OMB) would enforce AI procurement requirements for federal agencies and critical infrastructure, creating "powerful market incentives for secure-by-design technology."

Daria specifically asked for legislative support to explicitly establish DoD and Commerce research collaboration and position OMB as the authority for enforcing AI procurement standards. She estimated a total funding requirement of $100 million for this effort, arguing that it is a proactive investment against far greater potential losses from data breaches (e.g., the Equifax breach costing $4 billion, average 2024 breach costing $5 million).

Technical Deep Dive

▶ Watch: Jake's pitch: Secure signal archiving for government (25:00)

The three pitches, while distinct in their focus, collectively highlight several critical technical and architectural considerations for modern cybersecurity and AI policy.

Aditi Narasimhan's CARE (Continuous AI Reporting) proposal centers on the technical standardization and crowdsourcing of AI model evaluations. The core technical artifact here is the model card, a structured document that goes beyond mere descriptive metadata. It explicitly lists "specific types of flaws their model is designed to resist," transforming it into a verifiable contract. This technical specification would need to define not just the format of the flaw descriptions but also the methodology for testing resilience. The "crowdsourced AI evaluations" are themselves technical functions that "expose a specific flaw in an AI model." This implies a standardized API or execution environment where these evaluation functions can be run against AI models to objectively determine if a claimed resilience holds true. The parallel to the CVE program is crucial; just as CVEs identify specific vulnerabilities, CARE aims to categorize and test for AI-specific weaknesses like copyright infringement or data exfiltration through model outputs. The proposed "open-source tech stack" would be a critical component, providing the tools and infrastructure to manage, categorize, and execute these evaluation functions at scale, ensuring transparency and verifiability.

Jake Williams' Oversignal addresses a fundamental technical challenge: secure and compliant data retention for ephemeral, encrypted communications. The technical solution proposed is a Signal application clone for iOS and Android, which would maintain the familiar user experience while integrating a critical backend component: a security-hardened Linux server image. This server would be responsible for the secure archiving of messages. The "hardened" aspect implies adherence to best practices in server security, including minimal attack surface, robust access controls, and regular patching. Crucially, this server is designed for on-premise deployment or deployment within FedRAMP-accredited cloud environments, ensuring that agencies retain control over their data and meet stringent federal security requirements. The technical complexity lies in maintaining the end-to-end encryption principles of Signal while introducing a compliant archiving mechanism that doesn't compromise the security or privacy of the communication channel itself. This requires careful architectural design to ensure that only authorized, audited access to archived data is possible, while the live communication remains secure.

Daria Bahrami's strategy for Securing America's AI Future delves into the technical methodologies of AI security testing and evaluation. The proposal emphasizes data integrity and behavioral anomaly defense, recognizing that the foundation of reliable AI lies in the quality and trustworthiness of its training data and its predictable operation. The NIST working group would focus on developing technical evaluations for dataset contamination (e.g., poisoned data, adversarial examples introduced during training) and hidden biases (systematic errors or prejudices embedded in the data that lead to unfair or inaccurate model outcomes). This involves sophisticated data analysis, statistical methods, and potentially the development of new metrics for data quality in AI contexts. The CDAO working group, in collaboration with DIU, would concentrate on red teaming efforts. This entails actively probing AI models in adversarial environments to identify "AI failure modes" such as hallucinations (where an AI generates plausible but false information) or capability degradation (where an AI's performance diminishes under specific, often adversarial, inputs). Technical approaches here would involve adversarial machine learning techniques, stress testing, and the development of specialized attack vectors against various AI architectures. The goal is to develop robust benchmarks and testing methodologies that can be applied across diverse AI models, whether open-source, open-weight, or frontier models, ensuring a comprehensive understanding of their vulnerabilities and resilience.

Collectively, these proposals highlight a technical pivot towards proactive, standardized, and transparent security measures for AI and critical government communications. They underscore the need for open-source solutions, robust testing frameworks, and clear accountability mechanisms to build trust and resilience in emerging technologies.

Demo / Proof of Concept

▶ Watch: Sharks reflect on the policy pitch experience (53:00)

This session was structured as a "Shark Tank" style pitch competition for policy ideas, rather than a technical demonstration or proof of concept of software or hardware. While the pitches described potential technical solutions and their implementation, no live demonstrations of tools like Oversignal or the CARE evaluation suite were presented. The focus was entirely on the articulation of policy problems, proposed solutions, and their strategic implementation within the governmental and legislative frameworks.

Defensive Implications

▶ Watch: Announcement of Cyber Policy Shark Tank winners (57:00)

The policy proposals presented in the Cyber Policy Shark Tank carry significant defensive implications for organizations, government agencies, and the broader cybersecurity community. Translating these ideas into actionable strategies can enhance resilience against emerging threats, particularly in the rapidly evolving AI landscape and critical government communications.

For AI security, Aditi Narasimhan's CARE initiative suggests several defensive shifts:

  • Demand Transparency: Defenders should advocate for and, where possible, demand standardized public model cards from AI vendors. These cards, explicitly listing flaws a model is designed to resist, become a critical due diligence tool during AI procurement and deployment.
  • Leverage Crowdsourced Evaluations: Security teams can contribute to and utilize crowdsourced AI evaluation functions to rigorously test the AI models they deploy. This moves beyond opaque vendor claims to verifiable security postures.
  • Influence Procurement: Policymakers and federal agencies should integrate CARE standards into federal AI procurement requirements. By mandating these standards, the government can drive market demand for more secure-by-design AI models, benefiting all users.
  • Support AI-Specific Vulnerability Management: The creation of an FFRDC akin to MITRE's CVE program for AI flaws would provide a centralized, standardized repository for AI vulnerabilities, enabling more effective risk assessment and patch management for AI systems.

Jake Williams' Oversignal proposal highlights critical gaps in secure government communications and records retention:

  • Prioritize Compliant, Secure Messaging: Agencies must re-evaluate their communication platforms to ensure they meet federal records retention requirements while providing robust security. Relying on non-FedRAMP accredited or consumer-grade solutions for sensitive communications is a significant risk.
  • Consider Government-Owned Solutions: The concept of a government-funded and owned secure messaging and archiving solution like Oversignal offers a defensive posture that eliminates reliance on potentially vulnerable third-party vendors and ensures full control over data.
  • Address the "Human Issue": While a technical solution, the problem often stems from user behavior. Defenders need to provide secure, user-friendly alternatives that are readily adopted, rather than forcing insecure workarounds. Policy can reinforce the use of secure options through mandates and consequences for non-compliance.
  • On-Premise/Controlled Cloud Archiving: For highly sensitive data, the ability to archive messages on-premise or within tightly controlled, FedRAMP-accredited cloud environments is a crucial defensive control, minimizing exposure to external threats.

Daria Bahrami's strategy for Securing America's AI Future underscores the need for a holistic approach to AI defense:

  • Invest in AI Resiliency Research: Continued and coordinated investment in research on data integrity and behavioral anomaly defense is paramount. Defenders need better tools and methodologies to detect and prevent dataset contamination, mitigate hidden biases, and address AI failure modes like hallucinations and capability degradation.
  • Embrace Red Teaming for AI: Proactive red teaming of AI models in adversarial environments, as proposed by the CDAO working group, is an essential defensive practice. This allows organizations to discover and remediate vulnerabilities before they are exploited by adversaries.
  • Leverage Procurement for Secure AI: OMB's enforcement of AI procurement standards can serve as a powerful defensive lever. By requiring secure-by-design principles and robust evaluation standards in government contracts, agencies can influence the entire AI supply chain towards greater security.
  • Interagency Coordination: The proposed three-agency model (NIST, CDAO, OMB) demonstrates the necessity of a coordinated national strategy. For defenders, this means aligning with these national standards and frameworks, contributing to working groups, and leveraging the collective expertise fostered by such collaborations.

Overall, the defensive implications from these pitches converge on several key themes: the critical need for standardization in AI security, the importance of transparency from vendors, the power of government procurement as a market incentive for security, and the necessity of proactive testing (like red teaming) and secure architectural design for both AI systems and critical communications infrastructure. Defenders must engage with these policy discussions to ensure that technical realities are integrated into effective national security strategies.

Key Takeaways

  • Bridging the Policy-Tech Divide is Essential: The "Cyber Policy Shark Tank" successfully demonstrated how direct engagement between technical experts and policymakers can translate complex cybersecurity and AI challenges into actionable policy proposals, fostering implementable solutions.
  • AI Security Requires Standardized, Transparent Evaluation: The CARE (Continuous AI Reporting) proposal highlights the urgent need for standardized public model cards and crowdsourced AI evaluations to improve transparency and accountability in AI flaw disclosure, moving beyond opaque vendor claims.
  • Secure Communications and Records Retention Remain a Critical Gap: Federal agencies face persistent challenges in using secure, compliant messaging platforms, often resorting to insecure alternatives. Solutions like Oversignal emphasize the need for government-owned, hardened archiving solutions to meet federal records retention requirements and enhance national security.
  • Coordinated National Strategy for AI Resilience is Imperative: A multi-agency approach involving NIST, CDAO, and OMB is crucial for establishing and enforcing AI evaluation standards, focusing on data integrity, behavioral anomaly defense, and red teaming against AI failure modes.
  • Procurement as a Lever for Secure-by-Design Technology: Leveraging federal procurement policies to mandate AI security standards (e.g., standardized model cards, CARE evaluations) and secure communication solutions can create powerful market incentives for vendors to develop inherently more secure products.
  • Understanding Legislative Levers and Political Realities is Key: Successful policy pitches must not only be technically sound but also demonstrate an understanding of congressional jurisdiction, funding mechanisms, and the political context, including the "human issue" of technology adoption.

About the Speaker(s)

The Cyber Policy Shark Tank was hosted by Beau Woods, a prominent figure in the cybersecurity community known for his work with I Am the Cavalry and for initiating the "hackers on the hill" phenomenon, which aims to connect security researchers with policymakers.

The "cyber policy entrepreneurs" who pitched their ideas included:

  • Aditi Narasimhan, representing the AI Village, presented the CARE (Continuous AI Reporting) initiative, focusing on improving AI evaluation and flaw disclosure through standardized model cards and crowdsourced evaluations.
  • Jake Williams, a recognized cybersecurity expert, pitched Oversignal, a concept for a secure, government-owned Signal application clone with hardened archiving capabilities to address federal records retention challenges.
  • Daria Bahrami, Head of Policy at Dreadnode, an offensive AI security startup, proposed a national strategy for data integrity and behavioral anomaly defense through a coordinated three-agency model involving NIST, CDAO, and OMB.

The panel of "sharks," who provided critical feedback and evaluated the proposals based on their policy viability, comprised influential individuals from various government bodies:

  • Nicholas Leiserson, Senior Vice President for Policy at the Institute for Security and Technology, with prior experience at the White House Office of the National Cyber Director and a dozen years on Capitol Hill.
  • Emily Park, who leads the cybersecurity and critical infrastructure portfolio for Ranking Member Peters on the Senate Homeland Security and Governmental Affairs Committee, overseeing DHS and CISA.
  • Matthew Yetter, working for Senator Rounds, handling homeland security and national defense issues, with Senator Rounds chairing the Senate Armed Services Subcommittee on Cyber Security.
  • Alan McQuinn, professional staff for the House Science Committee under Ranking Member Zo Lofgren, leading portfolios on artificial intelligence, cybersecurity policy, and critical/emerging technologies, overseeing NSF and NIST.

The collective expertise of both the pitchers and the sharks underscored the high-level engagement and practical focus of the Cyber Policy Shark Tank session.

Reviews

Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT

This is how you do a policy panel without putting the room to sleep. Woods brought actual operators to pitch concrete legislative levers to congressional staffers who can actually draft the bills. It’s a masterclass in translating technical reality into Capitol Hill mechanics.

Heather Calloway (CISO) — SOLID

A grounded policy discussion that forces technical experts to map their ideas to actual regulatory and procurement levers. While aimed at federal agencies, the three pitches—especially on AI model cards and ephemeral messaging compliance—preview the exact mandates enterprise CISOs will face in the next 24 months. Worth your leadership team's time to understand where the compliance puck is going.

→ Top-rated talks at Critical Effect DC 2025

All talks from Critical Effect DC 2025