Lessons from Running a Product Security-Focused Bug Bounty Program

Lessons from Running a Product Security-Focused Bug Bounty Program

Speakers: Aditya Saligrama, Joey Holtzman

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch on YouTube

Reading time: ~8 minutes

TL;DR

Stanford's Applied Cyber group runs a product security clinic that has engaged with dozens of student startups over the past year and a half — and found critical vulnerabilities in essentially every single one. Misconfigured Firebase and Supabase deployments, broken authorization logic, exposed AWS API keys, and exploitable admin panels appear consistently across fintech, edtech, biotech, and B2B AI companies alike. Aditya Saligrama and Joey Holtzman detail the engagement model, the war stories, and what other university programs and early-stage security practitioners can take from it.

Introduction

Security clinics have been proliferating at universities for about a decade — Berkeley, MIT, CMU, and programs in Texas, Nevada, and San Diego all run them. Most follow a similar model: partner with local organizations (typically small businesses and municipal governments), help them implement corporate information security basics like SSO, MFA, Active Directory hardening, and incident response policies.

Stanford's version of the clinic, operated by the Applied Cyber student security group, works in a different environment. The "local organization" at Stanford is far more likely to be a student-founded startup than a city government. And the typical security risk is not a misconfigured Windows domain — it is a vulnerable API, an exposed database, or AWS API keys committed to a frontend repository. Aditya Saligrama, a recent Stanford graduate and former Applied Cyber president who now works at enterprise security company Formal, and Joey Holtzman, current Applied Cyber president and second-year computer science student, brought the clinic's experiences to BSidesSF 2025.

The Problem Stanford Startups Actually Have

▶ Watch: The vulnerability landscape in student startups (06:00)

Saligrama described how his security career at Stanford began: not through a formal program, but by independently finding security vulnerabilities in student startups and filing unsolicited vulnerability disclosures. This included one incident that ended with a startup threatening legal action. Nonetheless, the pattern was clear: student founders — many of whom were freshmen building their first serious products — were deploying applications with user data and no meaningful security review.

The most common vulnerability classes they encounter fall into several categories.

Backend-as-a-service misconfigurations are the most prevalent. Firebase and Supabase both operate on a model where client-side code (web or mobile apps) makes queries directly to the database, and developers are expected to write security rules to restrict what clients can access. In practice, many student founders skip this step entirely. In Firebase, the SDK allows arbitrary queries to the database — misconfigured security rules mean any client can read any user's data. In Supabase, the platform auto-generates a REST API over the PostgreSQL database; without Row Level Security configured, any authenticated user can query for data belonging to other users.

Authorization and authentication confusion is the second major class. APIs at many student startups check only that the requester is logged in — they do not verify that the authenticated user is authorized to perform the specific action they're requesting. The practical consequence: an attacker who is logged into an account can trigger actions on behalf of other users. Saligrama described a social media platform example where a logged-in attacker could cause two other users to mutually block each other, simply by providing their user IDs as input to an API that only checked for authentication.

Other recurring issues include unprotected GraphQL introspection endpoints (where requesting the schema reveals the entire data model) and IDOR (Insecure Direct Object Reference) vulnerabilities throughout custom APIs.

The Engagement Structure

▶ Watch: How the clinic runs engagements (15:00)

The clinic's engagement model is structured to fit within a two-hour window, divided roughly in half between review and active testing. Before the engagement, clients complete a worksheet detailing their tech stack and what the product actually does, and sign a mutual NDA and MOU. This pre-work allows the clinic to walk in already understanding the application rather than spending the first thirty minutes on orientation.

The engagement itself follows five steps. First, a short walkthrough where the client describes how a typical user interacts with the app. Second, threat modeling and infrastructure review — identifying where attackers are most likely to focus and walking through cloud configurations (AWS, Supabase, or whichever platforms the client uses). Third, a data security consultation covering what data is being stored, how it's stored, how users authenticate and communicate with the backend, and what data clients should and should not be able to access. Fourth — Holtzman's "favorite part" — live security testing, where the team actively tests the application using findings from the first three steps and verifies that the security controls the client described actually exist in the product ("many times we've gotten clients where they say 'oh yeah we have this authentication for this page,' and that's actually not the case"). Fifth, a debrief and a written report with vulnerabilities and recommended remediations.

Clients receive test accounts and staging environment access before the engagement, allowing the security team to test without touching production data or creating risk for actual users.

War Stories: What They've Actually Found

▶ Watch: Vulnerability war stories (30:00)

The case studies Saligrama and Holtzman described ranged from basic to alarming.

Client-side access control bypass. An early client had an edtech startup with an admin dashboard "protected" by client-side access control. Navigating to /admin showed an access control error — but hitting the browser's back button exposed the entire admin panel with full functionality and access to sensitive data.

Live incident: AWS API keys in the frontend. One startup reached out not to schedule a routine engagement but via Slack message: "We are actively under attack. Could you help us out please?" The team conducted live incident response and found the root cause: AWS API keys had been accidentally exposed in the frontend JavaScript. A crypto mining bot scanner had found the keys, spawned container clusters in the startup's AWS account, and was mining cryptocurrency and forwarding proceeds to the attackers. The clinic helped shut down the IAM rules the bot was using, terminate the clusters, clean up exposed resources — and helped communicate with AWS to attempt to waive the $6,000 bill the startup had accumulated in two days. The same startup also had a Python Flask endpoint that spawned virtual machines for biotech computation jobs, reachable via unauthenticated HTTP and vulnerable to command injection that provided remote code execution on their main web server.

SSRF via PDF analysis feature. A public interest tech project offered AI analysis of PDFs via URL submission. There was no validation on the submitted URL. One path: submit a request for the EC2 Instance Metadata Service endpoint (http://169.254.169.254/latest/meta-data/iam/security-credentials/) and retrieve the AWS IAM role credentials, enabling full account takeover. A more creative path: because TinyURL allows predictable shortened links with custom slugs, an attacker could construct a TinyURL link that redirected to the application's own PDF analysis endpoint — with a URL parameter pointing back to the TinyURL link. The result was an infinite redirect loop that consumed resources until the service crashed. A denial-of-service attack constructed entirely from the application's own URL processing logic.

Negative withdrawal amounts. A fintech startup had an API endpoint for account withdrawals that accepted negative numbers as input. Submitting a negative withdrawal value increased the account balance. "We could have wiped the entire company out with a single web request."

VPN bypass via trusted header injection. A startup claimed their admin panel was protected behind a VPN — accessible only from specific IP ranges. The enforcement mechanism was a Django plugin called admin-restrict, which had a known CVE from several years prior: it blindly trusted the X-Forwarded-For header, which is supposed to be set by a load balancer but can be set arbitrarily by any client. By setting X-Forwarded-For to an IP within the allowlisted range, the attacker bypassed the VPN restriction entirely.

Lessons for Other Programs and Practitioners

▶ Watch: Lessons learned and practical advice (46:00)

Several lessons emerged from the clinic's experience that apply beyond Stanford's specific context.

Pre-engagement documentation pays dividends. Having clients complete a tech stack worksheet before the session means the team arrives knowing what frameworks and platforms are in play, and can focus the two hours on security rather than orientation.

Automated tooling for common misconfigurations compounds over time. The same classes of vulnerabilities recur across Supabase deployments, Firebase integrations, and GraphQL APIs. Building automated checks for these common patterns saves significant time and makes the limited two-hour window more productive.

Early-stage startups are genuine attack targets. The clinic consistently had to push back against founders who assumed they were too small to be interesting. Crypto-mining bots do not target specific organizations — they scan the entire public internet for exposed credentials. A student startup with exposed AWS keys is as vulnerable as any enterprise. "We want to emphasize that founders should take their security in their products seriously."

Education cannot solve the problem alone. Many of the founders the clinic works with are freshmen who started building before they ever took a security course. Even if universities mandated security education in CS curricula, it would arrive too late for the developers who most urgently need it. The clinic model — meeting founders where they are, with a direct and practical engagement — is a complement to, not a replacement for, curriculum reform.

Notable Quotes

"In basically every single engagement that we have had, we have identified some major flaws in the security of their product. That's kind of helped vindicate our initial motivation for starting the clinic." — Joey Holtzman (32:00)

"We are actively under attack. Could you help us out please?" — Startup founder's Slack message, quoted by Aditya Saligrama (34:00)

"Early-stage startups are actual targets to attackers. Many people think attackers are after Google and Meta — but really, as Aditya pointed out, these attackers are going after early-stage startups a lot too." — Joey Holtzman (50:00)

Key Takeaways

• Firebase and Supabase misconfigurations are the most prevalent vulnerability class in student and early-stage startups. Client-side security rules that are missing, incomplete, or wrong give any authenticated user access to data belonging to other users.
• Authorization is not the same as authentication. Checking that a user is logged in before allowing an action is not sufficient — every sensitive action needs to verify that the caller is authorized to perform it on the specific target.
• Exposed cloud credentials are an immediate and automated risk. Crypto-mining bots scan the entire public internet continuously. An AWS API key in a frontend file is likely to be found and exploited within hours.
• Server-side request forgery (SSRF) in AI-adjacent features is a growing attack surface. Features that fetch or process external URLs — PDF analysis, AI document ingestion, webhook handlers — are high-value SSRF targets; the EC2 metadata endpoint is usually the first test.
• Very small teams doing two-hour engagements find critical bugs every time. The ROI on even lightweight security review at the early startup stage is exceptional. Founders who engage proactively find out about vulnerabilities before their users do.