Don't Trust, Verify! - How I Found a CSRF Bug Hiding in Plain Sight

Don't Trust, Verify! How I Found a CSRF Bug Hiding in Plain Sight

Speaker: Patrick O'Doherty

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch the full talk

Reading time: 8 minutes

TL;DR

A decade-old CSRF protection library called Gorilla CSRF contained a bug in which the entire same-origin enforcement code path was effectively inert in production — because it only ran when the request URL scheme was set to HTTPS, which Go's HTTP server never populates for incoming requests. A misleading test helper in Go's standard library made the unit tests pass green for years. Patrick O'Doherty traced the bug from customer question to root cause, submitted a patch, and found in the process that browser-native fetch metadata request headers can eliminate the entire class of CSRF tooling complexity.

Introduction

CSRF — Cross-Site Request Forgery — is one of the oldest attack classes on the web. In its classic form, an attacker tricks a victim's browser into making a request to a trusted site on the victim's behalf, without the victim's knowledge. The defense has been well-understood for decades: generate a server-side secret, embed it as a token in forms and cookies, and reject any request where the two don't match. It should be solved.

Patrick O'Doherty, a software security engineer on the Tailscale security team, arrived at BSidesSF 2025 with a story about discovering that a Go library implementing exactly this defense had been doing it wrong for approximately a decade — and that the bug was invisible because a Go standard library test helper had been generating fake request objects that could never appear in production. The result was a CSRF middleware that was, as O'Doherty put it, "CSRF without the CS. It was just RF."

The talk moved from that specific bug to a broader point: test suites verify behavior in artificial conditions, not production behavior. And browser standards have quietly produced a better solution to CSRF that security engineers largely haven't noticed.

The Double Submit Pattern and Its Limitations

▶ Watch: How CSRF protection is supposed to work (03:00)

Gorilla CSRF is a Go HTTP middleware library that implements the double submit cookie pattern. On each request, the server generates a cryptographically random token, stores it in a cookie, and also embeds it as a hidden input in every rendered form. When a form is submitted, the server verifies that both the cookie value and the form value are present and match. An attacker who tricks a victim into submitting a form cannot supply a matching pair because they cannot set valid cookies for the target domain.

But double submit tokens alone are not sufficient. O'Doherty walked through a specific attack scenario: what if the attacker launches the attack from an origin with a "same-site" relationship to the target? There are two paths to this situation. First, a machine-in-the-middle attack over plain HTTP can set cookies that are sent along to an HTTPS destination. Second, a related subdomain — like events.bank.com — can set cookies on the common parent domain that will be forwarded to bank.com. In either case, the attacker can inject a CSRF cookie that the server considers valid.

To close this gap, more complete CSRF frameworks add origin or Referer header checking. A server that validates both the token pair and the request's stated origin is protected against same-site attacks as well. Gorilla CSRF included this logic — or appeared to.

The Bug: A Dead Code Path

▶ Watch: The customer question that started the investigation (07:30)

The investigation began with a customer question to the Tailscale security team: if an attacker managed to gain an XSS foothold on a marketing site or related domain, could they use it to launch a CSRF attack against login.tailscale.com? O'Doherty's first answer was wrong — he had misremembered how browsers and cookies interact and gave an unsatisfying response. Rather than repeat the experience, he built a demonstration application to measure the behavior empirically.

The demo presented two origins sharing a top-level domain (e.g., app.example.est and attack.example.est). When a victim visited the attacker-controlled site, it retrieved a valid CSRF cookie and form value from the target origin, rendered a phishing form, and set a more specifically-pathed cookie on the shared parent domain — exploiting the browser's cookie ordering behavior, which prefers path specificity over domain specificity. This cookie would clobber any less-specific token the legitimate site had set.

Critically, Gorilla CSRF's tokens are not bound to any user identity. The server cannot distinguish between tokens issued to Alice versus Bob versus an attacker. So the demo didn't need to steal Alice's token — it could substitute its own valid token from the shared parent domain, and the server would accept it.

▶ Watch: Why the unit tests were lying (13:00)

O'Doherty expected the demo to fail — the same-origin enforcement code in Gorilla would catch it. Instead, the request passed. The unit tests, which specifically tested the same-origin enforcement, had been passing for years. How?

The answer required tracing two bugs in sequence. Go's HTTP standard library uses the same Request struct for both clients and servers. When your server code handles an incoming request, the RFC specifies that most clients send only the absolute path in the request line — not a full URL with scheme and host. As a result, the URL.Scheme field on server-side requests is never populated. It's always an empty string.

Gorilla's same-origin enforcement code checks whether the request scheme is "https" before running any origin validation. In production, the scheme is never "https" (because it's never set at all), so the entire same-origin code block never executes. The CSRF middleware accepted every request, including cross-site ones.

The reason the unit tests passed is a Go standard library helper: httptest.NewRequest. This function accepts either a path string or a full URL as its second argument. The Gorilla test suite passed full URLs like https://example.com/path, which caused httptest.NewRequest to populate the scheme field — producing a Request object that bears no resemblance to what any real HTTP client would send. The tests were asserting behavior against a Franken-request that could never appear in production. "Your unit tests will happily pass green," O'Doherty said. "But none of this is ever going to run in production."

The library was patched and the fix is available in Gorilla CSRF version 1.7.3.

The Swiss Cheese Model and the Alignment of Failures

▶ Watch: Analyzing the bug through the lens of accident investigation (18:30)

O'Doherty analyzed the bug using James Reason's Swiss cheese accident model: in any system with multiple safety layers, no single layer is perfect, but if the holes in each layer are randomly distributed, the probability of a failure path through all layers simultaneously is low. The Gorilla bug represented the opposite — mutual deficiencies that lined up to create a complete failure path:

1. No identity binding on tokens. Any valid token from the shared domain could be substituted, eliminating the need to steal Alice's credentials.
2. The httptest.NewRequest behavioral artifact. The standard library helper generated request objects that misled the test suite into covering a code path that never ran in production.
3. HTTP-only development environments. Many developers work locally over plain HTTP and only encounter HTTPS in production deployment. Because the bug only manifested in the HTTPS code path, it was invisible to anyone who hadn't specifically tested it in an HTTPS context.

Three independent weaknesses, each small in isolation, aligned to create a vulnerability that persisted for approximately ten years.

Fetch Metadata: The Better Solution

▶ Watch: Fetch metadata request headers as a CSRF replacement (22:00)

O'Doherty closed with what he described as a superior approach to CSRF protection that has been available since 2019 and has been broadly available across standard browser versions since 2023: fetch metadata request headers.

Modern browsers now automatically attach a set of headers to every request made to a trusted (HTTPS) origin, explicitly signaling the context of the request: whether it is same-origin or cross-site, whether the request was initiated by a document or a script, and what the intended destination type is. Where CSRF defenses previously required the server to infer these relationships from tokens and cookies — a process with many failure modes — the client now states them explicitly.

An implementation using fetch metadata requires no server-side credential management, no token generation, no cookie lifecycle to maintain. The logic becomes simple: if the Sec-Fetch-Site header says the request is cross-site and the resource isn't intended to be publicly accessible, reject it. If the request is same-origin, accept it. That's it.

Tailscale is currently running instrumentation to evaluate replacing their double-submit cookie CSRF protection with fetch metadata middleware entirely. The compliance rate is 99.98% — the non-compliant traffic consists almost entirely of older Safari releases catching up to the standard. "It's looking imminent," O'Doherty said.

O'Doherty noted some self-criticism on this point: these headers had been available for years, and he had been watching them pass through logs without investigating what they were for. He committed publicly to writing about them and contributing to the available knowledge base.

Notable Quotes

"The library was CSRF. It was just RF. There was no CS."

— Patrick O'Doherty, on the discovery that Gorilla CSRF's same-origin enforcement was entirely inert

"Your unit tests will happily pass green. The branches that you want to have coverage for are green. But none of this is ever going to run in production."

— Patrick O'Doherty, on the misleading behavior of httptest.NewRequest

"Previously we had to infer all of this. The clients now just tell us. I am in love with the simplicity of this solution."

— Patrick O'Doherty, on fetch metadata request headers

Key Takeaways

• Unit tests are the beginning, not the end. Tests that pass against request objects that can never appear in production provide false confidence. Build small experiments that poke the actual security controls and verify their behavior under realistic conditions.
• Validate assumptions between test, development, and production environments. The single bit of information that differed between the test suite and production — whether URL.Scheme was populated — was the bit that controlled whether the entire defense ran.
• Mutual deficiencies can align invisibly. The Gorilla bug persisted for roughly a decade because the three contributing weaknesses were each subtle individually, and none of them was visible in isolation.
• Fetch metadata request headers are widely available and underused. Available in all standard browsers since 2023, these headers provide explicit same-origin context from the client, eliminating the complexity and failure modes of token-based CSRF defenses. Teams running modern browser targets should evaluate them as a replacement.
• Upgrade Gorilla CSRF. If your Go codebase uses Gorilla CSRF and adjacent origins exist that could be leveraged in a cross-site attack, update to version 1.7.3 or later.