Secure Designs, UX Dragons, Vuln Dungeons: The Art of Secure Product Design

Secure Designs, UX Dragons, Vuln Dungeons: The Art of Secure Product Design

Speakers: Mike Shema, Kalyani Pawar (with guest Jack Cable)

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch the full talk

Reading time: ~8 minutes

TL;DR

Recorded live as episode 328 of the Application Security Weekly podcast, this panel-style talk argues that secure design failures are primarily organizational rather than technical — companies already know how to build secure software; they just choose not to prioritize it. The discussion covers secure defaults, developer UX, the liability gap, and how AI code generation is reproducing decades-old vulnerability classes at accelerating scale.

Introduction

The talk opens with a Dungeons & Dragons framing: AppSec practitioners are dungeon masters navigating threat models, but just as analysis paralysis can grind a tabletop game to a halt, organizations can get stuck in eternal design reviews without ever reaching a definition of "secure enough." Mike Shema, host of Application Security Weekly, and Kalyani Pawar, an AppSec expert with deep experience in startup security programs, frame the central challenge of their session around OWASP's A04 entry — Insecure Design — and why it remains stubbornly relevant despite decades of known mitigations.

The third voice in the discussion is Jack Cable, formerly of CISA, who co-authored guidance under the Secure by Design initiative and has since moved to a startup working on legacy application security debt. Together, the three trace a through-line from 1999-era usability research (specifically a Carnegie Mellon study on why developers couldn't use cryptographic APIs correctly) to today's AI-assisted coding environments, arguing that the fundamental problem — making the secure path the default path — has barely shifted.

▶ Watch: Introduction and OWASP A04 framing (02:00)

The Organizational Root Cause

One of the panel's sharpest observations is that insecure design is not a knowledge gap — it is a priority gap. Cable noted that the challenges are "not technical, they're organizational." Organizations do not build insecure products because they lack access to secure frameworks; they build them because security is not factored into product decisions from the start.

The CISA Secure by Design pledge, which reached 300 signatories around the time of the talk, attempted to address this by requiring companies not just to take security actions, but to publicly document and demonstrate progress. Cable acknowledged that not all signatories followed through, but pointed to the accountability mechanism as a necessary structural shift.

Pawar's experience in the startup ecosystem illustrated the practical version of this problem. Early-stage startups frequently give everyone admin access because it is faster, then face significant cultural resistance when that access is later restricted. The principle of least privilege is universally acknowledged but universally deferred. Her recommendation: treat security configuration as a financial investment with a modeled return, not as a tax on velocity.

▶ Watch: Secure by Design pledge and organizational accountability (06:00)

The Secure Defaults Problem — Log4j, XML, and AWS IMDSv2

The panel used three concrete examples to illustrate how insecure defaults compound over time.

Log4j: The 2021 Log4Shell vulnerability (CVE-2021-44228) was rooted in a message lookup feature that almost nobody needed or used, enabled by default. The scramble to patch it could have been avoided if the feature had been off by default — or not included at all.

XXE in XML parsers: XML parsers resolve external entities by default, meaning a developer who simply wants to parse a document unexpectedly gains file system read capability. The feature has existed for decades; it surprises developers because it violates the principle of least surprise.

AWS IMDSv2: The EC2 Instance Metadata Service had a property that, combined with server-side request forgery, granted full access to AWS environment credentials. AWS spent more than five years moving toward the secure-by-default IMDSv2, hampered by the impossibility of breaking backward compatibility with existing workloads. Kalyani framed this as a "patch versus rebuild" decision: small high-impact changes should be patched immediately; deep architectural problems with insecure defaults may justify a rebuild.

▶ Watch: Log4j, XML, and IMDSv2 as case studies (08:00)

The discussion also cited a striking Thinkst Canary finding: approximately 40% of users who saw example DNS configuration documentation containing the placeholder yourcompanyname.com literally created a DNS record for yourcompanyname.com, defeating the security property the configuration was intended to create. Security documentation that places cognitive burden on users will routinely fail.

Developer Experience as Security Surface

Cable made the case that the security community needs to think about developer experience (DevEx) the same way product teams think about user experience. Google's internal approach was cited: the company has built specific types that prevent raw user input from being passed into database queries, making SQL injection structurally impossible rather than just discouraged. React's template escaping and Let's Encrypt's short certificate lifetimes (now down to six-day lifetimes, compared to the historical two-to-five year certificates) were offered as further examples of the secure path becoming the default path through design rather than documentation.

▶ Watch: Developer experience and Google's approach (28:01)

Shema contrasted Curl with OpenSSL as a case study in code design quality: Curl, despite being written in C, demonstrates good design practices; OpenSSL, also in C, is widely regarded as difficult to use correctly. The panel debated whether this distinction — between projects that happen to be in memory-unsafe languages but are well-designed versus those that are poorly designed — is something LLMs can reliably identify. The consensus was that current LLMs can help surface known vulnerability classes but lack the contextual understanding to assess architectural design quality.

CISA's "Product Security Bad Practices" document, one of the last things Cable worked on before leaving the agency, listed 13 items companies "really shouldn't be doing in 2025" — including using memory-unsafe languages like C and C++ when alternatives like Rust exist that are comparably performant. The document was described as deliberately pointed: not advisory guidelines but clear statements of bad practice.

AI Code Generation: Repeating Old Mistakes at Scale

▶ Watch: LLMs, AI code generation, and secure design (30:02)

The panel's treatment of AI code generation was notably cautious. Cable's framing: AI is going to produce significantly more code, with fewer human eyes on it, at a pace that existing security tooling cannot match. If the default behavior of AI coding assistants is to reproduce insecure patterns — and the evidence suggests it is — then the security debt being accrued is growing faster than the capacity to address it.

Pawar drew an unflattering parallel between the current LLM ecosystem and the early days of mobile development, when apps were granted broad permissions and access to file systems and network resources without meaningful constraint. Android and iOS eventually built walled gardens with restricted permissions. Today's LLM agents and MCP (Model Context Protocol) servers are, as the panel observed, making many of the same mistakes: a calculator app that can read the file system should have been a red flag before deployment.

Cable expressed measured optimism about LLMs as discovery tools for vulnerability classes like authorization flaws (IDOR, broken access control) that are difficult to test systematically with traditional static analysis. The caveat was that LLMs should be combined with deterministic methods rather than treated as stand-alone security assessors.

CISA's proposed reframing of hardening guides — that they should become "loosening guides," with secure configuration as the default and deliberate weakening as the opt-in action — was presented as the appropriate direction for the next generation of infrastructure tooling.

Notable Quotes

"The reason that Johnny can't encrypt, that we end up with these bad UXs, is often that just this isn't a consideration that goes into building a product. It's not that we don't know how to build a relatively secure product, it just is that organizations don't prioritize it." — Jack Cable at 06:00

"Why was [the Log4j lookup feature] even there by default? You know, that's one of those shake your fist at the sky." — Mike Shema at 08:00

"We should flip hardening guides on the head — it should instead be loosening guides. The default configuration should be a secure one. If users want to weaken that and make the conscious decision to use a less secure product, then they can go do that." — Jack Cable at 36:02

Key Takeaways

• Secure design failures are organizational, not technical. The tools and frameworks to build secure software exist. The problem is that companies consistently choose velocity over security until a breach makes the trade-off visible.
• Secure defaults have the highest leverage. Features that are insecure by default (Log4j message lookups, XML external entity resolution, EC2 metadata access) impose remediation costs across an entire ecosystem. Designing secure defaults in reduces that cost to near zero.
• Developer UX is a security control. When the easy path for developers leads to insecure code, insecure code gets written. React's template escaping and Google's SQL-safe types demonstrate that structural prevention outperforms documentation and training.
• AI coding tools are reproducing known vulnerability classes. SQL injection, command injection, and memory safety bugs — all of which have known, scalable mitigations — are appearing in AI-generated code. The security community needs tooling that scales with AI output volume.
• Compliance is not security. SOC 2 and other compliance regimes measure effort, not outcomes. Cable's work at CISA pushed for outcome-based measurement of product security, including public transparency through the Secure by Design pledge.