Beyond the Stack

Beyond the Stack

Speakers: Tapan K, Engineering Manager

Conference: Payment Village

YouTube: https://www.youtube.com/watch?v=Tq2t9YTN6d8

Overview

In his compelling talk "Beyond the Stack" at Payment Village, Tapan K, an engineering manager with extensive experience in building and scaling payment systems, including global giants like Visa, presented a critical perspective on payment security. The talk challenges the conventional focus on hardening only the immediate payment stack, arguing that the most significant vulnerabilities often lurk in the myriad external systems, vendors, APIs, and third-party components that payment systems depend on but do not directly control.

Tapan K highlights that while organizations invest heavily in securing core financial operations, attackers frequently exploit overlooked gaps closer to the user and at the integration points between seemingly disparate systems. These "seams" in the payment ecosystem, often characterized by unclear ownership and insufficient scrutiny, can lead to catastrophic failures, resulting in massive financial losses, regulatory fines, and severe damage to brand reputation and customer trust. The talk serves as a vital call to action for payment professionals to broaden their security perimeter, advocating for a holistic approach that encompasses the entire end-to-end payment journey.

Why this talk matters is underscored by the real-world case studies Tapan K dissects, which illustrate how business logic flaws, outdated third-party scripts, and vulnerabilities in peripheral platforms can cascade into major breaches. He emphasizes that security is not a bolt-on feature but must be a foundational element, deeply ingrained in the architecture, testing, and operational culture from day one. For anyone involved in designing, developing, or securing payment systems, "Beyond the Stack" offers crucial insights and actionable strategies to identify and mitigate these often-hidden, yet profoundly dangerous, external threats.

Background

▶ Watch: Threats beyond your direct code control (1:45)

The journey of a payment, from a user's initial click to cash settling in a merchant's account, is a complex tapestry woven from numerous interconnected systems. Tapan K meticulously outlines this intricate flow, beginning with the user's interaction with a website or mobile application, which serves as the "front door" to the payment process. This initial touchpoint is critical, as any injected code or glitch here can compromise the transaction before it even begins. From there, the request moves into CRM systems, which store sensitive customer profiles, loyalty data, and order history—a veritable treasure chest for attackers if not adequately secured.

The complexity escalates with backend dependencies, comprising dozens of microservices, APIs, and third-party plugins. These components, while essential for functionality, introduce a vast landscape where hidden security gaps can thrive. The cart and checkout flow, dubbed the "conversion zone," is where elements like secure elements, tokenization, and PCI compliance become paramount. This is followed by the payment processor (e.g., Stripe), which translates the transaction into the appropriate payment rails. Subsequently, payment networks (e.g., Visa, Mastercard) facilitate the global movement of funds, connecting banks and processing millions of transactions per second. The customer's bank then verifies identity, checks funds, and approves or declines the payment, often leveraging opaque fraud detection mechanisms. Finally, funds settle into the merchant's account and are reconciled within their accounting system.

Tapan K points out that every single arrow connecting these blocks represents a potential failure or attack point, underscoring the vital need for designing systems with resilience in mind. Traditionally, major security investments are heavily prioritized in the "right side" of this journey—the core financial operations involving payment processors, networks, and banks. These areas are, rightly so, heavily regulated, monitored, and hardened. However, Tapan K argues that attackers often exploit the "left side"—the areas closer to the user and the interfaces between internal systems. These parts are often exposed, less regulated, and ironically, present the weakest links. Most real-world attacks, he contends, don't break the bank directly but rather slip through these overlooked gaps and inter-system vulnerabilities that lie "beyond the stack" of direct control, yet remain absolutely critical to the overall security posture of any payment ecosystem.

Key Findings

▶ Watch: Identifying overlooked attack points closer to users (5:30)

Tapan K's talk distills several critical findings from real-world payment system failures, highlighting patterns and warning signs that often go unaddressed. These findings emphasize that security extends far beyond the immediate code base and core payment infrastructure:

• Business Logic Vulnerabilities are a Major Attack Vector: Fraud isn't always about sophisticated hacking or credential theft. Often, it stems from subtle flaws in business logic, particularly at the "seams" where interconnected payment systems exchange data. These vulnerabilities can be exploited to trick systems into performing unintended actions, such as processing invalid refunds.
• External Dependencies are Critical Weak Links: Third-party scripts, outdated libraries, and external APIs, while providing functionality, significantly expand the attack surface. A single vulnerable component, even if not part of the core application, can provide an injection point for attackers to compromise sensitive data and payment flows.
• Overlooked Features Can Be Weaponized at Scale: Seemingly innocuous or convenient features, like automatic account reloads, can be weaponized by fraudsters, especially when combined with weak authentication or insufficient real-time fraud detection. These features, when lacking appropriate friction and verification, can become self-feeding fraud loops.
• Peripheral Platform Compromise Leads to Full Payment Flow Compromise: Security breaches are not always confined to the payment gateway itself. Vulnerabilities in adjacent or peripheral platforms, such as domain registrars or administrative panels, can lead to DNS hijacking or configuration changes that redirect legitimate traffic to fraudulent sites, effectively bypassing all payment security measures.
• Unclear Ownership Creates Security Blind Spots: One of the most silent yet significant risks in complex systems is ambiguous ownership. When responsibility for infrastructure, code paths, or integration points is unclear, vulnerabilities go unaddressed, edge cases are ignored, and accountability for security lapses is diluted.
• Dependencies are "Roommates with Keys": Every external component, API, or SaaS platform integrated into a payment system acts as a "roommate" with access to the system. While most are beneficial, a single misconfigured or compromised dependency can put the entire system at risk, expanding the attack surface beyond direct control.
• The Checkout Flow is Sacred: The checkout process is where trust, security, and revenue converge. Any dependency, visible or invisible, interacting with this flow becomes part of the security boundary. A vulnerability here can lead to injection attacks, PII leaks, or compromised payment logic.
• Testing Must Be Foundational and Comprehensive: Relying solely on basic unit tests or coverage metrics is insufficient. True security demands rigorous testing of integration points, edge cases, failure modes, and race conditions across multiple systems. Chaos testing is crucial for uncovering hidden fragilities.
• Security Must Be Baked In, Not Bolted On: Security cannot be an afterthought or a set of controls "sprinkled on" at the end of the development cycle. It must be a fundamental part of the design process, integrated into architecture discussions, data handling, authentication, authorization, and abuse vector analysis from day one.

These findings collectively underscore the speaker's central thesis: true payment security requires zooming out "beyond the stack" to understand and secure the entire ecosystem, embracing a proactive, ownership-driven, and continuously tested approach.

Technical Deep Dive

▶ Watch: Case Study: Revolut's cross-region business logic flaw (6:20)

Tapan K's technical deep dive illustrates how vulnerabilities manifest in the complex payment ecosystem, moving beyond theoretical discussions to concrete examples of exploitation. The core of his argument rests on the distinction between the heavily secured "right side" of the payment journey (processors, networks, banks) and the often-overlooked "left side" (user-facing systems, CRM, backend dependencies), where attackers find fertile ground.

One significant area of vulnerability lies in business logic flaws that emerge at the "seams" of interconnected systems, particularly across different regions or platforms. The Revolut (2021) incident serves as a prime example. In this case, the independent operation of Revolut's US and EU payment systems, lacking real-time communication and reconciliation, created a critical blind spot. Attackers exploited a flaw where high-value purchases made with US-linked accounts were correctly declined by the US system due to insufficient funds. However, the EU system, receiving only partial transaction data and operating under an assumption of trust, erroneously processed a refund as if a valid purchase had occurred. This logical gap resulted in fraudsters siphoning over $20 million by receiving money back for purchases that never went through. The technical flaw was a lack of end-to-end validation and settlement confirmation across regions, highlighting how implicit trust in cross-system communication can be weaponized.

Another prevalent threat discussed is the compromise of third-party dependencies and client-side code. The British Airways (2018) Magecart-style attack exemplifies this. Attackers identified an outdated, buggy third-party JavaScript library within the airline's web infrastructure. This wasn't core BA code but a dependency that became a perfect injection point. Malicious code was slipped directly into the payment page, executing silently in the customer's browser. As passengers typed their credit card details and Personally Identifiable Information (PII), the data was whisked away in real-time to the attackers' fraudulent payment workflow, all while the legitimate BA payment page continued to function normally. This incident, affecting an estimated 100,000 customers, showcased how a single, unpatched external component can lead to a catastrophic data breach, underscoring the need for rigorous supply chain security and continuous monitoring of all client-side dependencies.

The talk also delves into the weaponization of features, particularly when combined with weak authentication. The Starbucks (2015) incident involved the exploitation of its auto-reload feature. Attackers used credential stuffing and brute force attacks, leveraging leaked credentials from previous breaches (as users often reuse passwords), to gain unauthorized access to Starbucks accounts. Crucially, the absence of Multi-Factor Authentication (MFA) meant a single correct password granted full access. Once inside, fraudsters enabled and abused the auto-reload function to continuously siphon funds from linked credit cards or bank accounts into the Starbucks account, then rapidly transferred these balances to other accounts or gift cards under their control. This created a self-feeding fraud loop that continued until the linked payment source was depleted or blocked, impacting 1.5 million accounts and demonstrating how an overlooked feature, without adequate friction and real-time fraud detection, can be exploited at scale.

Finally, Tapan K emphasizes the vulnerability of peripheral platforms that, while not part of the payment stack, can compromise the entire payment flow. The Google Domains/Squarespace migration (2023) incident illustrates this. During a period after the migration, verification checks on domain transfers were not sufficiently stringent. Fraudsters initiated domain transfers to their own registrar accounts, bypassing meaningful owner verification. Once the DNS (Domain Name System) was under their control, they redirected legitimate traffic to cloned payment pages. The failure points here were multi-faceted: insufficient MFA during domain transfers at the registrar level, a lack of anomaly detection by merchant storefronts for sudden DNS changes, and payment processors failing to flag a surge of new merchant account creations with suspicious cloned content. This led to a reported direct financial loss of $3.4 million, demonstrating that even a highly PCI-compliant payment gateway is rendered useless if the upstream DNS or admin panel is compromised, as the traffic never even reaches the legitimate gateway.

These case studies collectively highlight that the true technical challenge lies in securing the integration points and inter-system communication, managing the expanded attack surface introduced by dependencies, and implementing robust business logic validation and anomaly detection across the entire payment ecosystem, far beyond the confines of the immediate payment processing code.

Demo / Proof of Concept

▶ Watch: Key lessons from Revolut's payment system breach (8:35)

Tapan K's presentation, "Beyond the Stack," focused on dissecting real-world industry case studies and outlining strategic defensive measures. The talk did not include a live demonstration or proof of concept performed by the speaker. Instead, the technical details were conveyed through detailed explanations of past breaches and their underlying vulnerabilities.

Defensive Implications

▶ Watch: Case Study: British Airways Magecart supply chain attack (9:10)

The insights from "Beyond the Stack" provide a robust framework for defenders to fortify their payment ecosystems against threats originating from outside the immediate stack. The defensive implications are broad, encompassing architectural design, operational processes, and a fundamental shift in security mindset:

1. Implement End-to-End Validation and Reconciliation: For any payment flow, especially those spanning multiple regions or disparate systems, end-to-end validation is non-negotiable. This means not just validating inputs but confirming outcomes and settlement across all integrated platforms. Treat cross-system communication as a high-risk integration point, implementing failsafe measures and real-time reconciliation to prevent business logic flaws like those seen in the Revolut case. Every transaction outcome should be confirmed by all parties involved, not merely assumed.

1. Rigorously Secure the Supply Chain and Client-Side Dependencies: All third-party JavaScript libraries, SDKs, APIs, and external components must be treated as part of your security boundary. This requires a proactive approach:

• Vetting: Thoroughly vet all third-party providers for their security practices.
• Patching: Maintain a strict patching regimen for all dependencies, ensuring that outdated or buggy components (like the one in the British Airways breach) are promptly updated or replaced.
• Monitoring: Implement Content Security Policies (CSPs) and real-time client-side monitoring solutions to detect unauthorized script injection or anomalous behavior on payment pages.

1. Strengthen Authentication and Introduce Friction for Automated Transactions: Any feature that automatically moves money, such as auto-reload functions, needs robust security controls.

• Multi-Factor Authentication (MFA): Mandate MFA for all sensitive account actions, especially those involving financial transfers or settings changes, to counter credential stuffing and brute-force attacks (as seen with Starbucks).
• Friction and Verification: Introduce intentional friction and additional verification steps for auto-reload or similar automated transactions, even for small amounts. This could include re-authentication, time delays, or transaction limits.
• Real-time Fraud Detection: Implement sophisticated, real-time fraud detection mechanisms that can identify and block suspicious patterns in automated transactions or account activity.

1. Harden Peripheral Platforms and Admin Access: The security of non-payment platforms that can impact the payment flow is paramount.

• DNS Security: Implement strong verification processes, including MFA, for all domain transfers and DNS changes. Monitor DNS records for unauthorized modifications.
• Admin Panel Security: Secure administrative panels and configuration management systems with the highest levels of authentication, authorization, and logging. Treat compromise of these systems as a direct threat to payment integrity.
• Anomaly Detection: Deploy systems that detect sudden, unusual changes in DNS configurations, admin settings, or merchant account creations, as highlighted by the Google Domains/Squarespace incident.

1. Establish Clear Ownership and Accountability: Ambiguous ownership is a critical vulnerability. Every component, integration point, workflow, and piece of documentation within the payment ecosystem must have a clear, named owner (person or well-defined group).

• Boundary Management: Assign specific ownership for the "seams" between systems, cross-team integrations, and vendor-managed components. Make it someone's explicit job to monitor and secure these interstitial areas.
• Accountability: Clear ownership fosters accountability, ensuring that security tasks are not dropped, edge cases are not ignored, and vulnerabilities are proactively addressed.

1. Maintain a Living Inventory of Dependencies: You cannot secure what you don't know you have. Develop and maintain a continuously updated "systems bill of materials" for all internal and external dependencies.

• Comprehensive Tracking: Document every dependency, including its version, data flows, ownership, and assessed risk level.
• Change Monitoring: Set up alerts for updates, deprecations, or any changes in dependency status. This inventory is a critical map for incident response and proactive risk management.

1. Embrace Comprehensive and Intentional Testing: Move beyond superficial testing to a culture of rigorous, reality-simulating validation.

• Integration Testing: Prioritize testing at integration points (APIs, third-party services, internal handoffs) where assumptions often reside and catastrophic failures occur. Test how systems handle timeouts, retries, and inconsistent authentication.
• Edge Case and Failure Mode Testing: Actively test for edge cases, failed modes, and race conditions across multiple systems.
• Chaos Testing: Systematically inject failures (e.g., killing services, throttling bandwidth, introducing latency) into staging environments to uncover hidden fragilities and validate system resilience. Start small and iterate.

1. Adopt a Security-First Architecture: Security must be baked into the design process from the very beginning, not bolted on later.

• Threat Modeling: Conduct thorough threat modeling at the whiteboarding stage, considering authentication, authorization, data handling, audit trails, and abuse vectors for every design decision.
• Least Privilege and Blast Radius: Design with least privilege in mind from day one and analyze the blast radius if a component is compromised. Assume breaches and limit lateral movement.
• Managed Services and Logging: For managed services, verify compliance standards. For internal tools, ensure appropriate logging and access error handling.

1. Develop a Robust Incident Response Strategy: Preparation is key to turning a disaster into a recoverable event.

• Documentation: Document comprehensive incident response plans, including clear roles and responsibilities.
• Tabletop Exercises: Regularly conduct tabletop exercises to practice detection, containment, and recovery under simulated pressure.
• Continuous Improvement: Learn from every incident, whether internal or external, to continuously refine security posture and response capabilities.

By implementing these defensive strategies, organizations can move "beyond the stack" to secure their entire payment ecosystem, mitigating risks that often originate from overlooked external dependencies and business logic flaws.

Key Takeaways

• Payment security extends far beyond the core payment stack; external dependencies, third-party systems, and business logic flaws are critical attack vectors.
• Unclear ownership at integration points and "seams" between systems creates significant security blind spots and allows vulnerabilities to fester.
• Client-side components and peripheral platforms (like DNS or admin panels) can be exploited to compromise the entire payment flow, even if the core gateway is secure.
• Any feature that automates money movement requires robust authentication (MFA), friction, and real-time fraud detection to prevent large-scale exploitation.
• Comprehensive, intentional testing—including integration, edge case, and chaos testing—is foundational to uncovering hidden fragilities across interconnected systems.
• Security must be a fundamental design principle from day one, not an afterthought, encompassing threat modeling, least privilege, and a robust incident response plan.

About the Speaker(s)

Tapan K is an accomplished Engineering Manager with profound expertise in building and scaling complex payment systems. His extensive background includes significant contributions to some of the world's largest payment infrastructures, notably at Visa and VisaNet. Tapan's experience spans both global enterprises and early-stage startups, providing him with a well-rounded and nuanced perspective on technical and organizational challenges within the payments industry. He holds an engineering master's degree, which underpins his practical work with a strong technical foundation, enabling him to navigate and address intricate security issues in the payment ecosystem effectively.