Paying More for Worse Security: An AWS Marketplace Horror Story

Paying More for Worse Security: An AWS Marketplace Horror Story

Speakers: Corey Quinn

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=pkPxTuhNl5U

Overview

In this eye-opening talk from fwd:cloudsec, Corey Quinn, author of the "Last Week in AWS" newsletter, exposes a pervasive and disturbing trend within the AWS Marketplace: a "horror story" where customers unknowingly pay significant premiums for outdated, unpatched, and often less secure versions of free operating systems. Quinn meticulously details a business model that, while apparently legal and even "AWS blessed," preys on enterprise customers, leveraging the platform's trust mechanisms and automation practices to insert compromised-by-neglect software into production environments.

Quinn's investigation reveals that this isn't an isolated incident but a burgeoning "cottage industry" of vendors selling free software with no added value, often with misleading pricing and non-existent support. The talk transcends mere financial waste, arguing that these practices constitute a critical platform security issue, despite AWS's official stance. It highlights a fundamental breakdown in trust within a critical cloud service, forcing security professionals to re-evaluate how they procure and manage software in the cloud.

The implications of this phenomenon are far-reaching, affecting not only an organization's bottom line but also its security posture. By deploying AMIs that are deliberately outdated in terms of kernel versions, security patches, and critical libraries, companies expose themselves to known vulnerabilities without realizing it. Quinn's talk serves as a stark warning and a call to action for defenders to scrutinize their AWS Marketplace usage and actively identify these hidden security and cost liabilities.

Background

▶ Watch: Corey's marketplace joke became a real business (1:00)

The genesis of this "horror story" traces back to a sarcastic joke Corey Quinn made on the internet in 2019. He outlined a hypothetical scheme to embezzle from an employer via the AWS Marketplace: spin up a custom Amazon Machine Image (AMI) of a free operating system like CentOS, sell it at a markup, reference it in an anonymous blog post, and then deploy it in production at one's own company. This absurd scenario, intended as satire, quickly became an unfortunate prophecy. Just 15 days after Quinn's post, a domain, supportedimimages.com, was registered, and a business mirroring his satirical premise began operations, and is still running today.

Quinn's deeper investigation into this issue was triggered two years later, in 2021, when a developer shared a thread on Reddit. The developer’s automated CentOS build system, using Packer, began failing. The root cause was the system inadvertently selecting a lookalike AMI with a nearly identical name from the AWS Marketplace, which had become the "best match" in their filtering logic. This incident brought the issue back to Quinn's attention earlier this year, prompting him to directly ask AWS if this constituted a security problem. In a high-level call, AWS unequivocally stated that it was "not a security issue."

This dismissal became a central point of contention for Quinn. He clarifies upfront that his findings do not involve malware, a zero-day exploit, or a dramatic chain of vulnerabilities. Instead, the problem is "dumber than that": it's a completely legal, fully sanctioned, and "apparently AWS blessed" business model that is, in his view, "somehow worse than if it were an actual crime." The core issue, as Quinn frames it, is a platform problem where Amazon not only permits this predatory practice but also profits from it, taking a percentage of every sale. His expertise lies in scrutinizing large AWS bills, and this particular anomaly, "wearing a stupid fake mustache," repeatedly appeared across various customer environments, prompting him to pull on the thread until the full scope of the issue was revealed.

Key Findings

▶ Watch: The problem: Legal, sanctioned, and dumber than malware (2:50)

Corey Quinn's investigation into the AWS Marketplace unveiled a disturbing landscape of predatory vendors exploiting the platform's trust mechanisms and customer ignorance. The key findings illuminate a systemic problem with significant financial and security implications:

• The "Product" is Free Software at a Premium: Vendors like supportedimimages.com and others sell virtually every free operating system imaginable—Ubuntu, CentOS, Debian, RHEL, AlmaLinux, Rocky Linux, Oracle Linux, Fedora, and even Amazon Linux itself. These are offered at a significant markup, despite being freely available from official sources or directly from AWS.
• No Added Value: Unlike legitimate marketplace sellers who bundle free software with configuration, tuning, or hardened stacks, these vendors provide no discernible value. Their "product" consists of automated build jobs that merely bump a version number and repost the image, lacking any customization, hardening, or additional features. Quinn notes, "You're not paying for what they did because there is no what they did. You're just paying."
• Bogus Support as Value Proposition: The sole "value" offered is often "extended support" with a 24-hour email response guarantee, accessed via a generic web form. This is presented as a premium service, yet it's demonstrably worse than free community options like Stack Overflow or IRC channels, which offer faster and more practical assistance.
• Significant and Hidden Costs: The financial burden is substantial. On a T2.micro instance, the software fee alone can be 18 times the cost of the actual instance. Scaling up, a T3 medium can incur an additional $1,200 per year, 10 M5 large instances can cost $12,000 annually, and some customers are reportedly spending up to $500,000 per year on these overcharged AMIs.
• Misleading Pricing Displays: Marketplace listings often show "starts at $0 an hour" or "up to 70% savings." This deceptive pricing is achieved by setting the software fee to zero only on extremely expensive, high-end instance types, such as the P3DN 24x large (an instance with eight GPUs costing $31 per hour before any software fees). This ensures the attractive "free" price appears prominently in search results.
• Security Degradation: Quinn's audit of a YUbuntu 24.04 image from one such seller revealed a significantly degraded security posture. The image was two point releases back compared to the official Canonical AMI, its kernel was two major versions back, and all security-sensitive packages—including OpenSSH, curl, gnutls, the crypto stack, and Intel microcode updates—were older and unpatched.
• No Malware, Just Neglect: While Quinn found no evidence of intentional malware, the neglect itself constitutes a security risk. The lack of updates and stale libraries mean these AMIs are vulnerable to known exploits, effectively shipping unpatched systems into production.
• Widespread Problem: This is not limited to supportedimimages.com. Quinn identified a "cottage industry" with easily north of 900 active listings from various sellers, including Procomputers (with ~287 listings), Gigabits, TIOVIT, Rin Labs, Cloud Image, and even Class Method Canada (a subsidiary of a legitimate AWS premier partner). The consistent pricing tiers across these sellers suggest a shared playbook or "franchise of grifting."
• Audacity in Offerings: Examples include Cloud Image selling an AMI for SQLite (a 750KB file that can be downloaded in seconds) as a paid product, or offering GNU Cash (a desktop accounting app) as a server AMI.
• AWS Complicity: AWS is not a victim; it's a vendor that makes money from these sales. The platform provides a "deployed on AWS trust badge" to all AMI products, regardless of their legitimacy or value. Historically, there was no easy way to report these listings, reinforcing their presence.

In essence, Quinn concludes that these vendors are not selling software or even legitimate support, but rather "the gap between the AWS rules that are written and the ones that are enforced."

Technical Deep Dive

▶ Watch: Unveiling supportedimimages.com: The opaque vendor (4:25)

The technical core of this marketplace horror story lies in how these predatory AMIs exploit common automation practices and the inherent trust placed in the AWS platform. Quinn meticulously dissected the mechanism of AMI hijacking and the resulting security implications.

The primary attack vector leverages the way developers and automated build systems select AMIs. In the Reddit case that first drew Quinn's attention, a developer's Packer build system was configured to filter for the official CentOS image by name and then select the most recent match. The malicious actors capitalize on this by naming their lookalike AMIs almost identically to official ones and then frequently updating them. By churning these images "every day or two," they ensure their impostor AMI always appears as the most recently published match, thus hijacking automated pipelines. The only reason the developer's pipeline failed in this instance was a "subscription needed" error, which revealed the underlying issue. The critical fix to prevent such hijacking is to pin the exact product code of the desired AMI, rather than relying on name-based or recency filtering. This crucial detail, however, is often only learned after an organization has already been impacted.

Quinn demonstrated the technical degradation by performing a direct audit. He launched a YUbuntu 24.04 image from one of the predatory sellers on a T3 medium instance and compared it side-by-side with an identical, official Canonical AMI. His findings were stark:

• Outdated Base Image: The seller's YUbuntu image was based on 24.04.2, while the official Canonical version was 24.04.4 – two point releases behind.
• Stale Kernel: The kernel in the marketplace AMI was two major versions back, a critical finding given that kernel updates often contain significant security patches.
• Unpatched Security-Sensitive Packages: A comprehensive check revealed that every security-sensitive package on the marketplace AMI was older. This included vital components like OpenSSH, curl, gnutls, the entire crypto stack, and Intel microcode updates. These older versions are likely vulnerable to known exploits, creating a gaping security hole.
• Misleading Build Dates: The cloudinit service, which stamps a build date into the pseudo file when AMIs are created, showed the seller's image was built on May 9, 2024. Yet, the AWS Marketplace listing falsely claimed it was published on March 16, 2026 – a date in the future, highlighting deliberate misrepresentation. Quinn likened this to "taxidermy," where an old product is presented with a new, misleading label.
• Absence of Hardening or Customization: Further investigation, including byte-for-byte comparisons, confirmed that there was no hardening, customization, or added value whatsoever. The SSH configuration and binaries were identical to the free version, as were the default users, firewall rules, and even the login banner. "There is no hardening, there is no customization, there's no fingerprints on this," Quinn stated.

Crucially, Quinn performed extensive grep operations across the entire file system and found no evidence of embedded malware. This reinforces his argument that the problem isn't a malicious payload, but rather the systematic neglect and deliberate shipping of outdated, unpatched software, which is a security issue in its own right.

Regarding AWS's internal processes, Quinn noted that AWS claims to run "checks on these images" as part of a "curation process." However, they are "very coy" about what these checks entail, refusing to disclose details. Quinn speculates that this process might involve basic scans like ClamAV and boot checks, but it clearly fails to assess the currency of the software, the presence of patches, or the actual value added. This superficial vetting allows vendors to gain a "deployed on AWS trust badge" without earning it, misleading customers into believing the AMIs are vetted for quality and security. The AWS "repackaged software policy" theoretically requires sellers to name themselves in the title, describe value added, and include specific disclosure language, but these rules are "decorative," as many listings violate them with impunity.

Demo / Proof of Concept

▶ Watch: The 'extended support' is a generic contact form (6:20)

While Corey Quinn's talk did not feature a traditional live demonstration of an exploit chain, he effectively provided a powerful proof of concept through his methodical investigation and direct comparison of the AMIs. His "demo" involved a practical, hands-on audit that exposed the core discrepancies and security risks.

Quinn described launching a YUbuntu 24.04 image from one of the problematic AWS Marketplace sellers on a T3 medium instance. He then ran this instance alongside an identical one provisioned from the official Canonical AMI. This side-by-side comparison served as the foundation for his audit.

The "proof" was presented through his detailed analysis: showing screenshots and directly stating the version numbers, kernel discrepancies, and lists of outdated security-sensitive packages (like OpenSSH, curl, gnutls, crypto stack, and Intel microcode updates). He highlighted the misleading build dates stamped by cloudinit versus the advertised publication dates, and the byte-for-byte identical configurations that proved the absence of any value-added hardening or customization.

This investigative approach, while not a live hacking demo, served a more critical purpose: it concretely demonstrated that customers were indeed "paying more for worse security" by running demonstrably older, unpatched, and unsupported operating systems sourced from the AWS Marketplace. It was a clear, evidence-based exposition of the problem rather than a theatrical display of an attack.

Defensive Implications

▶ Watch: Corey spins up an AMI to investigate further (7:00)

The findings presented by Corey Quinn necessitate a proactive and robust defensive strategy for any organization utilizing AWS, particularly the AWS Marketplace. The core implication is that the default trust model of the marketplace is compromised, requiring significant internal vigilance.

1. Mandatory Bill Auditing for Marketplace Charges: The most immediate and impactful defensive measure is to regularly and thoroughly audit AWS bills. Organizations should open their Cost Explorer and search specifically for billing entity = AWS marketplace. This will reveal all vendors and associated hourly fees. Any unfamiliar or suspicious vendors, especially those charging for free operating systems, should be flagged for immediate investigation. Quinn emphasizes, "Check your bill."
2. Scrutinize All Marketplace AMI Usage: Do not assume that an AMI from the AWS Marketplace is inherently secure, up-to-date, or provides value. For any third-party AMI in use, verify its source, version, patch level, and the actual value proposition. Compare its state against official vendor releases.
3. Pin Exact AMI Product Codes in Automation: This is a critical technical defense. Automated build pipelines (e.g., those using Packer or Terraform) must be configured to use the exact product code of the desired AMI, rather than relying on name-based filtering or selecting the "most recent" image. This prevents AMI hijacking by lookalike, frequently updated impostor AMIs.
4. Treat Rogue AMIs as Security Incidents: If outdated, unpatched, or unattributable operating systems from the marketplace are discovered in production environments, they should be immediately declared as a security incident. This triggers formal incident response protocols, ensuring prompt removal and investigation. As Quinn noted, security operations centers (SOCs) invariably treat such discoveries as high-priority incidents, ripping them out without question.
5. Educate Development and Operations Teams: All teams responsible for provisioning resources in AWS, especially those outside core security or infrastructure groups (e.g., marketing, individual developers), must be educated about this marketplace vulnerability. They need to understand the risks of blindly selecting AMIs and the importance of verifying sources and costs.
6. Leverage the "Report This Listing" Feature: While historically absent, AWS has recently added a "report this listing" button in the marketplace. Users encountering predatory or misleading listings should utilize this feature to inform AWS directly.
7. Implement Software Composition Analysis (SCA): Tools that perform Software Composition Analysis can help identify known vulnerabilities within the software components and libraries of deployed AMIs. This can help detect if an AMI is running outdated or unpatched versions of critical packages.
8. Understand Platform Trust Boundaries: Organizations must recognize that the "deployed on AWS trust badge" is not a guarantee of security or quality. The platform's trust machinery can be exploited, and customers bear the ultimate responsibility for verifying the integrity of the software they deploy.
9. Standardize on Official or Internally Hardened AMIs: Where possible, organizations should standardize on official AMIs directly from operating system vendors (e.g., Canonical, Red Hat) or internally built and hardened AMIs that are regularly patched and updated.

By implementing these defensive measures, organizations can protect themselves from both the financial drain and the significant security risks posed by this "AWS Marketplace Horror Story."

Key Takeaways

• AWS Marketplace harbors predatory vendors selling free, outdated, and unpatched operating systems at a significant premium, with no added value or legitimate support.
• Automated build pipelines are vulnerable to AMI hijacking, as lookalike AMIs with nearly identical names and frequent updates can trick systems into deploying unsecure, paid impostors instead of official, free versions.
• These marketplace AMIs introduce critical security risks by shipping older kernels and unpatched security-sensitive packages, creating a false sense of security while exposing organizations to known vulnerabilities.
• AWS profits from and implicitly blesses this model, despite these practices violating its own stated policies and being recognized as security incidents by enterprise security teams.
• Organizations must proactively audit their AWS bills using Cost Explorer to identify unexpected marketplace charges and immediately investigate unknown vendors or suspicious AMI usage.
• Pinning exact AMI product codes is crucial for automated deployments to prevent hijacking, and any discovery of such rogue AMIs in production should be treated as a high-priority security incident, leading to immediate removal.

About the Speaker(s)

Corey Quinn is a prominent and often outspoken voice in the cloud computing community, particularly within the AWS ecosystem. He is widely known as the author of "Last Week in AWS," a popular weekly newsletter characterized by its sarcastic and often "obnoxious" tone, which provides incisive commentary on AWS news and developments. Quinn's professional focus involves "looking at giant AWS bills for a living," where he specializes in identifying and unraveling "stupid" or unexpected charges that appear on customers' invoices. He describes himself as "the jerk that pulls on the thread until the whole sweater winds up coming undone," indicating his dedication to uncovering inefficiencies and questionable practices. He clarifies that he does not run a service to migrate customers off the marketplace but rather uses his expertise in bill analysis to expose systemic issues.