Collaborative response to emerging critical RCE vulnerabilities in exposed edge devices

Collaborative response to emerging critical RCE vulnerabilities in exposed edge devices

Speakers: Piotr Kijewski, CEO, The Shadowserver Foundation

Conference: Virus Bulletin

YouTube: https://www.youtube.com/watch?v=rMqdCUi_3PI

Overview

In an era where critical Remote Code Execution (RCE) vulnerabilities in exposed edge devices are rapidly exploited by diverse threat actors, the need for a coordinated and agile response has never been more pressing. This talk, delivered by Piotr Kijewski of The Shadowserver Foundation, delves into the unique approach of a non-profit organization in combating these pervasive threats. Kijewski articulates how Shadowserver leverages extensive global monitoring, sophisticated scanning techniques, and a vast network of honeypots, all underpinned by a deep commitment to collaboration, to identify vulnerable and compromised systems worldwide.

The presentation highlights Shadowserver's critical role as a global information sharing hub, providing free, actionable threat intelligence to national CERTs, network owners, and law enforcement agencies across 175 countries. By focusing on automation and a public benefit mission, Shadowserver demonstrates how a lean organization can achieve unparalleled visibility into internet-wide exposures and exploitation trends, often surpassing the capabilities of much larger commercial entities. The talk underscores the power of collective defense and the imperative for cross-sector cooperation in securing the internet's ever-expanding attack surface.

Kijewski, with a background rooted in national cybersecurity and incident response, positions Shadowserver not as a competitor to commercial vendors but as a crucial enabler for large-scale impact. His insights reveal the intricate dance between rapid vulnerability identification, ethical data collection, and timely dissemination of intelligence, showcasing how collaborative efforts can significantly reduce the window of opportunity for attackers and bolster global cyber resilience.

Background

▶ Watch: Shadowserver's non-profit, public benefit mission (3:15)

The Shadowserver Foundation operates as a dual non-profit entity (a 501c3 in the US and a Stichting in the Netherlands), driven by a singular mission: to make the internet more secure for all. Unlike commercial threat intelligence providers, Shadowserver shares information with network defenders at no cost, specifically tailored to their owned network assets. This model aims to raise the baseline of cybersecurity globally, providing access to critical data for entities that might otherwise be unable to afford it. Their services are extensive, reaching national CERTs, ISPs (over 3,000 subscribers), law enforcement, and critical infrastructure operators worldwide.

Shadowserver's operations are sustained through grants, donations, and sponsorships, with significant backing from national governments (including the UK, Canada, EU, Netherlands, and Luxembourg) and philanthropists. This funding model allows them to maintain a massive infrastructure, including a data center in Oakland, California, housing 70 racks and 1,100 servers, representing an estimated $40 million in equipment, managed by a mere three individuals. This extreme automation is crucial for processing the 15 petabytes of data they hold and the approximately 1 billion daily events they share.

Their core capabilities are built upon five primary services:

1. Global Monitoring and Victim Notification: Daily surveillance of exposed endpoints and identification of compromised hosts.
2. Large-Scale Internet-Wide Early Warning: Public dissemination of critical vulnerability information.
3. Law Enforcement Investigations and Support: Providing data for takedowns and cybercrime investigations.
4. Cybersecurity Capacity Building: Funded projects, particularly in Africa and the Indo-Pacific, to enhance local cybersecurity capabilities.
5. Funded Public Benefit Projects: Grants used to improve their services and expand free data sharing.

At the heart of their intelligence gathering are four major data sets:

• Sinkholing Data: Shadowserver sinkholes over 400 malware families, monitoring more than a million domains at any given time. This provides visibility into infected hosts beaconing out to these domains, including prominent botnets and sophisticated malware like SolarWinds Sunburst and the recent BackBox 2 malware affecting 8 million Android devices.
• Scanning Data: For 11 years, Shadowserver has been scanning the entire internet, similar to services like Shodan and Censys. These daily (sometimes multiple times a day) scans focus on detecting vulnerable and compromised devices and tracking threat actor infrastructure. They collect 8 million certificates daily and are notably the top vulnerability scanner for IPv6 addresses, scanning nearly four billion IPv6 addresses. Crucially, they never exploit vulnerabilities but conduct benign checks. If exploitation is required for assessment, they rely on partners.
• Sensor Data (Honeypots): Thousands of honeypots are strategically deployed across 1,000 data centers in over 100 countries. These honeypots mimic commonly attacked edge devices (firewalls, routers, appliances), allowing Shadowserver to capture clean, malicious telemetry from new attack campaigns and zero-days.
• Sandbox Data: An internal equivalent to VirusTotal, Shadowserver maintains one of the world's largest malware repositories, with over 2 billion samples by hash and collecting more than a million new samples daily. They operate their own antivirus engine and sandbox technology, including bare-metal analysis.

This extensive infrastructure and unique non-profit model position Shadowserver as a vital, unbiased, and highly effective player in the global fight against cyber threats, particularly the rapid exploitation of RCEs in edge devices.

Key Findings

▶ Watch: Ethical scanning practices and community collaboration (26:30)

A significant finding highlighted in the talk is Shadowserver's unexpected leadership in identifying newly exploited vulnerabilities in the wild. Despite its lean team and non-profit status, Kijewski proudly states that Shadowserver has frequently been among the first, if not the first, to report CVEs being actively exploited. This achievement is particularly notable when compared to the vast resources of commercial cybersecurity vendors. This operational data also contributes to the Known Exploited Vulnerability (KEV) Catalog and the Exploit Prediction Scoring System (EPSS), solidifying their role in global vulnerability management standards.

The success stems from Shadowserver's highly automated systems, global honeypot network, and, critically, its collaborative model. By working closely with national CERTs, security researchers, and even vendors, they receive early intelligence and validation that enables rapid development and deployment of new detection signatures. This agility allows them to quickly identify widespread exploitation patterns and disseminate remediation data, drastically reducing the window of opportunity for attackers. The case studies presented serve as concrete examples of this rapid response capability and the profound impact of their collaborative, data-driven approach on global cybersecurity.

Technical Deep Dive

▶ Watch: Citrix NetScaler: Rapid response and webshell detection (34:10)

Shadowserver's technical prowess lies in its ability to conduct internet-scale monitoring and analysis with remarkable efficiency and precision. Their scanning infrastructure operates continuously, performing daily scans across hundreds of ports and protocols. The collected data—including certificates, HTTP headers, cookies, and HTML bodies—is then rigorously fingerprinted against thousands of proprietary rules. This process attributes exposed assets to specific vendors, products, and versions, creating a comprehensive, daily snapshot of the global attack surface. This baseline data is invaluable for rapidly initiating targeted vulnerability scans. For instance, if a new RCE is announced for a specific product like FortiGate, Cisco, Citrix, or Microsoft Exchange, Shadowserver already possesses the inventory of these devices and can immediately mine existing data or launch targeted queries to assess vulnerability without full exploitation.

The development of these scans is often a collaborative effort. Government agencies and other vendors frequently assist in designing new scan methodologies or provide feedback to refine existing ones, allowing Shadowserver to achieve high accuracy and deploy new detections within hours or days of a vulnerability disclosure. Similarly, identifying compromised devices, web shells, or implants is often initiated by insights from local partners who have experienced an infection. Shadowserver then rapidly implements remote detection techniques into its scanning framework, effectively learning from a single incident to warn the entire community about similar infections.

Strict ethical boundaries govern all scanning activities. Shadowserver adheres to a "red line" policy, explicitly avoiding any actions that could be intrusive or harmful. This includes:

• No exploitation of systems.
• No logging in or unauthorized access to panels.
• No directory traversals.
• No sending of post data that could obtain sensitive information.
• No "write" actions on APIs.
• No use of credentials, ever.

Their motto is "taking nothing but pictures, leave nothing but footprints," aiming to be as unintrusive as possible, minimizing log generation, and avoiding interference with other defenders. The universal recognition of Shadowserver's scanning IPs also significantly reduces abuse complaints, demonstrating a widespread acceptance of their public benefit mission.

In parallel to scanning, Shadowserver's honeypot (sensor) network provides crucial real-time threat intelligence. Thousands of these sensors are deployed globally, mimicking a wide array of vulnerable edge devices such as firewalls, routers, and other network appliances. This setup allows them to observe and collect attack telemetry when new vulnerabilities emerge. Attackers performing reconnaissance or launching broad campaigns often hit these honeypots, providing clean, malicious traffic that is then decoded and classified by CVE or exploit type. This system is instrumental in discovering N-day and sometimes even zero-day exploits, which are then reported to vendors for remediation. The rapid deployment capability allows new honeypot configurations to be rolled out globally within minutes or hours in response to emerging threats.

The sheer volume of data processed by Shadowserver necessitates extreme automation. Their data center, with its 1,100 servers, processes 15 petabytes of data, generating approximately 1 billion event-specific alerts daily. This data is shared in feeds, primarily as CSV or JSON files, filtered by ownership (autonomous systems, CIDR blocks, domains, geolocation) to ensure that organizations only receive data relevant to their networks. This automated, high-volume, and ethically constrained approach allows Shadowserver to serve as a critical "global plumber," ensuring vital threat intelligence reaches the right hands quickly and at scale.

Demo / Proof of Concept

▶ Watch: SharePoint mass intrusions: Coordinated incident response (48:00)

While the talk did not feature a live technical demonstration, Piotr Kijewski presented several compelling case studies that effectively served as real-world proofs of concept for Shadowserver's capabilities and collaborative model. These examples illustrate their rapid response, detection methodologies, and impact:

Citrix NetScaler (CVE-2023-3519)

In mid-2023, a critical RCE vulnerability in Citrix NetScaler (now Citrix ADC and NetScaler Gateway), CVE-2023-3519, emerged. Shadowserver's pre-existing device inventory allowed them to immediately identify all exposed NetScaler instances.

• Rapid Response: On July 19, 2023, the day CISA added the vulnerability to its KEV list, Shadowserver, aided by community feedback, developed and deployed a preliminary CVE scan.
• Iterative Improvement: Initial scans missed some instances, but feedback from volunteers on the ground quickly helped refine identification, achieving near 100% accuracy with zero false positives.
• Honeypot Deployment: A Citrix honeypot was rapidly deployed, enabling observation of exploitation attempts as a Proof-of-Concept (PoC) was published, showing an explosion in attack activity.
• Web Shell Detection: Trusted partners provided information on China Chopper webshells being installed. Shadowserver developed and deployed a scan for these webshells within a week.
• Collaborative Expansion: Initially, Shadowserver identified around 600 webshells. However, faced with an ethical dilemma of brute-forcing 40,000 potential webshell names (which could generate excessive logs and appear intrusive), they partnered with Fox-IT and DIVD. These organizations, without the same scanning constraints, conducted more aggressive scans and shared their data. This collaboration led to the discovery of over 1,500 compromised instances, significantly expanding the scope of detection.
• Public Advisories: Shadowserver published a technical blog and public advisories, filling a gap in public information and leading to further collaboration with companies like Mandiant for incident response.

Cisco IOS XE "Bad Candy" Implants

In autumn 2023, Cisco published a blog detailing implants on Cisco IOS XE routers via the software management interface.

• Immediate Scan Development: Shadowserver immediately implemented scans based on the detailed information provided by Cisco.
• Attacker Adaptation: Attackers were also reading Cisco's blog and adapting their implants, leading to a "race" where Shadowserver constantly updated its scans as new implant variations emerged.
• Honeypot Monitoring: A Cisco IOS honeypot was rolled out to monitor implant scans and updates in real-time.
• Persistent Compromise: Initial scans revealed over 40,000 compromised Cisco IOS XE routers. Over time, while some cleaned up, a significant portion (around 50%) remained infected, with attackers no longer needing to hide. As of two days before the talk, over 20,000 Cisco routers were still infected globally, primarily in Mexico, US, India, Chile, and Peru, with Europe showing better remediation rates.
• Adversary Profiling: Kijewski noted a presentation by another company profiling the adversary (attributed to China) that showed them targeting tier 2 operators and ISPs, while "avoiding Cisco, Shadowserver, Amazon, Microsoft, and Chinese telcos," humorously highlighting Shadowserver's perceived impact.

Palo Alto Networks PAN-OS Vulnerability

Shadowserver collaborated directly with Palo Alto Networks on a critical vulnerability in the management interface of PAN-OS devices.

• Vendor Partnership: Shadowserver was tipped off about the vulnerability and, based on their existing scans, identified exposed and vulnerable management interfaces.
• Direct Information Sharing: Unusually, Shadowserver shared all its findings directly with Palo Alto. The vendor then leveraged its sales channels to reach affected customers, resulting in a "very sharp and very unusual" drop in exposure rates.
• Artifact Tracking: Partnering with Watchtower, Shadowserver used their PoC code and methodologies to track specific compromise artifacts on devices.
• Honeypot for Attacker TTPs: A PAN-OS honeypot was deployed to observe attacker queries, allowing Shadowserver to replicate these queries in their own scans to identify compromised artifacts more effectively.

SharePoint Compromises

A recent wave of SharePoint compromises, particularly in the Netherlands, saw Shadowserver dragged into a mass intrusion response.

• Manual and Automated Alerting: Initiated by I-Security and Watchtower, Shadowserver mobilized its team for both manual and automated alerting of affected parties.
• FQDN Challenge: The key challenge was that detecting all SharePoint instances required scanning fully qualified domain names (FQDNs) and virtual hosts (vhosts), as IP-based scanning was insufficient.
• Collaborative Data Expansion: New data batches from DIVD and collaboration with Leaks (who performed more aggressive FQDN scanning) significantly increased the detection rate. Further partnerships with Validen and CERT-Bund (BSI) refined their understanding and detection capabilities.
• Improved Visibility: Over two weeks, Shadowserver's detection of exposed SharePoint devices leaped from 9,000 to over 16,000, demonstrating the power of FQDN-based scanning.
• Persistent Threat: Despite the initial media attention, over 1,400 vulnerable SharePoint instances remain exposed, indicating ongoing targeted incidents that are not widely publicized. The incident even made the front page of Al-Jazeera, highlighting its global impact.

These case studies collectively demonstrate Shadowserver's multifaceted approach: leveraging global scanning, deploying targeted honeypots, fostering deep collaboration with various entities, and maintaining strict ethical guidelines, all to provide timely and impactful threat intelligence against emerging RCE vulnerabilities.

Defensive Implications

▶ Watch: Call to action: Collaborate for global cyber impact (52:00)

The insights from Shadowserver's work offer several critical defensive implications for organizations facing the relentless onslaught of RCE vulnerabilities in edge devices:

1. Leverage Free Threat Intelligence: Organizations, especially national CERTs, ISPs, and critical infrastructure, should subscribe to Shadowserver's free daily reports. This provides network-specific, actionable intelligence on exposed services, vulnerabilities, and compromises without cost, significantly raising their baseline security posture.
2. Prioritize Patching of Edge Devices: The rapid exploitation of RCEs in devices like Citrix NetScaler, Cisco IOS XE routers, and Palo Alto Networks firewalls underscores the extreme urgency of patching. Attackers are highly agile, often exploiting vulnerabilities within hours or days of public disclosure.
3. Recognize Persistent Vulnerability: A significant portion (often 50% or more) of vulnerable devices remain unpatched long after disclosures, creating a persistent and easily exploitable attack surface. Organizations must ensure robust patch management processes and allocate resources for continuous remediation, especially for internet-facing assets.
4. Implement Comprehensive Asset Discovery: Traditional IP-based scanning may not be sufficient for all assets. As seen with the SharePoint case, FQDN and vhost scanning are crucial for full visibility into complex environments and cloud-hosted services. Defenders need to understand their entire attack surface thoroughly.
5. Contribute to Collective Defense: The collaborative model championed by Shadowserver demonstrates that sharing threat intelligence, vulnerability details, or even anonymized compromise data can have a massive global impact. Organizations with unique insights or data should consider contributing to non-profits like Shadowserver to enable broader community protection.
6. Monitor for Compromise Artifacts: Beyond just patching, organizations should actively scan for indicators of compromise (IOCs), webshells, and implants, as attackers often maintain persistence even after initial vulnerabilities are addressed. Leveraging methodologies shared by trusted partners can enhance these detection capabilities.
7. Be Wary of Targeted Attacks: Even after widespread patching, remaining vulnerable devices become targets for more focused and less publicized attacks. Continuous monitoring and threat hunting are essential to detect these evolving threats.

By embracing these defensive strategies and actively participating in the collaborative ecosystem, defenders can significantly enhance their resilience against the continuous wave of critical RCE vulnerabilities.

Key Takeaways

• Collaboration is Paramount: Effective response to widespread RCE vulnerabilities requires deep collaboration between non-profits, national CERTs, law enforcement, security researchers, and even commercial vendors. No single entity can solve the problem alone.
• Shadowserver's Unique Role: As a non-profit, Shadowserver provides free, large-scale, and highly automated threat intelligence, acting as a critical global hub for distributing actionable data to network defenders worldwide.
• Automation is Key to Scale: Managing 15 petabytes of data and processing 1 billion daily events with a small team is only possible through extreme automation, enabling rapid response to emerging threats.
• Edge Devices are Prime Targets: Critical RCEs in internet-exposed edge devices (firewalls, routers, application gateways) are constantly emerging and quickly exploited, necessitating immediate attention from defenders.
• Persistent Vulnerability is a Major Problem: A significant portion of vulnerable devices often remains unpatched for extended periods, creating a long-term, easily exploitable attack surface for threat actors.
• Ethical Data Collection Drives Trust: Shadowserver's strict "red line" policies (no exploitation, no intrusive actions) build trust within the community, encouraging data sharing and reducing abuse complaints, which is crucial for effective mass scanning.

About the Speaker(s)

Piotr Kijewski is the CEO of The Shadowserver Foundation, a non-profit organization dedicated to making the internet more secure for all. His background is deeply rooted in national cybersecurity and large-scale incident response, rather than the commercial vendor space. He is one of the original creators of CERT.PL, Poland's national Computer Emergency Response Team, and has been with Shadowserver since 2016. Kijewski is actively involved in various other non-profit cybersecurity initiatives, including the HoneyNet Project, FIRST.org, the CyberPeace Institute, and DIVD (Dutch Institute for Vulnerability Disclosure). Despite his leadership role, he remains hands-on, often developing fingerprints, exploit signatures, and assisting with disruption operations. His extensive experience and commitment to collaborative, public-benefit cybersecurity efforts underscore his insights into global threat remediation.