A Playbook for Integration Servers

Plumbing the Plumber: A Playbook for Integration Servers

Speakers: Ryan Bonner, Senior Penetration Tester, ProCircular; Guðmundur Karlsson, Security Architect, Islands Banky

Conference: Recon Village

YouTube: https://www.youtube.com/watch?v=wgf5GKrY3nc

Overview

This talk, "Plumbing the Plumber: A Playbook for Integration Servers," delivered by Ryan Bonner and Guðmundur Karlsson at Recon Village, presents a comprehensive guide for identifying and exploiting vulnerabilities in WebMethods Integration Servers. These critical middleware platforms, recently acquired by IBM, act as the central nervous system for large enterprises, connecting disparate systems from legacy mainframes to modern cloud applications like Salesforce. The speakers highlight that despite their crucial role in banking, healthcare, insurance, and financial transactions, these servers often represent an "undocumented plumbing" of the corporate world, largely overlooked by security researchers.

The core message of the presentation is that WebMethods Integration Servers, and by extension, similar enterprise integration platforms, suffer from a combination of historical obscurity, complex documentation, and common developer misconfigurations. This creates a fertile ground for attackers to discover high-impact vulnerabilities. Bonner and Karlsson outline a systematic reconnaissance and exploitation methodology, revealing how simple missteps, such as inadvertently exposing internal services to unauthenticated users, can lead to critical data breaches, remote code execution, or denial of service. Their work serves as a stark reminder of the security risks inherent in complex, legacy enterprise infrastructure.

The talk is particularly significant because it demystifies a class of enterprise software that has historically been difficult for external security researchers to access and understand. By providing a "playbook," the speakers empower the security community to scrutinize these critical systems, which underpin the operations of global giants like AFLAC, Google, AT&T, Medtronic, and Johnson & Johnson. Their findings underscore the need for organizations to pay closer attention to these often-forgotten components of their infrastructure, as their compromise can have far-reaching societal consequences given their role in essential services.

Background

▶ Watch: Introduction to 'Plumbing the Plumber' and speakers (0:00)

WebMethods Integration Servers serve as crucial middleware or integration hubs within large corporate environments. Their primary function is to connect disparate systems, bridging the gap between legacy infrastructure (like mainframes) and modern applications (such as Salesforce). This involves sophisticated data transformation and the facilitation of business-to-business transactions, enabling seamless communication and data exchange across complex IT landscapes. The platform itself has a long history, initially built in 1996, acquired by Software AG in 2007, and most recently by IBM in 2024. This longevity means many instances have been operational for decades, often with configurations that predate modern security best practices.

The primary reason this problem exists and has remained largely unaddressed by the broader security community is a significant information asymmetry. For a considerable period, detailed documentation and training for WebMethods were prohibitively expensive and difficult to obtain. As noted by the speakers, private training could cost upwards of $8,000 to $10,000 for a single week, making it inaccessible for most independent security researchers. Furthermore, the estimated 30,000 pages of documentation were not centralized, scattered across various forums, private knowledge bases, and company-specific resources. This lack of public accessibility created a significant barrier, preventing researchers from understanding the platform's intricacies and potential attack surface. While IBM has recently begun centralizing this knowledge, the historical inaccessibility meant that these systems operated in a security vacuum.

This environment led to a widespread issue of "undocumented plumbing" in critical infrastructure. Developers, often under pressure to integrate complex systems, would make configuration choices that inadvertently introduced vulnerabilities. A common pitfall highlighted by the speakers is the practice of setting service permissions to "default" when encountering access issues. Unbeknownst to many developers, the "default" group often includes "anonymous" users, effectively making internal, potentially sensitive services accessible to anyone on the network, or even the internet, without authentication. Coupled with the fact that these servers have often been "on the internet for like 30 years and nobody has touched it," the combination of obscurity, complexity, and misconfiguration created a perfect storm for the widespread vulnerabilities the speakers discovered. The talk implicitly argues that this scenario is not unique to WebMethods but is likely prevalent across many legacy enterprise integration platforms that have escaped rigorous security scrutiny.

Key Findings

▶ Watch: Why these critical systems are important targets (3:20)

The research conducted by Ryan Bonner and Guðmundur Karlsson unveiled several critical findings regarding the security posture of WebMethods Integration Servers:

1. Widespread Exposure and Neglect: Despite their central role in processing sensitive data for major corporations (AFLAC, Google, AT&T, Metronic, Johnson & Johnson), many WebMethods instances are directly exposed to the internet. These legacy systems have often been deployed for decades and have not received the security attention warranted by their criticality, making them prime targets.

1. Historical Information Barrier: Historically, the sheer volume (estimated 30,000 pages) and decentralized nature of WebMethods documentation, coupled with the high cost of official training ($8,000-$10,000 per week), created an impenetrable barrier for security researchers. This lack of public knowledge allowed vulnerabilities to persist undetected for extended periods.

1. Prevalence of Default Credentials: A significant discovery was the widespread use of default credentials. Through their reconnaissance efforts, the speakers identified approximately "60ish 50ish default credentials just based on the Shodan list" of publicly exposed servers. This highlights a fundamental security hygiene failure across numerous organizations.

1. Accidental Anonymous Access: A core vulnerability mechanism identified is developers inadvertently exposing internal services. When a service fails due to permission issues, developers often "fix" it by changing permissions to "default," unaware that the "anonymous" group is typically part of this default, thereby making the service accessible without authentication. This misconfiguration is particularly dangerous in the VM public package, which contains hundreds of internal services.

1. Vast Attack Surface via API Endpoints: The speakers compiled a list of over 5,100 API endpoints from WebMethods documentation. Fuzzing these endpoints, even as an unauthenticated user, frequently revealed vulnerable services. Responses of 500 indicated a vulnerable service requiring specific input formatting, while 200 meant the service was vulnerable and successfully executed.

1. High-Impact Exploitation Potential: Exploiting these misconfigured services can lead to severe consequences:

• Sensitive Data Disclosure: Finding endpoints that return sensitive information, such as "B2B passwords," was reported as a direct path to critical bug bounties.
• Arbitrary File Read/Write: Services like get_file could be used to retrieve sensitive configuration files, though often limited to specific directories.
• Cross-Site Scripting (XSS): Services like HTML_decode can be leveraged for XSS attacks, bypassing traditional web application firewalls.
• Remote Code Execution (RCE): The OS_command service was highlighted as a "particular favorite" for RCE, though its enablement might depend on specific server configurations.
• Denial of Service (DoS): The speakers admitted to inadvertently shutting down "major universities" and even a "hospital" by executing certain vulnerable services, demonstrating the immediate operational impact of these flaws.

1. Transferable Methodology: The systematic approach to reconnaissance, documentation analysis, and automated fuzzing developed for WebMethods is highly transferable. The speakers explicitly stated their intention to apply this "plumbing the plumber" methodology to other similar integration server platforms, such as MuleSoft, in the coming year, suggesting a broader systemic issue across enterprise middleware.

1. Significant Bug Bounty Success: The methodology proved highly effective, leading to the submission of "80ish bug reports" across various Vendor Disclosure Programs (VDPs) and bug bounty programs, demonstrating the tangible security impact and financial rewards of this research.

Technical Deep Dive

▶ Watch: The challenge of accessing Web Methods knowledge (4:00)

The technical core of the "Plumbing the Plumber" methodology revolves around a systematic approach to reconnaissance, followed by automated testing for common misconfigurations and exposed functionality. This process, which the speakers distilled into a custom tool, aims to uncover the "hidden plumbing" of WebMethods Integration Servers.

1. Comprehensive Reconnaissance:

The first phase involves casting a wide net to identify potential WebMethods instances. The speakers leveraged several public internet scanning services and traditional search engines:

• Shodan: Described as a "search engine for hackers," Shodan was queried using www authenticate "integration server". This specific query yielded between 600 and 700 results, indicating a significant number of publicly exposed servers.
• ZoomEye: Noted for its broader port scanning range (approximately 3,800 ports compared to Shodan's ~1,000), ZoomEye provided additional coverage, particularly in Asian markets, and integrates with the Seebug Vulnerability Database.
• Fofa: Utilized for its robust rule-based fingerprinting capabilities, contributing to a comprehensive asset identification.
• Censys: Praised for its strong quality data and user-friendly query assistance, simplifying the process of crafting effective search queries.
• Google: The tried-and-true method of simple Google searches, such as "who uses web methods," proved surprisingly effective. Correlating job hiring posts with technology usage (e.g., via stack.com) helped identify approximately 2,800 companies using WebMethods.
• CT Log Scanning: For discovering hidden or less obvious infrastructure, the speakers recommended using Gungir, a tool developed by a friend named Gunner. Gungir queries Certificate Transparency logs, which often reveal subdomains and services not indexed by traditional scanners.

2. Automated Identification and Testing (integration_server.py):

To process the vast amount of data gathered during recon, the speakers developed a "terribly vibe-coded" custom tool named integration_server.py. This tool automates critical initial steps:

• Confirmation of Integration Server: It takes a list of IPs and domains, converts them, and then performs simple tests by looking for specific HTTP responses that confirm the presence of a WebMethods Integration Server. This is not "next-level anything," but rather a straightforward check for known indicators.
• Default Credential Testing: The tool then attempts to authenticate using default credentials. This simple yet highly effective test alone yielded "60ish 50ish default credentials just based on the Shodan list," highlighting a widespread and critical security lapse.
• API Endpoint Fuzzing: The most extensive part of the tool's functionality is fuzzing a list of over 5,100 API endpoints compiled from WebMethods documentation. These endpoints are typically invoked via simple GET requests in the format invoke/{service_path}. This mirrors how developers execute services in a browser, where a right-click "run in browser" action generates such a GET request.

3. Exploitation Mechanism and Response Codes:

The core vulnerability stems from developers inadvertently exposing internal services to unauthenticated users. This often occurs when a developer, facing a permission error, sets a service's access to "default," which includes the "anonymous" group. The VM public package alone contains hundreds of such services.

The response codes from the API endpoint fuzzing guide the attacker:

• 404 Not Found: The service does not exist on the target server.
• 401 Access Denied: The service exists but is not vulnerable to unauthenticated access (i.e., anonymous users cannot execute it).
• 500 Internal Server Error: This is a key indicator of vulnerability. The service exists and is accessible, but the request is malformed or lacks necessary parameters. At this point, the attacker must consult documentation to format the request correctly for exploitation.
• 200 OK: This signifies that the service is vulnerable and has been successfully executed. This is where the speakers admitted to inadvertently shutting down servers.

4. Examples of Exploitable Services:

The speakers provided several examples of high-impact services that are frequently misconfigured:

• removePassword / setPassword: While often returning a 500 initially, these services indicate a potential path to credential manipulation if the correct parameters are identified through documentation.
• HTML_decode: This service can be exploited for Cross-Site Scripting (XSS). By crafting malicious input, an attacker can bypass firewalls that might be looking for typical XSS vectors.
• get_file: Allows for arbitrary file disclosure, though it might be limited to specific directories. Even with limitations, these directories often contain sensitive configuration files. Fuzzing for command injection within such services can yield "crazy results."
• OS_command: This is a particularly dangerous service, offering potential Remote Code Execution (RCE). While its full functionality might depend on server configurations, its exposure represents a critical security flaw.

5. Identifying Custom Services:

Beyond the known vendor-supplied services, the speakers emphasized the importance of identifying a company's custom naming conventions for their own services. Developers, often not understanding the implications, will create their own services and, upon encountering permission issues, set them to "default." If an attacker can deduce these naming conventions, they are "almost guaranteed" to find "something really juicy," as these custom services often handle highly sensitive business logic.

The methodology is not merely about identifying a single vulnerability but about systematically mapping the attack surface of a complex, often neglected, enterprise platform.

Demo / Proof of Concept

▶ Watch: Specific Shodan queries to find integration servers (6:20)

While the planned video demonstration of the exploitation process encountered technical difficulties (the PDF would not play), the speakers provided a vivid and detailed verbal account of how a proof of concept (PoC) would be executed and the real-world impact they observed during their research.

The intended demonstration would have shown the simplicity of executing a WebMethods service. By right-clicking on any service within the development environment and selecting "run in a browser," the system automatically generates a GET endpoint. This endpoint follows a predictable pattern: invoke/{path_to_service}. The core PoC involves hitting these invoke endpoints, often without authentication, to test for exposure and vulnerability.

The speakers explicitly described the outcomes of hitting these endpoints:

• A 404 status code indicates the service is not present.
• A 401 means access is denied, suggesting the service exists but is not anonymously vulnerable.
• A 500 status code is a critical indicator: the service is vulnerable and accessible, but the request needs to be properly formatted with the correct variables or parameters, which would necessitate consulting the (now more centralized) documentation.
• A 200 status code signifies that the service is vulnerable and has been successfully executed.

It was at this point, receiving 200 responses, that the speakers admitted to inadvertently causing significant disruption. Guðmundur Karlsson humorously, yet soberly, stated, "I'm sorry if your web method server shut down in the last couple of months. Apologies. We didn't mean to." Ryan Bonner elaborated, confessing, "I really don't want to admit just how often I thought we were just being blocked by firewalls only to realize that no we we shut down the servers and we done that to major universities. I think we even did a hospital once." While they immediately walked back the "incrimination," the anecdote powerfully illustrates the immediate and critical impact of executing these exposed services.

Beyond server shutdowns, they detailed other critical PoCs:

• Data Disclosure: Guðmundur Karlsson twice received maximum critical bounties by simply reporting an endpoint that "returned a bunch of B2B passwords." Ryan Bonner also confirmed finding similar "smoking guns."
• Cross-Site Scripting (XSS): The HTML_decode service was cited as an example that could be used for XSS, noting that such an exploit would "surely bypass any firewall."
• Remote Code Execution (RCE): The OS_command service was highlighted as a "particular favorite" for RCE, although its exploitability might depend on server configurations.
• Arbitrary File Read: The get_file service, while potentially limited to specific directories, could yield sensitive files, making it worthwhile to fuzz for command injection opportunities within its parameters.

Even without a live screen share, the speakers effectively conveyed the ease of exploitation once an exposed service is identified and the severe consequences that can arise from these often-overlooked vulnerabilities.

Defensive Implications

▶ Watch: Utilizing ZoomEye, Fofa, and Censys for deeper recon (6:50)

The findings presented in "Plumbing the Plumber" offer critical, actionable insights for organizations utilizing WebMethods Integration Servers and similar enterprise middleware. Defending against the types of vulnerabilities discovered requires a multi-faceted approach, addressing network exposure, configuration management, developer education, and continuous monitoring.

1. Strict Network Segmentation and Access Control: The most immediate and crucial defensive measure is to never expose WebMethods Integration Servers directly to the internet. These are internal systems designed to bridge enterprise applications, not public-facing web servers. If external access is absolutely necessary for specific services, it must be routed through a robust Web Application Firewall (WAF) or reverse proxy with extremely granular and restrictive access policies, allowing only whitelisted IP addresses and specific, validated endpoints. Implementing strong network segmentation ensures that even if an internal host is compromised, the integration server remains isolated.

1. Enforce Principle of Least Privilege for Services: Organizations must rigorously audit and enforce the principle of least privilege for all services. Developers should be educated on the implications of permission settings, particularly avoiding the "default" group for sensitive services, as it often includes "anonymous" users. A thorough review of all services within the VM public package and any custom-developed services is essential to ensure that no critical functionality is inadvertently exposed without authentication.

1. Comprehensive Configuration Audits: Regular, detailed security audits of WebMethods configurations are paramount. This includes reviewing service permissions, user accounts, and network settings. Automated configuration scanning tools specific to WebMethods should be employed where available, alongside manual expert review. Special attention should be paid to identifying and disabling any unnecessary or potentially dangerous services.

1. Eliminate Default Credentials Immediately: Upon deployment, all default credentials for WebMethods components, including administrative accounts, should be immediately changed to strong, unique passwords. A robust password policy, multi-factor authentication (MFA) where supported, and regular credential rotation should be standard practice.

1. Secure Development Lifecycle (SDLC) Integration: Integrate security considerations directly into the development lifecycle for any custom WebMethods services. This includes mandatory security training for developers on the platform's unique security challenges, secure coding guidelines, and peer code reviews focused on permission settings and input validation. Developers must understand the risks associated with exposing internal functionality.

1. Robust Input Validation and Sanitization: All services, particularly those that accept user input or interact with the underlying operating system (e.g., OS_command, get_file), must implement rigorous input validation and sanitization. This is critical to prevent common vulnerabilities like command injection, XSS, and directory traversal. Parameterized queries should be used wherever possible to mitigate injection risks.

1. Proactive Patch Management: Keep WebMethods Integration Server software and all related components up-to-date with the latest security patches and updates from IBM/Software AG. Legacy systems are often neglected in patch cycles, but given their critical role, they should be prioritized.

1. Comprehensive Logging and Monitoring: Implement extensive logging for all integration server activities, including service invocations, authentication attempts (both successful and failed), and system errors. Integrate these logs into a Security Information and Event Management (SIEM) system for real-time monitoring and alerting on suspicious patterns, such as unusual service calls, repeated 500 errors from external sources, or unexpected server restarts.

1. Maintain an Accurate Asset Inventory: Organizations need a clear, up-to-date inventory of all WebMethods instances, their versions, network exposure, and the services they host. This helps identify the attack surface and prioritize defensive efforts.

By adopting these defensive strategies, organizations can significantly reduce their exposure to the types of high-impact vulnerabilities highlighted by Bonner and Karlsson, transforming these critical "plumbers" from potential points of failure into securely integrated components of their enterprise architecture.

Key Takeaways

• Legacy Enterprise Systems are High-Value Targets: Complex, long-standing middleware like WebMethods Integration Servers often operate as critical "plumbing" in large corporations but suffer from historical neglect, making them prime targets for attackers.
• Systematic Reconnaissance is Highly Effective: Utilizing a combination of internet scanners (Shodan, ZoomEye, Fofa, Censys), traditional search engines, and CT log scanning (Gungir) provides a comprehensive method for discovering exposed and often forgotten infrastructure.
• Developer Misconfigurations Drive Critical Vulnerabilities: A prevalent issue is developers inadvertently exposing internal, sensitive services to unauthenticated users by setting permissions to "default," which includes the "anonymous" group. This is a common and dangerous oversight.
• Automated Fuzzing Uncovers Widespread Flaws: Fuzzing thousands of API endpoints, even with simple GET requests, can quickly identify default credentials (50-60 instances found), information disclosure vulnerabilities (e.g., B2B passwords), and services leading to XSS, RCE, or denial of service.
• Impact Can Be Severe and Immediate: Exploiting these vulnerabilities can lead to critical consequences, including sensitive data breaches, arbitrary file access, remote code execution, and even accidental server shutdowns of major organizations.
• Methodology is Transferable to Other Middleware: The "plumbing the plumber" approach is not unique to WebMethods; it serves as a robust playbook applicable to other legacy integration platforms (e.g., MuleSoft) that may suffer from similar issues of obscurity and neglect.

About the Speaker(s)

Ryan Bonner is a Senior Penetration Tester at ProCircular. He identifies as an experienced bug bounty hunter and is a first-time presenter at Recon Village, having previously pretended to be a teacher. His work focuses on discovering security flaws in complex systems and advocating for better security practices.

Guðmundur Karlsson, who also goes by Carl or Kumali, is a Security Architect at Islands Banky. He has extensive hands-on experience with WebMethods, having worked with the platform for nearly a decade (skating it in 2022). Guðmundur is a top-ranked hacker on Iceland's bug bounty platform, Defend Iceland, and collaborated with Ryan Bonner to develop the integration_server.py tool and dissect WebMethods security.