Monitoring Solutions: Attacking IT Infrastructure at its Core

Stefan Schiller (Vulnerability Researcher · Sonar)

TROOPERS23 · Day 2 · Attack & Research · Attack & Research

Overview

In this compelling talk from TROOPERS, Stefan Schiller, a Vulnerability Researcher at Sonar, delves into the critical security implications of monitoring solutions. These often-overlooked systems are the backbone of modern IT infrastructure, providing essential insights into the health and performance of countless devices and services. Schiller highlights why these central components are prime targets for attackers and demonstrates how a series of seemingly minor vulnerabilities can be chained together to achieve full Remote Code Execution (RCE). The presentation features detailed technical deep dives into four popular monitoring solutions: Cacti, OpenNMS, Checkmk, and Netdata, revealing intricate attack paths from initial unauthenticated access to complete system compromise.

Watch on YouTube · Slides

Visual summary for Monitoring Solutions: Attacking IT Infrastructure at its Core by Stefan Schiller
Visual summary for Monitoring Solutions: Attacking IT Infrastructure at its Core by Stefan Schiller

Key moments

  1. 0:00 Introduction and talk overview
  2. 3:20 Why attackers target monitoring solutions
  3. 8:24 Cacti RCE: Old school PHP vulnerability chain
  4. 21:20 CheckMK RCE: Complex unauthenticated to RCE journey
  5. 38:50 Netdata RCE: Command injection via streaming feature
  6. 46:50 Key conclusions, attack vectors, and prevention advice
  7. 51:30 Vendor reactions, collaboration, and audience Q&A

Monitoring Solutions: Attacking IT Infrastructure at its Core

Speakers: Stefan Schiller, Vulnerability Researcher, Sonar

Conference: TROOPERS

YouTube: https://www.youtube.com/watch?v=hGne0DbR6bY

Overview

In this compelling talk from TROOPERS, Stefan Schiller, a Vulnerability Researcher at Sonar, delves into the critical security implications of monitoring solutions. These often-overlooked systems are the backbone of modern IT infrastructure, providing essential insights into the health and performance of countless devices and services. Schiller highlights why these central components are prime targets for attackers and demonstrates how a series of seemingly minor vulnerabilities can be chained together to achieve full Remote Code Execution (RCE). The presentation features detailed technical deep dives into four popular monitoring solutions: Cacti, OpenNMS, Checkmk, and Netdata, revealing intricate attack paths from initial unauthenticated access to complete system compromise.

Schiller's work at Sonar involves discovering vulnerabilities in popular open-source software to improve product security. With a track record of over 150 critical vulnerabilities discovered, his insights into offensive security are both profound and practical. This talk serves as a crucial wake-up call for developers and security professionals, underscoring the necessity of robust security practices even in "internal-only" systems and the severe consequences of underestimating an attacker's ingenuity.

Background

▶ Watch: Introduction and talk overview (0:00)

Modern IT services, from e-commerce to emergency healthcare, are entirely reliant on robust and reliable IT infrastructure. Any disruption to this infrastructure can lead to significant financial losses or, in critical sectors, severe societal impact. To prevent such failures, comprehensive and constant monitoring of all involved devices is essential. Manually monitoring each device individually is unfeasible for most organizations due to the sheer volume. This necessitates centralized monitoring solutions, which collect data from all devices, visualize it, and highlight important events.

Monitoring solutions occupy a privileged position within a network. They maintain connectivity to all monitored devices, often storing sensitive credentials for data collection, and in some cases, can even execute commands directly on these devices. This central role makes them incredibly attractive targets for attackers. A successful compromise of a monitoring solution can grant an attacker access to a treasure trove of sensitive data, leverage powerful functionalities, and provide a perfect pivot point to spread across the entire internal network, potentially compromising the entire infrastructure.

Technically, a common monitoring solution comprises two fundamental components: the web interface and the data collection component. The web interface is responsible for displaying collected information, typically offering both unauthenticated functionalities (like authentication pages) and authenticated dashboards and configuration settings. The data collection component interacts with monitored devices, either polling information from passive agents (e.g., SNMP agents) or receiving data pushed by active agents. The attack surface of the web interface is often limited in its unauthenticated parts, as exposing too much functionality would be counterintuitive. However, the data collection component, especially when allowing active agents to push data, can present a larger and less scrutinized attack surface, as it often involves processing external input. Prior research, such as a 2014 talk and earlier Nagios RCE research presented at TROOPERS, has consistently highlighted the critical vulnerabilities found in these systems.

Key Findings

▶ Watch: Cacti RCE: Old school PHP vulnerability chain (8:24)

The talk's core finding is that monitoring solutions, despite their critical role, frequently harbor complex chains of vulnerabilities that can be exploited for Remote Code Execution (RCE) from an initially unauthenticated state. Schiller demonstrates that a single, isolated vulnerability is rarely sufficient; instead, attackers often leverage a series of lower-impact flaws, escalating privileges or capabilities at each step, until RCE is achieved.

A recurring pattern observed across the diverse case studies is the initial attack vector often targeting the data collection component. This component, designed to receive or pull data from various sources, tends to have a more exposed attack surface than the web interface's unauthenticated login pages. Furthermore, the processing of data from external or seemingly "internal" sources within this component is frequently less scrutinized for security vulnerabilities.

Once an initial foothold or a way to bypass authentication is established, the focus shifts to the authenticated web interface. Schiller points out that this part of the application, while requiring authentication, often becomes the primary vector for achieving RCE. This is attributed to its significantly larger attack surface and a common "internal only" or "authenticated users only" mindset among developers, leading to less stringent security controls. This mindset falsely assumes that once authentication is passed, the user is trusted, making the authenticated functionalities ripe for exploitation through command injection, deserialization, or other severe vulnerabilities.

Finally, the talk emphasizes the paramount importance of input sanitization and defense-in-depth. Many of the demonstrated vulnerabilities could have been prevented by properly sanitizing, escaping, or encoding all user-supplied data, regardless of whether it was directly from a GET parameter or derived from an external data feed. The complex chains also highlight that even mitigating one "link" in such an exploitation chain can break the entire attack, making a multi-layered security approach essential.

Technical Deep Dive

▶ Watch: CheckMK RCE: Complex unauthenticated to RCE journey (21:20)

Stefan Schiller meticulously details the exploitation chains for four distinct monitoring solutions, showcasing a variety of programming languages and architectural patterns.

Cacti: Unauthenticated RCE via X-Forwarded-For Bypass and Command Injection

Cacti, a venerable PHP/MySQL application, provides the first case study. The initial attack surface on the web interface is generally small, with most files including auth.php for authentication. However, the file remote_agent.php is an exception, containing a custom authorization check via the remote_client_authorized function. This function determines the client's IP address using get_client_address and then resolves it to a hostname using gethostbyaddress. It then checks if this hostname matches an entry in the poller MySQL table, which by default contains only the server's own hostname.

The critical flaw lies in get_client_address, which considers various attacker-controllable HTTP headers, including X-Forwarded-For. An attacker can spoof their IP address to 127.0.0.1 (the localhost IP) using this header. get_client_address returns 127.0.0.1, gethostbyaddress resolves it to the server's hostname, and the remote_client_authorized function returns true, granting unauthenticated access to remote_agent.php.

Once authenticated to this endpoint, the poll_for_data function becomes accessible. This function retrieves two GET parameters: host_id (sanitized as an integer via get_filter_request_var) and poller_id (retrieved unsanitized via get_in_filter_request_var). A little later, if a database item's action is set to _POLLER_ACTION_SCRIPT_PHP_, the poller_id variable is concatenated into a string passed to proc_open. By setting poller_id to an arbitrary value like ;id;, an attacker can inject additional commands, leading to Remote Code Execution. The host_id can be brute-forced in seconds, making this a straightforward RCE.

OpenNMS: IPv6 Spoofing, XSS, and GPDetector RCE

OpenNMS, an Enterprise Java solution, presented a more complex chain. The initial web interface attack surface is limited. The data collection component, however, is more promising, featuring a custom SNMP listener that processes SNMP traps. These traps are converted into events or alarms based on XML definitions. For alarms, a reductionKey attribute is used to group similar alarms. While event descriptions are sanitized, the reductionKey for alarms is not. If an attacker injects an XSS payload into the reductionKey field of an SNMP trap, it will be rendered unsanitized on the dashboard when an admin views the alarm.

The challenge is to deliver this trap as an unauthenticated attacker. The SNMP trap listener typically drops traps from unconfigured hosts, but localhost is a default valid host. Standard IPv4 spoofing of 127.0.0.1 from an external interface is blocked by the Linux kernel. However, by encapsulating the trap in an IPv6 packet and setting the source IP to the IPv4-mapped IPv6 address ::ffff:127.0.0.1, the kernel accepts the packet. OpenNMS then translates this to 127.0.0.1, effectively bypassing the host authentication.

With an XSS payload injected into an alarm, an administrator viewing the dashboard triggers the JavaScript. This payload can then interact with the authenticated web interface. Of particular interest is the ability to add a new detector of type GPDetector. This detector class has a script attribute that can be arbitrarily set. When a discovery is initiated using this GPDetector, its connect method calls ExecRunner.exec with the value of the script attribute. This allows an attacker to inject and execute arbitrary operating system commands, achieving Remote Code Execution after the XSS is triggered.

Checkmk: SSRF, LQL Injection, File Deletion/Read Race, and PHP Code Injection

Checkmk, a Python/C++ solution, features the most intricate exploitation chain. The initial unauthenticated web interface is highly restricted. The data collection component, specifically the Agent Receiver which handles agent registration, is more exposed. It accepts HTTP Basic Authentication credentials (which are not immediately validated) and a hostname parameter. This hostname is unsanitized and used to construct a URL path to the main web interface for further processing. This leads to a Server-Side Request Forgery (SSRF) vulnerability, allowing an attacker to make requests to localhost-only endpoints via path traversal (/..).

The SSRF can be leveraged to access the graph_image.py endpoint. This endpoint, after checking for localhost origin, retrieves performance data from the C++ monitoring core using a custom protocol called Live Status Query Language (LQL). A crucial finding is that the ensure_connected function, used to establish the LQL connection, accesses a force_os_user GET parameter. This parameter's value is inserted directly into the LQL request without sanitization. By injecting newlines and a Keep-Alive header, an attacker can inject entirely new LQL queries after the initial one.

One powerful LQL command discovered is COMMAND process_file <path> delete. This command, designed for Nagios compatibility, processes a specified file and then arbitrarily deletes it, regardless of its content. This arbitrary file deletion primitive is key.

Checkmk's session cookie hash relies on a secret value stored in auth.secret. If this file is deleted, the Python backend will recreate it upon an attempted login (even failed). However, this recreation is not atomic: the file is first opened (creating an empty file) then written to. During this brief window, the PHP-based Nagvis add-on (seamlessly integrated) can be accessed. Nagvis's loadSecret function is simpler; if auth.secret is empty or missing, it returns an empty string without error. An attacker can race the Python recreation by deleting auth.secret, attempting a login (triggering recreation), and then immediately accessing Nagvis with a forged session cookie calculated using an empty secret. This grants authenticated access to Nagvis.

Within Nagvis, the get_ova_url action calls read_ova_url, which uses file_get_contents on a user-controlled URL parameter without validating it as an actual URL or checking for local file paths. This provides an arbitrary file read vulnerability.

This file read can be used to read automation_secret, which contains the plaintext password of an automation user. Automation users are typically restricted from the main web interface, a check based on the presence of automation_secret. An attacker can then delete automation_secret via the LQL arbitrary file deletion, and then log in with the plaintext automation user password. The check for automation user status fails due to the missing file, granting full web interface access as an automation user.

Finally, for seamless integration, Checkmk regularly generates PHP files for Nagvis, containing user and role information. The format_php function inserts user data, such as the language_setting, into these PHP files. While single quotes are escaped with a backslash, a backslash followed by a single quote (\') results in the backslash escaping the escaping backslash, leaving the single quote unescaped. By setting the automation user's language setting to a carefully crafted string like en\';system('id');#, an attacker can inject arbitrary PHP code into the generated file, leading to Remote Code Execution when Nagvis evaluates it.

Netdata: Messenger ID Logic Flaw and Command Injection

Netdata features a unique decentralized architecture, with the monitoring software running independently on each device. An interesting feature is streaming, where one device can stream its data to another (e.g., Device 2 streams to Device 1) allowing an admin to view Device 2's data via Device 1. Establishing a streaming connection requires an API streaming key configured on Device 1.

The vulnerability arises from a logic flaw in how Netdata handles the messenger_id parameter during streaming setup. This messenger_id is a unique identifier for the streaming device. While the API key is secret, the messenger_id is not secret and can be retrieved by any unauthenticated attacker from Device 1's /info endpoint. Crucially, the function app_config_get_boolean used to verify if a messenger_id is enabled (to explicitly disallow certain devices) has a default value of true for its enabled property. This means an attacker can use any known messenger_id (retrieved from /info) to establish an unauthorized streaming connection to Device 1.

Once a streaming connection is established, the attacker can send streaming data. Netdata includes a Health Alarms feature that invokes an external script when certain thresholds are exceeded. The command to invoke this script is constructed using a long format string, which is then passed to spawn_exec_command. One of the parameters inserted into this format string is the registry_hostname of the streaming device. The registry_hostname is an attacker-controllable value set during the initial streaming connection setup. By injecting a command into the registry_hostname parameter, an attacker can achieve Remote Code Execution when a crafted health alarm is triggered (e.g., by sending data that exceeds a threshold for a new metric defined by the attacker).

Demo / Proof of Concept

▶ Watch: Key conclusions, attack vectors, and prevention advice (46:50)

Stefan Schiller provided live demonstrations for each of the four monitoring solutions, showcasing the full exploitation chains. For Cacti, the audience witnessed an exploit script brute-forcing host_id and poller_id to inject a reverse shell command, quickly yielding a shell on the Cacti server. The OpenNMS demo involved sending a spoofed IPv6 SNMP trap containing an XSS payload, which, upon admin viewing, triggered JavaScript to create a GPDetector and establish a reverse shell. The Checkmk demonstration highlighted the intricate race condition for file deletion and cookie forging, followed by reading the automation user's password, logging in, changing the language setting, and achieving a reverse shell. Finally, for Netdata, the POC retrieved a valid messenger_id, initiated a streaming connection with a command-injected registry_hostname, triggered an alarm with crafted data, and resulted in a reverse shell on the target Netdata instance. All demos successfully illustrated the practical impact of these chained vulnerabilities.

Defensive Implications

▶ Watch: Vendor reactions, collaboration, and audience Q&A (51:30)

The findings from this talk offer critical defensive implications for both developers and IT security professionals managing monitoring solutions.

For Developers:

  1. Never Trust User Input: This is the most fundamental and frequently violated principle. Every variable, whether directly from a GET parameter, an HTTP header, or derived from an external data feed (like an SNMP trap or streaming data), must be treated as untrusted. Proper sanitization, escaping, and encoding should be applied rigorously before any variable is used in a command execution, database query, or rendered in a web interface. The Cacti, OpenNMS, and Netdata cases clearly demonstrate the consequences of failing to do so.
  2. Implement Defense-in-Depth for Authenticated Functionality: The "internal only" or "authenticated users only" mindset is dangerous. Assume that an attacker will eventually bypass initial authentication. Consequently, all authenticated functionalities, especially those involving file operations, command execution, or configuration changes, must be secured with the same (or even higher) rigor as unauthenticated endpoints. Checkmk's journey from SSRF to RCE through authenticated functions is a prime example.
  3. Address All Vulnerabilities, Regardless of Perceived Impact: As demonstrated by Checkmk, seemingly low-impact vulnerabilities (like SSRF or arbitrary file deletion) can be chained together to achieve catastrophic results. Mitigating even one link in such an exploitation chain can break the entire attack flow, making a holistic approach to vulnerability management essential.
  4. Beware of Race Conditions and Logic Flaws: The Checkmk and Netdata cases highlight how subtle timing issues and logical inconsistencies in how different components handle state (e.g., file existence, default values) can be exploited. Code reviews should explicitly look for such patterns.
  5. Scrutinize Data Collection Components: These components often expose a larger attack surface and process diverse, potentially malicious, input. They require particular attention to input validation and security.

For IT Security Professionals and System Administrators:

  1. Patch Promptly: Monitoring solutions are high-value targets. Ensure that all monitoring software and its components (including add-ons like Nagvis) are kept up-to-date with the latest security patches.
  2. Isolate and Segment: Where possible, deploy monitoring solutions in isolated network segments. Limit their network connectivity to only the necessary monitored devices and administrative interfaces. This can contain the impact of a compromise.
  3. Monitor Your Monitors: Implement security monitoring for your monitoring solutions themselves. Look for unusual activity, unauthorized access attempts, or unexpected command executions.
  4. Audit Configurations: Regularly review the configurations of agents, listeners, and the main monitoring application. Ensure that default credentials are changed, unnecessary services are disabled, and exposed functionalities are minimized.
  5. Be Aware of IPv6 Attack Vectors: The OpenNMS example underscores that IPv6, despite its slower adoption, can introduce new attack vectors due to differing kernel or application-level handling compared to IPv4. Ensure network security controls (firewalls, IDS/IPS) are robustly configured for both IPv4 and IPv6 traffic.
  6. Strong Authentication and Access Control: Implement strong, multi-factor authentication for administrative access to monitoring solutions. Enforce granular role-based access control to limit the capabilities of even authenticated users.

Key Takeaways

  • Monitoring solutions are high-value targets for attackers: Their central position and extensive network access make them ideal pivot points for broader infrastructure compromise.
  • Exploitation often involves multi-stage vulnerability chains: Rarely is a single vulnerability sufficient; attackers combine low-impact flaws to escalate privileges and achieve RCE.
  • Thorough input sanitization is paramount for all data sources: Even seemingly internal or derived data, including HTTP headers, SNMP trap values, or streaming parameters, must be treated as untrusted and properly sanitized, escaped, or encoded.
  • Authenticated functionality requires robust security: Developers and defenders must overcome the "internal only" mindset and apply defense-in-depth principles to all features, assuming that initial authentication may be bypassed.
  • Logic flaws and race conditions present significant attack surfaces: Subtle implementation details, such as non-atomic file operations or default parameter values in configuration checks, can be leveraged for critical bypasses.
  • IPv6 can introduce unexpected attack vectors: Differences in how operating systems and applications handle IPv6 traffic, especially regarding localhost spoofing, can lead to authentication bypasses.

About the Speaker(s)

Stefan Schiller is a Vulnerability Researcher at Sonar, a company known for its products like SonarQube, which helps developers write clean and secure code. With a lifelong passion for software programming, Stefan transitioned into offensive IT security, joining Sonar's research and development team. His work involves discovering vulnerabilities in popular open-source software, using the knowledge gained to enhance Sonar's product engine. Stefan and his team have a remarkable track record, having discovered over 150 critical vulnerabilities, earned multiple Pwnie Awards nominations, and successfully participated in Pwn2Own. His expertise lies in uncovering complex exploitation chains that lead to remote code execution in widely used applications.

Reviews

Dr. Zero (Offensive Security Researcher) — MUST SEE

Schiller systematically dismantles four major monitoring platforms with elegant, multi-stage exploit chains. This is a masterclass in turning low-severity bugs into complete infrastructure compromise. Dead boxes, no marketing bullshit.

Heather Calloway (CISO) — STRONG ACCEPT

This is required reading for your infrastructure and operations teams. It demonstrates exactly how attackers chain minor bugs into full remote code execution across four major IT monitoring platforms. When your visibility tools become the adversary's primary pivot point, you lose the enterprise.

→ Top-rated talks at TROOPERS23

All talks from TROOPERS23