The Tale of the Curiously Empty String

The Tale of the Curiously Empty String

Speakers: Notnotcharlie (Red Team Operator, GitLab)

Conference: CHCon

YouTube: https://www.youtube.com/watch?v=n-M15PH8nhI

Overview

In "The Tale of the Curiously Empty String," Notnotcharlie, a Red Team Operator at GitLab, delivered a compelling talk that masterfully blends narrative storytelling with critical insights into modern cybersecurity challenges. The presentation, initially framed as an incident response saga inspired by Edgar Allan Poe, quickly transitions from a specific technical vulnerability to a profound exploration of organizational culture, developer psychology, and the systemic issues that lead to the reintroduction of known security flaws. This talk transcends a typical technical deep dive, positing that many recurring vulnerabilities are not merely technical oversights but symptoms of deeper process gaps and misaligned priorities within an organization.

The core of Notnotcharlie's message is that effective security goes beyond patching individual vulnerabilities or blocking malicious users. It demands a holistic approach that involves understanding the motivations of developers, investing in secure tooling, and fostering a collaborative, non-judgmental environment where security is integrated into workflows rather than perceived as an obstacle. By dissecting a templating attack that leveraged a Python Remote Code Execution (RCE) vulnerability, Notnotcharlie illustrates how an incident responder's initial, symptom-focused actions, while necessary in the short term, fail to address the root cause, leaving the organization vulnerable to future, similar attacks. The talk serves as a powerful call to action for security professionals to bridge the gap with development teams, transforming security from a siloed function into a shared responsibility underpinned by empathy and pragmatic solutions.

Background

▶ Watch: Edgar Allan Poe parody introduces "curiously empty string" (0:45)

The talk opens with a dramatic, poetic recounting of a late-night incident responder's discovery of a peculiar log entry: a curiously empty string in a field that should never be blank. This initial anomaly quickly escalates into the detection of an active attack against an internal endpoint. The attacker, identified as "Lenor," was observed sending mathematical expressions (e.g., 8*8) to various log fields, indicating an attempt to identify and exploit a server-side template injection vulnerability. Upon successful evaluation of these expressions, Lenor progressed to attempting Python hierarchy traversal and system calls, aiming for Remote Code Execution (RCE) on the targeted system.

Crucially, Notnotcharlie emphasizes that this particular RCE vulnerability, involving Python's object model, is far from new, having been introduced approximately 10 years prior. This historical context underscores a central theme of the talk: many security incidents are not the result of novel zero-day exploits, but rather the reintroduction of well-understood vulnerabilities. The speaker highlights that developers, often under pressure to deliver features quickly, may inadvertently reintroduce these flaws by taking shortcuts or using insecure patterns if the secure path is perceived as too difficult or time-consuming. The immediate incident response—blocking the malicious user and attributing the vulnerability to developer "laziness"—is presented as an insufficient, symptomatic fix that fails to address the underlying process gaps and misaligned priorities that allowed the vulnerability to exist in the first place. This sets the stage for a deeper discussion on transforming incident response from reactive patching to systemic problem-solving and cultural change.

Key Findings

▶ Watch: Deep dive into the Python RCE vulnerability details (8:20)

The talk reveals two primary sets of findings: a specific technical vulnerability discovery and, more significantly, systemic insights into why such vulnerabilities recur and how organizations can truly secure themselves.

On the technical front, the curiously empty string in application logs served as the initial indicator of a templating attack. The attacker was found to be probing an internal, developer-intended Python endpoint by injecting mathematical operations (8*8) to determine if the input was being evaluated server-side. Once confirmed, the attacker escalated their efforts by executing a sophisticated Python Remote Code Execution (RCE) chain. This chain involved leveraging Python's object introspection capabilities: starting from an empty string, accessing its class ("".__class__), traversing the Method Resolution Order (MRO) (.__mro__), identifying a suitable subclass (specifically warnings.catch_warnings), and then using its __init__.__globals__['__builtins__'] to import the os module, thereby enabling arbitrary system command execution. This demonstrated a clear path to full compromise via a known, decade-old vulnerability.

Beyond the technical specifics, the more profound key findings revolve around incident response efficacy and organizational security posture:

1. Insufficient Incident Response: The initial incident response—blocking the attacker and blaming the developer—was deemed inadequate. While it stopped the immediate threat, it failed to address the root cause, leaving the attack vector wide open for future exploitation by a different attacker or the same attacker using new credentials. A truly thorough response requires evicting the attacker, checking for other footholds (e.g., backdoors, SSH keys, C2 beacons), rotating credentials, and performing a comprehensive threat hunt.
2. Process Gaps and Misaligned Priorities: The vulnerability was not merely a coding error but a symptom of deeper organizational issues. Developers often prioritize functionality and speed under tight deadlines, leading them to take shortcuts if secure development practices are perceived as cumbersome. This creates a process gap and highlights misaligned priorities between development and security teams.
3. The Perils of Blame-Based Culture: A security culture that assigns blame discourages developers from engaging with security teams. If developers fear judgment, they are less likely to seek guidance or report potential issues, leading to shadow IT and the silent reintroduction of vulnerabilities.
4. The Need for Systemic Solutions: Continuously addressing symptoms (e.g., patching individual instances of insecure code) is unsustainable and does not scale. Instead, organizations must invest in systemic solutions that make the secure path the path of least resistance for developers, such as providing robust, easy-to-use secure tooling and infrastructure (e.g., self-service test sandboxes).
5. Collaboration Over Confrontation: Bridging the gap between security engineers (who think in attack vectors) and developers (who think in features and maintainability) requires genuine, non-judgmental engagement and organizational support. This involves understanding developer needs and co-creating solutions that serve both security and development goals.

In essence, Notnotcharlie’s talk highlights that many security "problems" are fundamentally "people problems," requiring a shift from purely technical remediation to addressing cultural, process, and psychological factors within the organization.

Technical Deep Dive

▶ Watch: Why blame-based security culture fails to prevent issues (11:00)

The technical core of the talk revolves around a classic, albeit often reintroduced, Python Remote Code Execution (RCE) vulnerability. The attack vector originated from a server-side templating vulnerability on an internal endpoint, likely intended for developer use or debugging, which improperly evaluated user-supplied input.

The initial phase of the attack involved probing for template injection. The attacker, "Lenor," would inject simple mathematical expressions like 8*8 into various input fields. The presence of 64 in the subsequent web response or logs indicated that the input was being processed and evaluated by the server's templating engine, confirming the injection vulnerability. This is a common reconnaissance technique to identify exploitable template engines.

Once the templating vulnerability was confirmed, the attacker proceeded to achieve Remote Code Execution by exploiting Python's object model. Notnotcharlie detailed the specific chain of introspection and method calls required:

1. Starting Point: An Empty String: The attack begins with a fundamental Python object: an empty string (""). This object provides access to its class and, subsequently, the entire Python object hierarchy.
2. Accessing the Class Object: The attacker uses "".__class__ to obtain the string class object (<class 'str'>). In Python, every object has a __class__ attribute that points to its type.
3. Traversing the Method Resolution Order (MRO): From the class object, the attacker leverages .__mro__. The Method Resolution Order is a tuple of class objects that are searched for methods. By traversing this, the attacker can access base classes and, eventually, the object class, which is the root of all Python classes.
4. Finding Subclasses: The object class (or another suitable base class in the MRO) has a .__subclasses__() method. This method returns a list of all classes that directly or indirectly inherit from it. This is a critical step, as it allows the attacker to discover and access other classes defined within the running Python environment.
5. Identifying a Suitable Subclass for RCE: The attacker then iterates through the list returned by .__subclasses__() to find a class that can facilitate RCE. A common target, explicitly mentioned by Notnotcharlie, is warnings.catch_warnings. This class is part of the standard library and, crucially, its __init__ method's global scope contains the __builtins__ dictionary.
6. Accessing Built-in Functions: Once warnings.catch_warnings (or a similar class) is identified, the attacker can access its __init__.__globals__['__builtins__']. The __builtins__ dictionary contains references to Python's built-in functions, including __import__.
7. Importing the os Module: With access to __import__, the attacker can then import os. The os module provides a portable way of using operating system dependent functionality, such as os.system(), os.popen(), or os.read().
8. Executing Arbitrary Commands: Finally, by calling os.system() with a string argument (e.g., os.system('ls -la')), the attacker can execute arbitrary shell commands on the underlying operating system, achieving full Remote Code Execution.

The sequence described by Notnotcharlie is a well-known technique for Python sandbox escapes and RCE in templating engines like Jinja2, Mako, or Django templates if they are misconfigured to allow arbitrary object access. The fact that this vulnerability is "not new" and was introduced "10 years ago" underscores the talk's broader point: the re-emergence of such known issues often stems from systemic failures rather than a lack of documented knowledge. The endpoint targeted was an "internal endpoint meant for devs," highlighting how even seemingly isolated or temporary internal tools can become critical attack surfaces if not secured rigorously.

Demo / Proof of Concept

▶ Watch: Understanding developer motivations: "people do things that make sense" (14:00)

While Notnotcharlie’s talk did not feature a live demonstration or a speaker-led proof of concept, the narrative structure vividly describes the attacker's actions and the incident responder's discovery, effectively illustrating the stages of the exploitation. The "poem" segment and subsequent technical explanation serve as a detailed recounting of the attacker's "demo" of the vulnerability.

The attacker, referred to as "Lenor," initiated their proof of concept by sending simple mathematical expressions, such as 8*8, to various input fields on an internal Python endpoint. The observation of 64 in the application logs or web responses confirmed that the server was evaluating these expressions, thus demonstrating a successful server-side template injection. This initial step is a common and effective way to prove the existence of a templating vulnerability.

Following this successful initial probe, Lenor escalated their attack, demonstrating the ability to achieve Remote Code Execution (RCE). This was inferred by the incident responder observing attempts at "Python hierarchy walking" and "system calls" in the logs. The specific RCE chain, as detailed by Notnotcharlie, involves:

1. Accessing the class of an empty string: "".__class__
2. Traversing the Method Resolution Order (MRO): .__mro__
3. Identifying and accessing specific subclasses, such as warnings.catch_warnings, via .__subclasses__()
4. Leveraging .__init__.__globals__['__builtins__'] to gain access to built-in functions.
5. Importing the os module using __import__.
6. Executing arbitrary operating system commands, likely via os.system().

The talk therefore acts as a post-mortem analysis of an attacker's successful proof of concept, outlining the precise technical steps taken by the adversary to demonstrate and exploit the vulnerability. This approach effectively conveys the technical gravity of the issue without requiring a live demonstration from the speaker, emphasizing the real-world impact of such flaws.

Defensive Implications

▶ Watch: Envisioning a collaborative, systemic incident response process (19:40)

The defensive implications presented in "The Tale of the Curiously Empty String" extend far beyond technical patches, emphasizing a holistic approach to security that integrates process improvements, cultural shifts, and strategic investments.

1. Thorough Incident Response (Beyond the Immediate Fix): The talk critically highlights the inadequacy of merely blocking an attacker and moving on. A truly effective incident response must be systematic and comprehensive:

• Evict the Attacker: Ensure all active sessions and footholds are neutralized.
• Check for Persistent Footholds: Diligently search for backdoors, rogue SSH keys, unauthorized cron jobs, C2 beacons, and any other mechanisms the attacker might have established for future access.
• Rotate Credentials: Compromised credentials must be immediately rotated across all affected systems and services.
• Perform a Threat Hunt: Proactively search for evidence of similar attacks or exploitation attempts by other adversaries, assuming that if one attacker found the vulnerability, others might have too.
• Root Cause Analysis: Go beyond the "what" and investigate the "why." Why did the vulnerability exist? Why was it exploited?

1. Diagnose Process Gaps and Misaligned Priorities: Blaming developers for "laziness" is counterproductive. Instead, security teams must diagnose the wider organizational problems that lead to insecure practices. This often involves:

• Understanding Developer Motivations: Developers are under pressure to deliver features rapidly. If the secure path is difficult, time-consuming, or poorly documented, they will inevitably seek shortcuts.
• Identifying Organizational Pressures: Recognize how tight deadlines, insufficient resources, and a lack of clear security guidelines contribute to vulnerabilities.

1. Foster a Collaborative, Non-Judgmental Security Culture: A blame-based culture creates a chasm between security and development. To prevent developers from avoiding security teams and resorting to shadow IT, organizations must:

• Encourage Informal Engagement: Create easy, non-threatening channels (e.g., chat channels, informal meetups) for developers to ask questions and seek help from security professionals without the formality or perceived judgment of a ticketing system.
• Provide Accessible Guidance: Ensure security best practices and secure coding patterns are easily discoverable and understandable, ideally within existing developer documentation or handbooks.
• Emphasize Shared Responsibility: Position security as a collective goal, not solely the security team's burden.

1. Invest in Secure Engineering Tooling and Infrastructure: The most impactful defensive strategy is to make the secure path the easiest path. This requires organizational investment in:

• Self-Service, Secure Development Environments: Provide tools like easily spinnable, repeatable test sandboxes that are disconnected from production, populated with production-like (but sanitized) data, and automatically shut down. This allows developers to verify production edge cases safely without needing to create insecure internal endpoints.
• Secure-by-Design Frameworks and Libraries: Offer pre-approved, security-hardened libraries and frameworks that developers can use, reducing the likelihood of introducing known vulnerabilities.
• Automated Security Testing: Integrate static application security testing (SAST), dynamic application security testing (DAST), and software composition analysis (SCA) tools directly into CI/CD pipelines to catch vulnerabilities early.

1. Align Security and Development Priorities: Security engineers think in attack vectors; developers think in features, maintainability, and delivery speed. Success requires aligning these perspectives:

• Collaborative Solution Design: Involve developers in designing security solutions that work for their workflows, ensuring they are pragmatic and adopted.
• Pragmatism: Security solutions must be practical and solve real problems for developers, not just abstract security risks.

1. Iterate and Start Small: Transforming an organization's security posture is a journey, not a destination. Start by identifying a single, recurring vulnerability type and collaboratively implementing a systemic solution. This builds trust and demonstrates the value of the new approach.

Ultimately, the talk advocates for a proactive, empathetic, and integrated security strategy where the security team acts as an enabler and partner, rather than a gatekeeper, helping the entire organization build securely by default.

Key Takeaways

• Systemic Issues Drive Repeatable Vulnerabilities: Many security incidents, including Remote Code Execution via known Python object model exploits, are not due to novel zero-days but the reintroduction of old vulnerabilities stemming from process gaps, misaligned priorities, and organizational pressures on developers.
• Thorough Incident Response is Crucial: Beyond immediate containment (e.g., blocking an attacker), effective incident response requires deep investigation for persistent footholds, credential rotation, threat hunting, and comprehensive root cause analysis to prevent recurrence.
• Blame-Based Security Cultures are Counterproductive: A culture of judgment and blame discourages collaboration between security and development teams, leading to developers taking insecure shortcuts and avoiding security guidance, ultimately increasing overall risk.
• Make the Secure Path the Easiest Path: Organizations must invest in secure engineering tooling and infrastructure, such as easily accessible, production-like test sandboxes, to make secure development practices the default and most convenient option for developers.
• Foster Genuine, Non-Judgmental Collaboration: Bridging the gap between security and development requires empathy, understanding developer priorities, and creating informal, non-threatening channels for communication and problem-solving.
• Security Requires Organizational Support and Investment: Achieving a robust security posture is not solely a technical problem; it necessitates organizational commitment to changing culture, investing in developer-centric security tools, and continually iterating on processes.

About the Speaker(s)

The primary speaker for "The Tale of the Curiously Empty String" was Notnotcharlie, who serves as a Red Team Operator at GitLab. Notnotcharlie identifies as a former developer who has transitioned into offensive security operations, focusing on adversary emulation against GitLab's internal infrastructure and products. This background provides a unique perspective, allowing them to understand both the developer's mindset and the attacker's methodologies. Notnotcharlie was also assisted during the presentation by Tristan Reid, who was responsible for sound effects.