Disregard Buzzwords: Effectively Building Secure Stuff

Disregard Buzzwords: Effectively Building Secure Stuff

Speakers: Adrian Hayes, Managing Director and Principal Consultant, Pulse Security

Conference: Kawaiicon

YouTube: https://www.youtube.com/watch?v=GodMRLCGMPA

Overview

In his Kawaiicon talk, "Disregard Buzzwords: Effectively Building Secure Stuff," Adrian Hayes, Managing Director and Principal Consultant at Pulse Security, delivers a compelling and pragmatic perspective on constructing truly secure IT environments. Drawing from 15 years of experience in pentesting and the unique challenge of building Pulse Security's own production network, Hayes distills effective security into three core, actionable principles: understanding the nature of security controls (concrete vs. fuzzy), ensuring the reasonability and comprehensibility of systems, and diligently managing the attack surface.

This presentation is particularly significant because it bridges the often-wide gap between theoretical security advice and the practical realities of implementation. Hayes candidly shares the lessons learned from having to "eat their own dog food"—applying the very security recommendations they give clients to their own high-security infrastructure. The talk moves beyond abstract concepts, grounding its recommendations in real-world scenarios, including pentesting anecdotes that vividly illustrate the impact of strong versus weak security choices.

Hayes's approach emphasizes a shift from chasing the latest security buzzwords to focusing on foundational, well-understood controls that offer predictable and measurable protection. He advocates for a deliberate, analytical process in system design, where security is not an afterthought but an intrinsic part of the architecture, enabling organizations to build robust defenses that genuinely mitigate risk rather than merely appearing secure.

Background

▶ Watch: Introduction and the 'load-bearing justs' (0:00)

Pulse Security, a firm specializing in pentesting, digital forensics, and incident response for a diverse range of clients from financial services to industrial automation, found itself in a unique position when establishing its own production network. As a pentesting company, the imperative for a highly secure internal network was paramount, given the sensitive data they handle. This necessity presented an opportunity to implement, from the ground up, the security concepts and mitigations they frequently recommended to clients.

Hayes highlights a common pitfall in the security industry: the "just do a thing" mentality. Pentesters, including himself in earlier days, might advise clients to "just segment your giant network" or "just decommission that mainframe." However, the practical challenges of implementing such changes in complex, legacy environments are often immense. By building their own network, Pulse Security gained invaluable perspective, transforming theoretical advice into hands-on experience and becoming better pentesters in the process.

The network itself is built on a foundation of two Proxmox servers, representing a modest but highly capable infrastructure. It hosts approximately 16 distinct workloads, ranging from file sharing and internal team chat to reporting applications. A key design principle was the absence of a traditional physical internal network, embracing a remote-first, "work from anywhere" model even before the COVID-19 pandemic made it commonplace. The entire setup is largely open source, virtualized (virtual networks, firewalls, switches, machines), and software-defined, offering significant flexibility and visibility. A critical aspect of their design is the extensive use of ephemeral infrastructure for client engagements, where servers or workloads are spun up for a specific task and then ruthlessly deleted, embodying the principle that "if it isn't there, it can't get hacked." This pragmatic approach underscores the talk's focus on tangible, effective security measures.

Key Findings

▶ Watch: Red team story: infiltrating an office with fuzzy controls (6:00)

Adrian Hayes outlines three fundamental concepts that guided Pulse Security's approach to building their highly secure production network, which he argues are applicable to any system or organization:

1. Concrete vs. Fuzzy Security Controls: This principle distinguishes between security measures that are mature, predictable, easy to understand, and testable (concrete) and those that are newer, unpredictable, or harder to quantify (fuzzy). Hayes advocates for leaning heavily on concrete controls, as they offer a more reliable and measurable defense posture. Examples of concrete controls include strict packet filtering, cryptographic key authentication, and application whitelisting, while fuzzy controls might encompass machine learning-based detection or general security awareness training.

1. Reasonability: The core idea here is that if you cannot understand an IT environment, you cannot secure it. Reasonability is achieved through two main pillars: consistency in system architecture, configurations, and management, and observability, which involves having the means to deeply monitor and inspect every aspect of the system's behavior. A consistent and observable system allows security practitioners to accurately model threats, predict system behavior, and effectively troubleshoot security issues.

1. Attack Surface Management: This concept focuses on minimizing the points of interaction an attacker can leverage against a system. Hayes posits that "if you can't interact with something, you can't hack it." This involves aggressive reduction of exposed services, stringent egress filtering to control outbound traffic, and architectural patterns like reverse proxies to wall off application functionality, thereby reducing the likelihood of exploitable vulnerabilities.

These three concepts collectively form a robust framework for designing and implementing security that is not only strong but also comprehensible, maintainable, and resilient against evolving threats.

Technical Deep Dive

▶ Watch: Minimize your attack surface: a core security principle (17:00)

Hayes delves into the practical application of his three core concepts, illustrating them with both Pulse Security's own network design and various pentesting anecdotes.

Concrete vs. Fuzzy Security Controls

The distinction between concrete and fuzzy security controls is central to Hayes's philosophy. Concrete controls are characterized by their maturity, predictability, ease of understanding, and testability. An example is basic packet filtering (firewalling), where a rule allows a specific IP address to connect to a defined port (e.g., 443 for an API). Such a rule is unambiguous; its effect is predictable, and its bypass is genuinely challenging for an attacker. Other concrete controls implemented by Pulse include cryptographic key authentication over passwords, multi-factor authentication (MFA), and certificate-based authentication for VPNs, which are demonstrably more robust than password-only schemes. For high-priority workloads, Pulse also employs execution controls like process sandboxing and application whitelisting, preventing unauthorized code execution or writing to disk. A web server, for instance, should not be able to write and execute a malware stager on disk if properly sandboxed. Furthermore, the practice of ruthlessly deleting data and infrastructure after use (ephemeral infrastructure) is a concrete, direct way to reduce long-term risk.

Conversely, fuzzy controls are less predictable and harder to test. Examples include machine learning (ML), artificial intelligence (AI), and Large Language Models (LLMs), which provide statistically likely results but can be susceptible to adversarial input. Security awareness training is another fuzzy control; while valuable, its effectiveness is difficult to measure or guarantee. Similarly, heuristic behavioral malware detection, common in many EDR solutions, relies on patterns that can be bypassed or misinterpreted.

Hayes illustrates the dangers of over-reliance on fuzzy controls with a red team story from Pulse's early days. A colleague fished credentials, and Hayes physically infiltrated an office, installed a remote access Trojan from a flash drive on a break room computer, and exfiltrated data. This exploit leveraged several fuzzy controls: password-only authentication, fuzzy physical security (reliance on staff vigilance rather than access controls), and fuzzy anti-malware (allowing untrusted binary execution). While a "good story," Hayes admits this type of black-box testing didn't provide measurable insights for improvement. Had concrete controls like key card authentication, MFA, and application whitelisting been in place, the attack would have been impossible. Pulse Security no longer offers this type of "cosplaying as an adversary" testing, preferring more structured and measurable approaches.

Reasonability

The concept of reasonability underscores that securing a system is impossible without a thorough understanding of its components and interactions. This is achieved through consistency and observability.

Consistency means maintaining standardized configurations across the environment. Pulse's network, despite hosting 16 diverse workloads, adheres to a consistent architecture: the same operating system, virtualization platform, network configuration, firewalling rules, logging mechanisms, and management channels. This standardization simplifies reasoning about the network's security posture. Introducing new technologies or features creates a tension with consistency; each exception makes the system harder to understand and secure. Hayes acknowledges this tension but strongly advocates for consistency, even if it means foregoing some "hot new features."

Observability is enhanced by Pulse's commitment to open source and self-hosted solutions. This allows deep inspection: enabling additional logs, monitoring network traffic, logging DNS queries, tracking file access, and even reading application source code. This granular visibility is crucial for placing a workload in a highly restricted sandbox and then observing its behavior to fine-tune permissions. A key goal for Pulse is not only preventing external attackers from getting in but also ensuring that if a workload is compromised, it cannot get out and attack other internal systems. This proactive monitoring ensures that workloads remain contained within their "boxes."

Attack Surface Management

Minimizing the attack surface is a highly effective strategy for reducing vulnerabilities. Hayes emphasizes that "if you can't interact with something, you can't hack it." He recounts a fascinating pentest of a ship under construction in a South Korean shipyard. The ship had three network segments: guest Wi-Fi, corporate network, and the critical ship management network (radar, engine control). Hayes's goal was to pivot from the guest network to ship management.

Starting on the guest network, he performed discovery and port scanning. A crucial technique involved running a traceroute to the internet, which revealed the IP addresses of intermediate routers. By scanning these newly discovered subnets, he uncovered several Cisco SSH servers without password authentication. He then brute-forced common Cisco accounts, eventually gaining administrative access to the ship's entire network infrastructure (switches, firewalls, routers). From there, he carefully reconfigured the network to access different segments, eventually identifying a radar control interface on the ship management network. He immediately stopped, proving access without risking operational disruption to a system under construction. The key takeaway: the attack surface exposed from the guest network was far greater than it should have been, allowing access to core network management.

Pulse Security implements robust attack surface management through two primary mechanisms:

1. Actually Good Firewalling (Egress Filtering): While most organizations focus on ingress filtering (preventing attackers from getting in), Hayes stresses the critical importance of egress filtering (controlling outbound traffic). This is vital for mitigating supply chain attacks or backdoor updates. If a compromised workload (e.g., from a malicious GitHub update) tries to exfiltrate data or establish a command-and-control (C2) channel, strict egress rules can block it. Pulse's workloads have very limited internet access. To facilitate necessary operations, they use an internal NTP server, non-recursive DNS (to prevent C2 tunneling via DNS queries), and a filtering HTTP proxy. This proxy whitelists specific domains for updates (e.g., Debian mirrors), allowing necessary outbound communication while blocking everything else.

A significant challenge was discovered by Hayes's colleague Michael: whitelisting a domain that is part of a Content Delivery Network (CDN) can inadvertently allow access to any domain hosted on that CDN. An attacker could register their own malicious domain on the same CDN and bypass the egress filter. This "corner case," which turned out to be a common scenario for many whitelisted update sources, necessitated rapid adjustments to Pulse's egress filtering strategy. Michael's article, "HTTP really isn't that simple," details this issue. From a pentester's perspective, strong egress filtering is a powerful deterrent, often frustrating attempts to establish connect-back shells.

1. Walling Off Application Functionality with a Reverse Proxy: Modern applications often expose hundreds of API endpoints, which, in theory, are protected by strong authentication and access controls. However, in practice, misconfigurations (e.g., a missed authentication requirement on an endpoint) are common vulnerabilities. To mitigate this, Pulse implements a reverse proxy between VPN-connected users and the application attack surface. All user traffic destined for internal applications must first pass through this proxy, which enforces strong authentication. This architecture significantly reduces the direct exposure of application endpoints to untrusted networks, forcing attackers to compromise the authentication layer of the proxy first, a much harder task than finding a single misconfigured API endpoint.

By diligently applying these principles, Pulse Security has built a network that is not only highly secure but also transparent and manageable, demonstrating that effective security is about deliberate design and continuous vigilance, not just buzzwords.

Demo / Proof of Concept

▶ Watch: Real-world 'ship hacking' due to attack surface issues (19:00)

This talk did not include a live demonstration or specific proof-of-concept software. Instead, Adrian Hayes effectively utilized detailed anecdotes from his pentesting career and the architectural decisions made in building Pulse Security's internal network to illustrate the practical implications of his core security principles.

Defensive Implications

▶ Watch: Walling off application functionality with reverse proxies (27:00)

The insights shared by Adrian Hayes offer clear, actionable guidance for defenders seeking to build more secure and resilient systems:

• Prioritize Concrete Controls: Shift focus from unproven or fuzzy security solutions to those that are mature, predictable, and measurable. Implement multi-factor authentication (MFA), cryptographic key authentication, and certificate-based authentication wherever possible. Leverage process sandboxing and application whitelisting for critical workloads to restrict unauthorized execution and file operations. Ruthlessly delete data and decommission infrastructure when no longer needed to eliminate potential attack vectors.

• Embrace Reasonability Through Consistency and Observability:
• Standardize Configurations: Strive for consistency across your IT environment—operating systems, virtualization platforms, network configurations, firewall rules, and logging mechanisms. This reduces complexity and makes systems easier to understand, manage, and secure. Resist the urge to adopt every new technology if it compromises this consistency without a clear, overriding security benefit.
• Maximize Visibility: Utilize open-source tools and self-hosted solutions where appropriate to gain deep insights into system behavior. Implement comprehensive logging, network traffic monitoring, and DNS query logging. The ability to inspect source code and monitor workloads within restricted environments is invaluable for identifying and mitigating risks.

• Aggressively Manage Attack Surface:
• Implement Strict Egress Filtering: This is a non-negotiable control. Beyond just controlling inbound traffic, meticulously filter all outbound connections from internal workloads. Use internal NTP servers, non-recursive DNS, and filtering HTTP proxies that whitelist only absolutely necessary domains for updates and operations. Be acutely aware of the CDN bypass issue and design your egress filters to mitigate it.
• Wall Off Application Functionality: Deploy reverse proxies with strong, centralized authentication in front of your application's API attack surface. This prevents direct interaction with potentially hundreds of application endpoints, forcing attackers to compromise a single, hardened authentication layer.
• Minimize Exposed Services: Regularly review and disable any unnecessary services, ports, or functionalities that are accessible to potential attackers, both externally and internally.

• Continuous Learning and Adaptation: The "CDN bypass" anecdote highlights that even well-designed controls can have unforeseen weaknesses. Defenders must remain vigilant, research potential vulnerabilities in their existing controls, and be prepared to adapt their strategies quickly. Regularly re-evaluate and test security mechanisms to ensure their ongoing effectiveness.

By focusing on these foundational, practical principles, organizations can build a robust defensive posture that effectively mitigates real-world threats, moving beyond superficial security measures to create genuinely secure systems.

Key Takeaways

• Prioritize Concrete Security Controls: Favor mature, predictable, and measurable controls like strong authentication (MFA, cryptographic keys), strict firewalling, application whitelisting, and sandboxing over fuzzy, less predictable measures such as heuristic EDR or general security awareness training.
• Achieve Reasonability Through Consistency and Observability: Build systems that are easy to understand and reason about by maintaining consistent configurations across the environment and maximizing observability through comprehensive logging, monitoring, and the ability to inspect underlying code.
• Ruthlessly Minimize Attack Surface: Actively reduce the number of exposed services and interaction points for attackers. Implement strict egress filtering to prevent data exfiltration and C2 communication, and use reverse proxies to wall off application functionality behind strong authentication.
• Embrace Ephemeral Infrastructure: For sensitive tasks or temporary needs, spin up resources, use them, and then delete them. "If it isn't there, it can't get hacked."
• Be Aware of Egress Filtering Nuances: While critical, egress filtering can be complex. Understand potential bypasses, such as the CDN issue where whitelisting a CDN domain can inadvertently open access to other domains hosted on that same CDN.
• Move Beyond Passwords: Whenever possible, replace password-only authentication with more robust methods like cryptographic key authentication or multi-factor authentication to significantly enhance security.

About the Speaker(s)

Adrian Hayes is the Managing Director and a Principal Consultant at Pulse Security, a company specializing in pentesting, digital forensics, and incident response. With 15 years of experience in the pentesting field, Adrian has worked with a wide array of clients, from major financial institutions and telecommunications companies to SaaS startups, industrial automation firms, and even the automotive and airline industries. In recent years, his role has shifted to primarily managing Pulse Security, while still engaging in consulting and occasional pentesting, allowing him to develop strong opinions and practical insights on effective security practices.