Azure Networking Dark Arts: The Implicit Paths Your Diagrams Don't Show-Achia Rosenfeld & Kobi Rubin

Azure Networking Dark Arts: The Implicit Paths Your Diagrams Don't Show

Speakers: Achia Rosenfeld (Software Engineer, Act Security), Kobi Rubin (Head of Research, Act Security)

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=AWKsyqhxY0o

Overview

In this fwd:cloudsec presentation, "Azure Networking Dark Arts: The Implicit Paths Your Diagrams Don't Show," Achia Rosenfeld and Kobi Rubin from Act Security peel back the layers of Azure's often opaque networking behaviors. They illuminate how Azure's "easy-to-use" design frequently obscures critical routing decisions, leading to implicit traffic paths that bypass security controls and defy conventional network diagrams. The talk serves as a crucial wake-up call for organizations operating complex Azure environments, demonstrating how subtle, undocumented defaults can introduce significant security vulnerabilities.

Rosenfeld and Rubin, drawing from their expertise in cross-cloud network and identity analysis, meticulously detail two distinct scenarios where Azure's automatic route injection and service interactions lead to unexpected and insecure traffic flows. Their investigation highlights the limitations of Azure's built-in diagnostic tools and underscores the necessity for rigorous, independent verification of network configurations. This presentation is essential for cloud architects, security engineers, and anyone responsible for maintaining a secure posture in Azure, as it exposes the "dark arts" that can undermine even the most meticulously planned security strategies.

The core message resonates with the reality of many large-scale cloud deployments: networks evolve organically, often without a master plan, leading to a patchwork of configurations. In such environments, understanding the actual traffic flow, rather than just the intended one, becomes paramount. The speakers emphasize that while Azure provides robust security controls, the challenge lies in knowing what you have and how it's truly connected, especially when the platform itself introduces hidden variables that deviate from explicit configurations.

Background

▶ Watch: Azure's implicit paths and hidden behaviors explained (2:00)

The genesis of the problems discussed in "Azure Networking Dark Arts" lies in the inherent complexity and rapid evolution of large cloud networks. As Achia Rosenfeld aptly describes, most large networks are built like a "puzzle with missing pieces," growing through iterative fixes and patches without a cohesive master plan. This organic growth model, common in on-premises environments, is exacerbated in the cloud where abstraction layers and managed services introduce new layers of hidden complexity. Organizations migrating to or expanding within Azure often find themselves grappling with environments that are "easy to use" but profoundly "not easy to see," making comprehensive security a significant challenge.

Azure, like other hyperscale cloud providers, aims to simplify network configuration through automation and default behaviors. While this approach streamlines deployment for many users, it also means the platform can "do things you didn't ask for" silently. These implicit actions, often buried deep within documentation or discovered only through extensive troubleshooting, can create routing paths that bypass explicit security controls like firewalls. The fundamental issue is a disconnect between the logical network diagrams that engineers design and the actual, operational network topology influenced by Azure's underlying mechanisms.

The talk frames its examples within a common hub-and-spoke architecture, a prevalent design pattern in Azure for centralizing network services like firewalls. In this model, a central "hub" virtual network (VNet) hosts shared services, while "spoke" VNets host applications and resources. Traffic between spokes or from spokes to external resources (like on-premises networks or public internet) is typically intended to flow through the hub's firewall for inspection and policy enforcement. The scenarios presented demonstrate how Azure's default behaviors can subtly undermine this fundamental security principle, leading to direct connections that circumvent the firewall, leaving sensitive data exposed or creating unmonitored communication channels.

Key Findings

▶ Watch: Discovering Azure's hidden, injected service endpoint route (5:00)

The research presented by Achia Rosenfeld and Kobi Rubin unveiled several critical findings regarding Azure networking, challenging common assumptions and highlighting areas where implicit platform behaviors can lead to significant security gaps:

• Implicit Route Injection: Azure silently injects routes into route tables, particularly when services like Service Endpoints or VNet peering are configured. These injected routes often have higher specificity (e.g., a /32 for a specific IP) and, due to the longest prefix match rule, take precedence over broader user-defined routes (UDRs) intended to direct traffic through security appliances like firewalls.
• Bypassing Security Controls: The implicitly injected routes can cause traffic to bypass central security controls (e.g., an Azure Firewall in a hub VNet). This direct routing means that traffic intended for inspection and policy enforcement can flow unimpeded, creating unmonitored and potentially insecure paths.
• Misleading Diagnostic Tools: Azure's built-in diagnostic tools, such as Network Watcher's Next Hop and Connection Troubleshoot, can provide incomplete or misleading information. While they might indicate "valid" connectivity or a specific next hop, they don't always reveal the full, end-to-end path or account for asymmetric routing issues that can prevent actual application-layer communication. In the case of Connection Troubleshoot, it may report a valid connection even when application traffic fails due to routing discrepancies.
• Asymmetric Routing Challenges: The interaction between different Azure networking services, specifically Azure VPN Gateway and Azure Firewall, can lead to asymmetric routing. This occurs when the outbound traffic path differs from the inbound return path, causing stateful firewalls to drop legitimate packets because they don't recognize the return leg of a session.
• Defaults are "Additive and Silent": Azure's default behaviors are often "hidden in documentation," requiring extensive research and testing to uncover. They don't explicitly ask for permission and are added to the network configuration without explicit administrator action, contributing to network drift and configuration opacity.
• Network Diagrams are Hypotheses: The presentation strongly emphasizes that network diagrams, while useful, represent an hypothesis of the network's structure, not necessarily its reality. The actual traffic flows can diverge significantly due to Azure's implicit routing decisions, necessitating continuous verification.

Technical Deep Dive

▶ Watch: Overriding hidden route and encountering new blockage (6:30)

The core of the "Azure Networking Dark Arts" presentation lies in its detailed exploration of two distinct scenarios where Azure's implicit behaviors undermine explicit security configurations. These scenarios serve as powerful illustrations of how deeply ingrained platform defaults can create unexpected and potentially insecure network paths.

Scenario 1: Cosmos DB Service Endpoint Bypass

The first scenario involved a common hub-and-spoke architecture, where an Azure Virtual Desktop (AVD) spoke needed to access a Cosmos DB instance. The security objective was clear: block all public access to Cosmos DB and ensure that all legitimate traffic from the AVD spoke flowed through the central Azure Firewall in the hub VNet for inspection and auditing.

To achieve this, the team initially implemented Service Endpoints. A Service Endpoint allows specific Azure services (like Cosmos DB) to be accessed privately from a designated subnet within a VNet, leveraging Microsoft's backbone network, thereby bypassing the public internet. The configuration steps were:

1. Configure Service Endpoint on Cosmos DB: This restricts access to only specified VNets/subnets.
2. Add a User-Defined Route (UDR) to the AVD Spoke Subnet: A UDR was added to the spoke's route table, specifying that all outbound traffic (0.0.0.0/0) should have the Azure Firewall's internal IP address as its next hop. This was intended to force all traffic through the firewall.
3. Configure Service Endpoint Access for the AVD Spoke Subnet: The AVD spoke subnet was enabled for Cosmos DB Service Endpoint access.

Initial testing appeared successful: public access to Cosmos DB was blocked, and a Python script from a VM in the AVD spoke could successfully fetch data. However, a crucial discrepancy was uncovered using Azure Network Watcher's Next Hop feature. When checking the next hop for traffic from the AVD spoke VM to the Cosmos DB IP, the result was the Service Endpoint, not the Azure Firewall as expected.

The investigation revealed that when the AVD spoke subnet was configured for Cosmos DB Service Endpoint access, Azure silently injected a more specific route (Cosmos DB IP -> Service Endpoint) into the spoke's effective route table. Because of the longest prefix match rule, this injected route, being more specific than the 0.0.0.0/0 UDR pointing to the firewall, took precedence. Consequently, traffic to Cosmos DB was bypassing the firewall entirely.

To rectify this, a more specific UDR was added to the AVD spoke's route table: all traffic destined for the Cosmos DB Service Tag should have the Azure Firewall's IP as its next hop. This UDR, being specific to the Cosmos DB service but pointing to the firewall, now won the longest prefix match. Re-testing with Network Watcher confirmed the next hop was indeed the firewall.

However, a new problem emerged: the connection from the AVD spoke to Cosmos DB was now blocked. The reason was that while traffic was correctly routed to the firewall, there was no corresponding configuration allowing the firewall (or the hub VNet) to reach the Cosmos DB Service Endpoint. The final fixes involved:

1. Remove the old Virtual Network Rule from the Cosmos DB Service Endpoint for the AVD spoke.
2. Add a new Virtual Network Rule for the hub VNet (specifically, the firewall subnet) to the Cosmos DB Service Endpoint.
3. Configure the firewall subnet to have access to the Cosmos DB Service Endpoint.

This multi-step correction ultimately secured the end-to-end flow, ensuring all traffic passed through the firewall and public access remained blocked.

Scenario 2: Site-to-Site VPN Asymmetric Routing

The second scenario involved connecting an Azure environment to an on-premises SQL Server database via a Site-to-Site VPN. The goal was to allow VMs in Azure spokes to access the on-prem database, centralizing DNS resolution in Azure. A VPN Gateway was deployed in the hub VNet, which automatically creates a Gateway Subnet – an Azure-managed subnet where user-defined routes cannot be directly applied or inspected via effective routes.

Initial testing from a VM within the hub VNet successfully connected to the on-prem SQL Server, confirming the basic VPN connectivity. The challenge arose when trying to connect from a VM in an AVD spoke. Azure's Connection Troubleshoot tool reported a "full green valid connection" for the MSSQL port, indicating no issues with connectivity, NSGs, or route tables. Despite this, attempts to query the SQL Server from the spoke VM failed.

Troubleshooting involved ruling out various possibilities:

• VPN or on-prem packet drops were dismissed since the hub VM could connect.
• DNS resolution was verified from the AVD spoke and found to be working.

This left either gateway routing or the Azure Firewall as potential culprits. Since direct inspection of the Gateway Subnet's routes was not possible, the team resorted to a packet capture on the AVD spoke VM. The capture revealed a critical clue: the VM sent a SYN packet, received a SYN-ACK from the SQL Server, and then sent an ACK. However, subsequent data packets were ignored by the server, and eventually, the server re-sent a SYN-ACK.

This pattern strongly suggested asymmetric routing. The hypothesis was:

1. The initial SYN packet from the AVD spoke correctly traversed the hub, went through the Azure Firewall, and then via the VPN Gateway to the on-prem SQL Server.
2. The SYN-ACK response from the on-prem SQL Server, instead of returning through the firewall, was routed directly from the VPN Gateway to the AVD spoke.
3. Because the Azure Firewall is stateful, it never saw the SYN-ACK and therefore dropped the subsequent ACK (and any data packets) from the AVD spoke, as it considered them part of an unknown session.

The root cause was identified as a default VNet peering route that allowed direct communication between the Gateway Subnet and the spoke VNets, bypassing the firewall for return traffic. To fix this, a User-Defined Route (UDR) was added to the Gateway Subnet. This UDR specified that all traffic destined for the spoke VNets should have the Azure Firewall's IP as its next hop. This forced the return traffic through the firewall, ensuring symmetric routing. After this change, the connection from the AVD spoke to the on-prem SQL Server worked successfully. This scenario starkly demonstrated that Azure VPN Gateway and Azure Firewall, while both Azure tools, do not inherently work together without explicit routing configuration to prevent asymmetric flows.

Demo / Proof of Concept

▶ Watch: Final fix: configuring hub and firewall for service endpoint (7:45)

The speakers effectively demonstrated their findings through a series of practical tests and diagnostics within an Azure environment. While not a live, interactive demo in the traditional sense, the talk walked through the methodology used to uncover these hidden behaviors.

For the Cosmos DB Service Endpoint scenario, the proof of concept involved:

• Python Script: A simple Python script was used to attempt fetching data from the Cosmos DB. This script was executed from various locations: a local PC (to confirm public access was blocked), and a VM within the AVD spoke subnet (to test legitimate access).
• Azure Network Watcher's Next Hop: This crucial diagnostic tool was employed to inspect the actual next hop for traffic from the AVD spoke VM to the Cosmos DB IP. This directly revealed the unexpected routing to the Service Endpoint, bypassing the firewall.
• Verification of Effective Routes: The effective routes on the spoke VM were examined to confirm the presence of the Azure-injected route and to understand how the longest prefix match rule was causing the firewall bypass.

For the Site-to-Site VPN scenario, the demonstration included:

• Python Script: Again, a Python script was used to attempt a connection and query the on-premises SQL Server from a VM in the hub VNet (to confirm basic VPN functionality) and then from a VM in the AVD spoke.
• Azure Connection Troubleshoot: This tool was used to simulate connectivity from the spoke VM to the on-prem database. The key finding here was that despite the tool reporting a "full green valid connection," the actual application-layer connectivity failed, highlighting the tool's limitations.
• Packet Capture: A packet capture was performed on the AVD spoke VM. This was critical for diagnosing the asymmetric routing issue, as it allowed the speakers to observe the specific sequence of SYN, SYN-ACK, and ACK packets, revealing that the return SYN-ACK was not being seen by the firewall.

These practical demonstrations, although presented via screenshots and explanations, provided concrete evidence of the implicit routing issues and the methods required to diagnose and resolve them, going beyond what Azure's native tools initially revealed.

Defensive Implications

▶ Watch: Introduction to Site-to-Site VPN challenges (8:35)

The insights shared in "Azure Networking Dark Arts" carry profound defensive implications for any organization operating in Azure. The core message is a cautionary tale against implicit trust and highlights the necessity of proactive, deep-dive verification of network behaviors.

1. Never Trust Default Routing Implicity: Defenders must understand that Azure's default behaviors and implicit route injections can silently override explicit configurations. Simply deploying a firewall and UDRs is not enough; the actual traffic paths must be continuously validated. Assume that Azure will introduce routes you didn't define.
2. Verify Actual Paths, Not Just Connectivity: As demonstrated with Connection Troubleshoot, a "valid connection" check does not guarantee that traffic flows through the intended security controls. Defenders must move beyond basic reachability tests to verify the exact network path traffic takes, ensuring it adheres to security policies (e.g., passing through a firewall for inspection). Tools like Network Watcher's Next Hop are useful, but packet captures might be necessary for deeper analysis.
3. Implement a Rigorous Reverification Process: Every change to network configuration – deploying a new VNet, modifying an NSG, adding a route, or configuring a new service endpoint – has the potential to impact the entire network and introduce unintended routing. Organizations must establish a process for reverifying network flows and security posture after every change, not just as a one-time setup. This should be an integral part of the change management lifecycle.
4. Deep Dive into Documentation and Service Interactions: The "dark arts" are often "hidden in documentation." Security teams need to invest time in thoroughly understanding the nuances of how different Azure services interact, especially regarding networking. This includes understanding the specific routing implications of services like Service Endpoints, Private Endpoints, VPN Gateways, ExpressRoute, and Azure Firewall, and how their defaults might conflict.
5. Beware of Asymmetric Routing: The VPN Gateway scenario is a stark reminder that even well-intentioned combinations of Azure services can lead to complex issues like asymmetric routing. Defenders must be aware of this phenomenon, particularly when integrating on-premises networks or using stateful firewalls, and proactively configure UDRs to ensure symmetric traffic flows.
6. Network Diagrams are Just the Start: Treat network diagrams as a hypothesis, not a definitive truth. Regularly audit the effective network configuration and traffic flows to ensure they align with the documented design. Consider automated tools that can analyze effective routes and simulate traffic paths across complex Azure environments.
7. Prioritize Auditing and Logging: When traffic bypasses a firewall, it also bypasses its logging capabilities. Ensuring all traffic passes through a central point (like a firewall) is crucial for comprehensive auditing and incident response. This requires fixing the routing issues at their source.

By adopting a skeptical and verification-centric approach to Azure networking, defenders can illuminate these "dark arts" and build more resilient and genuinely secure cloud environments.

Key Takeaways

• Azure's "Dark Arts" are Real: Azure implicitly injects routes and has default behaviors that can silently alter network traffic paths, often bypassing explicit security controls like firewalls.
• Verify the Path, Not Just Connectivity: A successful connection does not guarantee that traffic followed the intended, secure route. Azure's diagnostic tools can be misleading; deep verification of the full network path is essential.
• Longest Prefix Match is Key: Understand how more specific, Azure-injected routes (e.g., for Service Endpoints) can override broader user-defined routes, leading to unexpected traffic flows.
• Asymmetric Routing is a Threat: Combining services like Azure VPN Gateway and Azure Firewall can lead to asymmetric routing, where inbound and outbound paths differ, causing stateful firewalls to drop legitimate traffic.
• Network Diagrams are Hypotheses: Treat your network diagrams as a starting point. The true network topology is dictated by Azure's effective routes and service interactions, which must be continuously validated.
• Reverification is Crucial: Implement a rigorous process to reverify network security and traffic paths after every change (VNet, NSG, route, service configuration) to prevent configuration drift and hidden vulnerabilities.

About the Speaker(s)

Achia Rosenfeld is a Software Engineer at Act Security. Her work involves building advanced tools for analyzing cross-cloud network and identity configurations.

Kobi Rubin is the Head of Research at Act Security. He leads the research efforts focused on identifying and detecting security gaps in complex cloud environments across various providers.

Act Security, the company both speakers represent, specializes in developing tools that provide deep insights into network and identity configurations across multiple cloud platforms, helping organizations identify and mitigate security vulnerabilities that often arise from the hidden complexities discussed in this presentation. Their expertise lies in unraveling the intricate and often opaque behaviors of cloud networking to ensure robust security postures.