Entente: Cross-silo Intrusion Detection on Network Log Graphs with Federated Learning

Entente: Cross-silo Intrusion Detection on Network Log Graphs with Federated Learning

Speakers: Jiacen Xu

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=JWY8CBjrQIY

Overview

Organizations operating across multiple regions face a fundamental tension: they need to collaborate on intrusion detection to catch cross-silo attacks, but regulations like GDPR prohibit sharing the network log data required for centralized training. This talk presents Entente, a federated learning framework specifically tailored for graph-based network intrusion detection systems (GRAINS) that enables cross-silo collaboration without data sharing. Named after the World War I diplomatic agreements where sovereign nations collaborated without merging their authority, Entente achieves 94-84% AP and over 99% AUC, outperforming all federated learning baselines and even beating some centralized training approaches.

Presented by Jiacen Xu, a security researcher at Microsoft, the framework introduces three key innovations: Barabasi-Albert reference graph synthesis for client similarity calculation, adaptive contribution scaling with dynamic norm-bounding for robust aggregation, and graph augmentation using one-hop cross-client information from firewall logs. The system was evaluated on three enterprise-scale datasets totaling over 1 billion events and demonstrated robustness against model poisoning attacks with bounded attack success rates below 10%.

Background

▶ Watch: The Entente concept: cross-silo collaboration without data sharing (0:00)

Graph-based network intrusion detection systems (GRAINS) represent network activity as graphs where nodes represent hosts and edges represent network flows. These systems train on labeled subgraphs to classify normal versus malicious activity, with real-world datasets capturing complex attack patterns like lateral movement across enterprise networks. However, current GRAINS assume centralized data -- all logs from all network segments available in one place.

Regulations like GDPR and internal data boundary policies make centralized training impossible for multi-region organizations. A straightforward application of federated learning (e.g., FedAvg) fails because of three compounding challenges: data heterogeneity (network topologies and attack patterns vary dramatically across silos), class imbalance (attacks are rare events, distributed unevenly), and robustness (attackers may poison the training process from compromised silos).

The researchers demonstrated the severity of these challenges with statistics from their datasets: under different silo configurations, standard deviations of node and edge counts across silos are approximately half the total count, indicating highly non-IID (non-identically distributed) data that defeats simple averaging approaches.

Key Findings

▶ Watch: System design: client-side graph augmentation and server-side aggregation (4:00)

Entente achieves effectiveness close to centralized GRAINS while maintaining privacy, scalability, and robustness:

Effectiveness: Entente achieves 94-84% AP and over 99% AUC across three datasets, outperforming all federated learning baselines (FedAvg, FedOpt, FedProx) and surprisingly beating some centralized training approaches. The researchers attribute this to Entente's superior handling of long sequences and data heterogeneity.

Scalability: Communication overhead is low -- for the E-ULeR GRAINS model, only 1.74 MB is shared between client and server per round. Training time scales reasonably with client count, and GPU memory cost remains stable since each client trains only a local model.

Robustness: Against model poisoning attacks where an attacker controls one silo and injects malicious edges (making malicious events appear benign) with scaled model updates, Entente bounds attack success rate to less than 10%. The adaptive contribution scaling and dynamic norm-bounding prevent attackers from dominating the global model through scaled updates.

A surprising finding is that federated learning can outperform centralized training in some scenarios, likely because the heterogeneous data distribution across silos provides a form of natural regularization that prevents overfitting.

Technical Deep Dive

▶ Watch: Adaptive contribution scaling and dynamic norm-bounding (8:00)

Entente's architecture operates across client (silo) and server components with three key innovations:

Graph Augmentation: Each silo generates temporal snapshots of its network graph, where nodes represent hosts and edges represent network flows. A critical design choice is the one-hop cross-client augmentation: since firewalls already log inbound and outbound flows, each silo can include one hop of cross-client connectivity without any additional privacy leakage. This information is already stored locally in firewall logs.

Reference Graph Synthesis and Similarity: Rather than directly comparing client graphs (computationally expensive), Entente generates Barabasi-Albert (BA) reference graphs -- scale-free graphs where most nodes have few connections while a small number of hubs have high degree. This topology closely resembles real network architectures with minimal prior knowledge. Client similarity is calculated using the Weisfeiler-Lehman subtree histogram with Jaccard similarity, which extracts multi-set local subtrees for fast comparison rather than expensive direct graph comparison.

Adaptive Contribution Scaling and Dynamic Norm-Bounding: Instead of simple averaging (FedAvg), Entente calculates client weights based on three factors: the Jaccard similarity from the reference graph comparison, cosine similarity of the model weights, and norm-bounded L2 distance. This equation weights clients with more similar network topologies more heavily while bounding update norms to prevent poisoning attacks from scaling their malicious updates. The theoretical guarantee is a bounded iteration-wise shift, ensuring convergence even under adversarial conditions.

Demo / Proof of Concept

▶ Watch: Evaluation: 94% AP, 99% AUC on enterprise-scale datasets (10:00)

The evaluation used three enterprise-scale datasets: OPTC (8 days, 92 million events), LOS (58 states, 1 billion events), and Pivoting (1 day, 74 million events). Two GRAINS models were tested: E-ULeR and JBL. Federated learning baselines included FedAvg, FedAvg-N (sampling N clients), FedOpt (adaptive optimizer), and FedProx (regularization-based).

The robustness evaluation simulated a scenario where one silo is controlled by an attacker who injects malicious edges and scales model updates by a factor gamma. Even in the unbounded scaling case, the system showed training divergence rather than successful poisoning, indicating the adaptive contribution scaling provides effective defense.

Defensive Implications

▶ Watch: Q&A: generalizability and federated learning beating centralized training (14:00)

Entente provides a practical path for organizations subject to data sovereignty regulations to deploy collaborative intrusion detection without violating compliance requirements. The framework is particularly relevant for:

• Global enterprises with network segments across different regulatory jurisdictions
• Government agencies that need to share threat intelligence across regions without moving data
• Industry verticals (financial services, healthcare) where cross-organization threat detection is valuable but data sharing is prohibited

The low communication overhead (1.74 MB per round) and stable GPU memory requirements make the system deployable on existing infrastructure without significant investment. The robustness guarantees against model poisoning address a critical concern for any federated deployment where some participants may be compromised.

Security teams should evaluate whether their current SIEM or NDR solutions could benefit from cross-silo training, particularly for detecting lateral movement and cross-network attacks that no single silo can observe completely. The one-hop augmentation using existing firewall logs is a pragmatic approach that requires no new data collection.

Key Takeaways

• Entente enables cross-silo intrusion detection with federated learning, achieving 94-84% AP and 99%+ AUC without sharing raw network data
• The framework outperforms all federated learning baselines and even beats some centralized training approaches in heterogeneous data scenarios
• Barabasi-Albert reference graphs provide an efficient proxy for calculating client similarity without expensive direct graph comparison
• Adaptive contribution scaling with dynamic norm-bounding limits model poisoning attack success to less than 10%
• Communication overhead is minimal (1.74 MB per round for E-ULeR), making deployment practical on existing infrastructure
• One-hop cross-client augmentation using firewall logs improves detection without additional privacy leakage

About the Speaker(s)

Jiacen Xu is a security researcher at Microsoft. The speaker demonstrated deep familiarity with both the federated learning and graph-based intrusion detection domains, engaging substantively with audience questions about generalizability to other anomaly detection tasks and the performance implications of small or homogeneous client graphs. Xu acknowledged that the system is primarily designed for organization-level network topologies and that adaptation to very small networks would require additional research.