Mastering Efficiency in Argo CD: Scaling Smarter, Not Costlier - Alexander Matyushentsev, Akuity

Mastering Efficiency in Argo CD: Scaling Smarter, Not Costlier

Speakers: Alexander Matyushentsev, Co-creator of Argo Project & Co-founder of Akuity

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=bpsclYlGl2s

Overview

This talk, "Mastering Efficiency in Argo CD: Scaling Smarter, Not Costlier," delivered by Alexander Matyushentsev, a co-creator of the Argo project and co-founder of Akuity, delves into the often-overlooked financial implications of running Argo CD at enterprise scale. While Argo CD is renowned for its efficiency and cost-effectiveness in small deployments, this presentation highlights how its operational costs can escalate significantly when managing thousands of applications across numerous Kubernetes clusters. Matyushentsev shares critical insights and practical optimizations derived from Akuity's extensive experience running Argo CD for various organizations, aiming to equip attendees with strategies to reduce their cloud bills.

The core message of the talk is a proactive call to action for organizations leveraging Argo CD for large-scale GitOps deployments. Matyushentsev argues that at a certain threshold – specifically, when managing thousands of applications – the cost of the Argo CD control plane can become comparable to an engineer's salary, making optimization a financially sound investment. The presentation meticulously breaks down where these costs originate, particularly focusing on hidden expenses like network traffic and inefficient compute resource utilization, and then offers concrete, actionable solutions that have been contributed back to the open-source project or require specific configuration.

This discussion is crucial for platform engineers, DevOps teams, and cloud architects who are either currently operating large Argo CD environments or planning to scale their GitOps adoption. By providing real-world cost metrics and detailing specific technical adjustments, Matyushentsev empowers organizations to move beyond the initial "free" phase of Argo CD and strategically manage its operational footprint as their infrastructure grows. The insights shared are not merely theoretical but are born from the practical challenges and successes of running Argo CD in demanding production environments, making them highly relevant and impactful for anyone seeking to optimize their cloud spend and operational efficiency.

Background

▶ Watch: Introduction: Scaling Argo CD efficiency and cost (0:00)

Argo CD's reputation for being lightweight and efficient is well-deserved, particularly for initial deployments. A fresh installation on a small cluster typically consumes minimal CPU and memory – often less than 100 MB of RAM for all components combined. In such scenarios, Argo CD pods usually land on existing nodes with spare capacity, making its resource consumption virtually "free" as it leverages existing infrastructure. The pressure it applies to the Kubernetes API server is negligible for small setups, making it an ideal choice for getting started with GitOps.

However, this cost-free paradigm shifts dramatically when organizations begin to leverage one of Argo CD's most powerful features: multicluster management. As the scope expands to manage numerous clusters and thousands of applications, a centralized control plane becomes necessary. This control plane, which serves as the single source of truth for deployments, must be highly available (HA) to ensure continuous operations and survive upgrades without downtime. Achieving HA in Kubernetes environments necessitates running multiple replicas of every critical component. For Argo CD, this means transitioning from a single-pod setup for core services like the API server and repo server to multiple replicas. A significant change is seen with Redis, which typically moves from a single pod to a five-pod setup (one stateful set with HA proxy and two slaves for replication) in HA mode.

This transition to an HA, multicluster control plane introduces several non-obvious costs that contribute significantly to the cloud bill:

1. Dedicated Instances: While multiple instances are needed for HA, it's not enough to simply have beefy nodes. At least three instances are typically required to satisfy pod anti-affinity rules and distribute components across failure domains.
2. Dedicated Control Plane Cluster: Reusing a workload cluster to host Argo CD is highly discouraged. Workload spikes on application clusters can overload the API server, impacting Argo CD's stability, and vice-versa. A dedicated control plane cluster, often a managed Kubernetes service from a cloud provider, adds a base cost of several hundred dollars per month, even before Argo CD workloads are factored in.
3. Network Traffic: This is often the most significant and hidden cost. Cloud providers like AWS often categorize these costs under generic sections like "EC2 other," making them difficult to track. Yet, network traffic can account for up to 30% of the total Argo CD bill, especially cross-availability zone (cross-AZ) and Internet egress traffic.

To quantify these costs, Akuity conducted an experiment on GCP. They set up a control plane cluster with three nodes, sized to accommodate a large Argo CD instance, and deployed 800 applications across three managed clusters. The cost of the managed workload clusters was excluded from the calculation. The results were stark: this setup cost approximately $40 per day, translating to an annual cost of around $11,000. Crucially, 30% of this cost was attributed to network traffic, with a significant portion being cross-AZ and Internet egress traffic due to workload clusters connecting over the Internet.

Extrapolating these findings, Matyushentsev presented a "naive but accurate" cost model: each Argo CD application costs approximately $1.49 per year. This simple metric reveals the scale of the problem:

• Managing 3,000 applications could cost around $50,000 annually for the control plane.
• Managing 6,000 applications (often translating to close to 100 clusters) pushes the annual cost into six-figure territory, making it comparable to an engineer's salary.

This background establishes that while Argo CD is initially cheap, scaling it requires a strategic approach to cost management, focusing on optimizing compute, network, and control plane infrastructure to avoid spiraling expenses.

Key Findings

▶ Watch: HA mode and multi-cluster management increases costs (3:30)

The talk highlights several critical findings regarding Argo CD's cost implications at scale, derived from real-world operational experience:

1. Scaling Argo CD Becomes Costly: While initially efficient, Argo CD's operational costs can escalate significantly when managing thousands of applications (e.g., 6,000 applications can lead to six-figure annual costs for the control plane). This makes optimization a financial imperative at enterprise scale.

1. Network Traffic is a Major Hidden Expense: Network egress and cross-availability zone traffic often account for a substantial portion of the cloud bill, sometimes as high as 30%. This cost is frequently obscured in cloud provider billing and can spike dramatically under certain conditions.

1. Gigantic Application CRDs Drive Network Costs: Storing excessively large application CRDs (approaching the 1MB Kubernetes limit) due to practices like inlining Helm values or maintaining extensive history, leads to "video streaming-like" traffic. Each patch request results in a full resource JSON reply, generating 2MB of traffic per reconciliation for a single application.

1. Numerous Managed Resources Increase Traffic Exponentially: Applications managing a large number of Kubernetes resources (e.g., a deployment with hundreds of pods) cause exponential growth in Redis traffic. This is because Argo CD serializes and updates a JSON list of all resources in Redis, and frequent changes to these numerous resources result in larger, more frequent updates.

1. Inefficient Redis Defaults Impact Compute: The default repl-backlog-size setting in Redis (500MB) is overly generous for Argo CD's use case as a "throwaway cache." This default leads to unnecessarily high memory requests for Redis replicas, significantly increasing compute costs.

1. Repo Server Concurrency Issues Waste Compute: The repo server, responsible for manifest generation (Helm templating, Kustomize builds), is CPU and memory intensive. By default, it processes requests sequentially for applications within the same Git repository, forcing organizations to provision many replicas to maintain responsiveness, leading to wasted compute resources.

1. Dedicated Control Plane Clusters Can Be Over-Provisioned: Provisioning a full, production-ready Kubernetes cluster for each Argo CD instance can be wasteful, as Argo CD primarily uses the API server for CRD storage and access to a few secrets and config maps. Many advanced Kubernetes features go underutilized, leading to resource inefficiency.

These findings collectively underscore that while Argo CD provides powerful GitOps capabilities, its default configurations and common usage patterns can lead to substantial, often hidden, costs at scale. Addressing these inefficiencies through targeted optimizations is key to achieving a truly cost-effective GitOps strategy.

Technical Deep Dive

▶ Watch: Hidden costs: Dedicated clusters, network traffic (5:30)

Alexander Matyushentsev presented a comprehensive set of technical optimizations across networking, compute, and control plane infrastructure, all aimed at reducing the operational cost of Argo CD at scale. Many of these insights have been integrated into the open-source project or are configurable via specific settings.

Networking Cost Optimizations

1. Reducing Application CRD Size:

• Problem: Customers were creating gigantic application CRDs (close to 1MB, the Kubernetes limit) by inlining large Helm values.yaml files directly into the Application spec. This was exacerbated by Argo CD's rollback feature, which stores copies of value files in the application's history. When the application controller patches an Application CRD (e.g., to update the last reconciliation timestamp, which is only a few bytes), the Kubernetes API server responds with the entire resource JSON. For a 1MB CRD, this means 2MB of traffic per patch request, leading to "video streaming-like" traffic (gigabytes per minute) across thousands of applications, costing hundreds of dollars per day.
• Solution 1: User Behavior & historyLimit:
• Monitor: Watch and monitor the size of Application resources in the control plane cluster. If they approach 1MB, it's a red flag for cross-AZ traffic costs.
• Educate: Convince users to move inlined Helm values into separate .yaml files in the Git repository, leveraging Argo CD's multiple sources feature.
• Configure spec.historyLimit: Reduce the number of historical application versions stored. For example, setting spec.historyLimit: 2 significantly shrinks the CRD size, especially if users rely on Git for rollback rather than Argo CD's internal history.
• Solution 2: controller.resourceHealth.persist Setting:
• Configuration: Set controller.resourceHealth.persist: true in the argocd-cmd-params-cm ConfigMap.
• Effect: This instructs the application controller to stop storing health-related metadata directly in the Application CRD. This metadata is duplicated in Redis, and unless a very specific custom tool relies on the CRD metadata without using the Argo CD API server, disabling its persistence in the CRD is safe. This dramatically reduces the number of patches the controller sends to the Kubernetes API server, directly cutting down traffic.

1. Optimizing Resource Sharding for Large Applications:

• Problem: Applications managing a large number of Kubernetes resources (e.g., a deployment with hundreds of pods, resulting in 100+ distinct resources visible in the UI) cause exponential traffic growth. Argo CD stores a JSON-serialized list of all these resources in a single Redis key. When many pods frequently change state, the size of this JSON object increases, and updates become more frequent, leading to "video streaming-like" situations for applications with thousands of resources.
• Solution: ARGOCD_CONTROLLER_RECONCILE_RESOURCE_SHARD_COUNT:
• Configuration: Set the environment variable ARGOCD_CONTROLLER_RECONCILE_RESOURCE_SHARD_COUNT=50 for the application controller.
• Effect: This instructs the controller to split information about application resources into smaller chunks (shards) rather than storing them all in a single Redis key. The value 50 was found to be optimal, balancing CPU overhead and significant traffic reduction (up to 10x in extreme cases). This is a relatively new setting without a good default yet.

1. Excluding Chatty Resources from Watches:

• Problem: The Argo CD application controller watches managed Kubernetes clusters, attempting to catch every event that could affect application state. Some resources, like endpoints and endpoint slices, change very frequently but are often not directly managed by Argo CD or are not critical for its reconciliation logic. This generates excessive internet traffic (10x more expensive than internal traffic) from the controller to the managed clusters.
• Solution: resource.exclusions:
• Configuration: Use the resource.exclusions setting in Argo CD's configuration to tell it not to watch endpoints and endpoint slices.
• Example:

• Effect: Unless there's a specific requirement to manage these resources manually via Argo CD, excluding them can save tens of dollars per day by significantly reducing unnecessary traffic.

Compute Cost Optimizations

1. Tuning Redis Replication Backlog Size:

• Problem: In HA mode, Argo CD uses Redis with a master and two slaves replicating changes. The default Redis setting repl-backlog-size is 500MB. While suitable for many applications, Argo CD uses Redis primarily as a "throwaway cache" for UI visualization. A few seconds of replication lag is acceptable. The large default backlog size forces high memory requests for Redis pods, leading to over-provisioned nodes.
• Solution: repl-backlog-size Modification:
• Configuration: Change repl-backlog-size in the Redis configuration to 64mb.
• Effect: This seemingly small change can lead to dramatic savings. In Akuity's experience, it reduced the number of nodes from 150 to 80 for 100 Argo CD instances, saving $200-300 per day. This is a safe change as inconsistent replication has minimal impact on Argo CD's functionality.

1. Enabling Concurrent Manifest Generation in Repo Server:

• Problem: The repo server is a CPU-heavy component, responsible for running helm template and kustomize build commands. Historically, to avoid side effects, the repo server processes requests for applications within the same Git repository sequentially. If a single commit affects manifests for multiple applications in that repo, Argo CD processes them one by one, leading to slow reconciliation and forcing users to run many repo server replicas to maintain responsiveness, wasting CPU and memory.
• Solution 1: ARGOCD_HELM_ALLOW_CONCURRENCY:
• Configuration: Set the environment variable ARGOCD_HELM_ALLOW_CONCURRENCY=true for the repo server. (This will be the default in Argo CD 3.0).
• Effect: This enables concurrent processing of Helm chart generations, allowing the repo server to handle multiple applications from the same repository in parallel, thus reducing the need for numerous replicas.
• Solution 2: Avoid Kustomize Image Overrides:
• Problem: When Kustomize image overrides are specified directly in the Application spec, Argo CD has to modify files in a local clone of the Git repository, which necessitates sequential manifest generation.
• Recommendation: Avoid this pattern for better performance and lower costs.
• Solution 3: repo.server.parallel.limits:
• Configuration: Use the repo.server.parallel.limits flag (or environment variable) to control the maximum number of parallel manifest generations.
• Rule of Thumb: Set the limit to the smaller of (CPU cores allocated to repo server, memory in GB allocated to repo server). For example, if a repo server pod has 4 CPU cores and 8GB memory, set the limit to 4. This reserves one core and one gigabyte for each helm or kustomize invocation.

Control Plane Cost Optimizations

1. Consolidating Argo CD Instances with K3s:

• Problem: Provisioning a dedicated, full-fledged Kubernetes cluster for each Argo CD instance is often wasteful. Argo CD primarily uses the Kubernetes API server for storing Application CRDs and accessing a few secrets and config maps. Many features of a production-ready Kubernetes cluster go unused, leading to significant resource waste.
• Solution: Single Host Cluster with Multiple K3s Instances:
• Architecture: Run a single, larger Kubernetes cluster (the "host cluster"). On this host cluster, deploy multiple lightweight Kubernetes distributions like K3s. Each K3s instance then hosts a separate Argo CD control plane. Argo CD uses the K3s API server to store its metadata.
• Effect: This allows safe isolation of multiple Argo CD instances (potentially with different versions) on a shared, more efficiently utilized host cluster. This strategy can save hundreds of dollars per Argo CD instance by reducing the overhead of dedicated full Kubernetes clusters.

These detailed technical adjustments demonstrate that significant cost savings in Argo CD can be achieved by understanding its resource consumption patterns and applying targeted optimizations at various layers of the infrastructure.

Demo / Proof of Concept

▶ Watch: Real-world experiment: $40/day for 800 applications (7:50)

The talk did not feature a live demonstration or a real-time proof of concept of the optimizations discussed. Instead, Alexander Matyushentsev presented the findings and solutions based on real-world experience and data collected from Akuity's operations.

A key part of the talk involved a cost experiment designed to quantify the expenses of running Argo CD at scale. This experiment used a GCP cluster as a control plane with three nodes, installed Argo CD, and managed 800 applications across three external managed clusters. The results, including the daily cost of $40 and the breakdown of network traffic (30% of the bill), served as the empirical basis for the problem statement and the justification for the subsequent optimizations. While not a live demo, this experiment acted as a crucial proof of concept to validate the escalating costs of Argo CD in large environments. The speaker also referenced internal Akutiy data, such as the reduction from 150 nodes to 80 after implementing Redis optimization, as evidence of the effectiveness of their solutions.

Defensive Implications

▶ Watch: Extrapolating costs: $1.49 per Argo CD application (8:20)

The insights presented in this talk offer crucial defensive implications for organizations running or planning to scale Argo CD. Defenders, platform teams, and SREs should proactively implement these strategies to prevent unexpected cost overruns and maintain a robust, efficient GitOps platform.

1. Proactive Cost Monitoring and Alerting: Implement detailed monitoring for Argo CD's resource consumption, particularly network egress and cross-AZ traffic, as well as CPU and memory usage of core components (repo server, application controller, Redis). Set up alerts for unusual spikes or trends that could indicate inefficiencies or unexpected costs. Cloud provider cost allocation tags should be leveraged to accurately attribute expenses to the Argo CD control plane.

1. Enforce Application Spec Best Practices: Educate developers and GitOps users on best practices for defining Application resources. Specifically, discourage the inlining of large Helm values.yaml files directly into the Application spec. Instead, promote storing these values in Git repositories and using Argo CD's multiple sources feature. This prevents the creation of gigantic CRDs that inflate network costs.

1. Configure spec.historyLimit Judiciously: Review and adjust the spec.historyLimit for Application resources. If Git is the primary source of truth for rollbacks, a low history limit (e.g., 2) is often sufficient and significantly reduces CRD size, thus mitigating network traffic.

1. Adopt Key Configuration Optimizations:

• Networking: Immediately enable controller.resourceHealth.persist: true in argocd-cmd-params-cm to reduce API server pressure. Consider setting ARGOCD_CONTROLLER_RECONCILE_RESOURCE_SHARD_COUNT=50 for the application controller, especially if managing applications with many Kubernetes resources.
• Traffic Reduction: Implement resource.exclusions to ignore chatty resources like endpoints and endpoint slices unless there's a specific, justified need to manage them with Argo CD.
• Compute: Tune the Redis repl-backlog-size to 64mb to optimize memory usage. Enable ARGOCD_HELM_ALLOW_CONCURRENCY=true for the repo server and configure repo.server.parallel.limits based on allocated CPU/memory to enhance manifest generation efficiency.

1. Strategic Control Plane Architecture: For organizations running multiple Argo CD instances, evaluate moving from dedicated full Kubernetes clusters to a consolidated architecture using a single host cluster running multiple lightweight Kubernetes distributions like K3s. This can lead to substantial savings in control plane infrastructure costs.

1. Regularly Update Argo CD: The speaker noted that many optimizations are incorporated into newer Argo CD versions. Regularly updating to the latest stable release ensures access to these performance improvements and cost-saving features.

By implementing these defensive measures, organizations can ensure their Argo CD deployments remain efficient and cost-effective, even as they scale to manage increasingly complex and extensive infrastructure.

Key Takeaways

• Argo CD's Cost Escalates at Scale: While initially "free," Argo CD becomes a significant financial investment (potentially costing hundreds of thousands of dollars annually) when managing thousands of applications and hundreds of clusters. Proactive optimization is essential.
• Network Traffic is a Hidden Cost Driver: Cross-availability zone and Internet egress traffic can account for up to 30% of the total Argo CD bill. Monitoring and optimizing this traffic is crucial for cost control.
• Application CRD Size Matters: Overly large Application CRDs (e.g., due to inlined Helm values or extensive history) lead to excessive network traffic. Reducing spec.historyLimit and enabling controller.resourceHealth.persist: true are effective mitigations.
• Optimize Redis and Repo Server for Compute: Default Redis repl-backlog-size is often too high for Argo CD's cache use case, wasting memory. The repo server's sequential processing of manifests for multiple applications in a single Git repo can waste CPU. Tuning Redis to 64mb and enabling repo server concurrency (ARGOCD_HELM_ALLOW_CONCURRENCY=true, repo.server.parallel.limits) yield significant compute savings.
• Consolidate Control Planes with K3s: Provisioning a full Kubernetes cluster for each Argo CD instance is often inefficient. Running multiple Argo CD instances on lightweight K3s clusters hosted within a single, larger Kubernetes cluster can save hundreds of dollars per instance.
• Leverage Latest Features and Configurations: Many optimizations are available through specific environment variables or configuration flags, some of which are new or not enabled by default. Staying updated with Argo CD versions and actively configuring these settings is key to efficiency.

About the Speaker(s)

Alexander Matyushentsev is a prominent figure in the Kubernetes and GitOps communities, recognized as one of the co-creators of the Argo project. He has dedicated seven to eight years to the development and evolution of Argo CD, making substantial contributions to its architecture and feature set. Currently, Alexander is also a co-founder of Akuity, a company that specializes in running and managing Argo CD at scale for other organizations. His insights and recommendations in this talk are directly informed by Akuity's practical experience in operating large, production-grade Argo CD environments and addressing the real-world challenges of cost efficiency and scalability.