Building Cursor: AI-Native Development Tools

Building Cursor: AI-Native Development Tools

Speakers: Sualeh Asif (CTO & Co-Founder, Cursor)

Conference: CS153 Infra @ Scale 2025

YouTube: https://www.youtube.com/watch?v=4jDQi9P9UIw

Overview

In this insightful talk at CS153 Infra @ Scale 2025, Sualeh Asif, CTO and Co-Founder of Cursor, provided an unfiltered, deep dive into the formidable infrastructure powering Cursor, an AI-native development tool that has scaled by a factor of 100 or more within the last year. The discussion illuminated the intricate challenges and innovative solutions involved in building and maintaining a global-scale AI coding platform, from managing massive inference workloads to navigating the complexities of distributed databases and third-party model providers.

Asif's presentation was a candid exploration of the high-stakes world of modern infrastructure, where constant scaling, unexpected failures, and the relentless pursuit of reliability are the norm. He detailed the architectural decisions, operational nightmares, and heroic incident responses that have shaped Cursor's journey, offering invaluable lessons for anyone operating at the bleeding edge of AI and cloud computing. The talk underscored that while AI promises to automate many aspects of software development, the underlying infrastructure demands human ingenuity, robust engineering, and a pragmatic approach to problem-solving.

The significance of this talk lies in its transparency regarding the real-world operational challenges of deploying AI at scale, particularly in a developer-facing product. It highlights the often-unseen complexities behind seemingly seamless AI experiences, revealing the constant battle against entropy, the critical role of database choices, and the delicate dance of managing dependencies on rapidly evolving frontier models. For engineers and architects, it serves as a crucial case study in building resilient, high-performance systems in a dynamic and unpredictable technological landscape.

Background

▶ Watch: Introduction and Cursor's massive operational scale (0:00)

Cursor emerged into a rapidly evolving landscape of AI-assisted coding, where tools like GitHub Copilot had already established a baseline for autocomplete functionality. However, Asif and his co-founders recognized a much higher ceiling for AI's potential in software engineering—one extending far beyond simple autocompletion to automating significant portions of the development workflow. This vision necessitated a robust and highly scalable infrastructure capable of processing vast amounts of code, performing complex inference operations, and providing a seamless, low-latency user experience globally.

The fundamental problem Cursor addresses is enhancing developer productivity through intelligent code assistance. This involves understanding a user's entire codebase, generating contextually relevant suggestions, and even performing complex code transformations. Achieving this requires a sophisticated backend that can handle diverse, often proprietary, codebases while maintaining speed and reliability. Prior work in this space, primarily Copilot, focused on autocompletion, but Cursor aimed for a more agentic approach, deeply integrating AI into the editor experience to the point where it feels "like all your data is sort of copy paste" rather than a model call.

Architecturally, Cursor's infrastructure is built around three core pillars:

1. Indexing Systems: These systems are responsible for understanding and processing user codebases. Beyond simple retrieval, Cursor invests heavily in indexing Git repositories and other source control mechanisms to build a deep understanding of the project structure and history. This involves sophisticated data structures like Merkle Trees to efficiently track changes across vast codebases.
2. Model Section: This encompasses the various AI models powering Cursor's features. The most demanding is the autocomplete model, which runs on every keystroke, generating tens of thousands of model calls per second. This necessitates a massive, globally distributed GPU fleet (e.g., 2,000 H100s or 1,000 H100s) to ensure low latency. Cursor also utilizes an "apply model" for more complex code transformations and relies on Frontier Models from third-party providers for other tasks.
3. Product Experience & Streaming Infrastructure: This pillar focuses on the user-facing aspects and the backend systems that continuously improve Cursor. It involves "clever tips and tricks" to make the AI feel fast and native, as well as a robust streaming infrastructure for storing and processing incoming data to refine models and user experience over time. This data is not directly queried by users but is crucial for continuous product improvement.

A significant challenge in this domain is the cold start problem inherent in inference services. When a service crashes or scales up, new nodes can be overwhelmed by incoming requests before they become fully healthy, leading to cascading failures. This issue, coupled with the inherent unreliability of even mature model providers, creates a high-stakes environment where infrastructure decisions and incident response are paramount.

Key Findings

▶ Watch: Autocomplete model: 20,000 calls/sec, global GPUs (2:40)

Cursor's journey at scale has yielded several critical insights into building and operating AI-native development tools:

1. Extreme Scale of AI Inference: Cursor operates at an astonishing scale, performing approximately 100 million model calls per day on its custom models, excluding traffic to frontier models. The autocomplete feature alone triggers 20,000 model calls per second at peak, supported by a global fleet of 1,000-2,000 H100 GPUs distributed across multiple regions (Phoenix, Virginia, London, Tokyo). Their indexing systems process over a billion documents daily and hundreds of billions over the company's lifetime, underscoring the sheer data volume involved.
2. Hybrid Inference Strategy: Cursor employs a dual approach to inference: self-hosting custom, performance-critical models (like autocomplete) on its own GPU infrastructure, and leveraging Frontier Model providers (e.g., Anthropic, OpenAI, Google Cloud) for more generalized or compute-intensive tasks. This hybrid strategy allows for optimized cost, performance, and reliability, though managing rate limits and reliability with third-party providers remains a significant challenge.
3. Database Simplicity over Complexity: A major lesson learned was the importance of choosing simple, proven database solutions over complex, globally distributed ones, especially in the early stages. The initial attempt to use YugabyteDB (a Spanner-descendant) for indexing failed due to issues with long transactions and consensus protocols. Migrating to a single-node PostgreSQL instance on AWS RDS proved vastly more stable and performant, despite PostgreSQL's less "scalable" reputation. The adage "don't choose a complicated database, go with a hyperscaler" became a guiding principle.
4. The Perils of Database Internals (PostgreSQL specifically): Even with PostgreSQL, unique challenges arose. A critical incident was caused by the misunderstanding of PostgreSQL's UPDATE operation, which internally functions as a DELETE followed by an ADD. In a write-heavy workload like code indexing, this creates an enormous number of "dead rows" that consume disk space and require aggressive vacuuming. When the vacuum process couldn't keep up, the database swelled to 22 terabytes, grinding to a halt and highlighting the need for deep understanding of chosen database's internal mechanics.
5. Distributed Systems Race Conditions and Hidden Errors: The indexing system experienced a "gnarly week" due to a confluence of issues: a bug preventing large files from being cached in DynamoDB, leading to repeated embedding calls, and a race condition in the queuing system responsible for updating Merkle Tree hashes. This created a vicious cycle where large files were never committed, continually re-triggered processing, and exhausted resources, all while global error rates appeared normal. This demonstrates how subtle interactions in distributed systems can lead to catastrophic, hard-to-diagnose failures.
6. Incident Response and Refactoring Under Duress: Cursor's most severe incidents highlight the critical role of highly skilled engineers and the necessity of radical solutions under pressure. In the 22TB PostgreSQL outage, the team had to perform an emergency, full-stack refactor and migration to object storage (S3/R2-like) for the largest tables, effectively "killing the database" for that workload. This "full-on process" involved assigning multiple engineers to tackle different aspects simultaneously, with one co-founder successfully rewriting a core workload to object storage faster than others could resuscitate the existing system.
7. Security in AI-Native Tools: Given the sensitive nature of user code, Cursor places a high emphasis on security. They employ encryption techniques for vector embeddings, storing them with keys that reside on the user's device. This ensures that even if the vector database is compromised, the embeddings cannot be reverse-engineered into readable code without the client-side key, adding a crucial layer of protection.

Technical Deep Dive

▶ Watch: Three pillars: Indexing, Models, and 'Apply Model' tricks (4:00)

Cursor's infrastructure is a testament to sophisticated engineering designed to handle immense scale and complexity while maintaining a seamless user experience.

Indexing System: The Merkle Tree Approach

The core of Cursor's code understanding mechanism is its indexing system, which leverages Merkle Trees for efficient change detection and synchronization.

1. Client-Server Reconciliation: When a user opens Cursor, a Merkle Tree is computed on the client-side for their codebase. A corresponding Merkle Tree exists on Cursor's servers.
2. Hashing Files and Folders: Every file is hashed. Each folder's hash is then computed as the hash of its children (files and subfolders), propagating up to a single root hash for the entire repository.
3. Efficient Change Detection: To identify changes (e.g., after a git checkout or system restart), Cursor compares the client's root hash with the server's root hash.

• If the root hashes differ, it means something has changed within the codebase.
• The system then descends the tree, comparing hashes of child folders. If a child folder's hash differs, it descends further into that folder.
• This process continues until the specific files that have changed are identified. This bidirectional communication minimizes data transfer and speeds up synchronization.

Database Evolution and Challenges

Cursor's database strategy underwent significant evolution, driven by scaling challenges:

1. Initial YugabyteDB Attempt:

• Choice: Inspired by Google's globally distributed Spanner, Cursor initially opted for YugabyteDB, a PostgreSQL-compatible distributed SQL database known for its scalability.
• Failure: Despite its theoretical advantages, YugabyteDB proved problematic for Cursor's specific workload, which involved a large number of long transactions. In a globally distributed setup, these transactions required complex consensus protocols, leading to performance bottlenecks and instability. Asif noted, "We cannot get Yugabyte to run," despite significant investment.
• Lesson: The critical takeaway was to prioritize simplicity and proven solutions, especially when dealing with complex, high-throughput workloads.

1. Migration to AWS RDS (PostgreSQL):

• Solution: The workload was migrated to a standard PostgreSQL instance on AWS RDS. This immediately resolved the performance issues, proving that a simpler, single-node database could handle the load more effectively than a distributed one if chosen appropriately.
• New Challenges – The "Hot Database": Even PostgreSQL presented its own set of challenges.
• UPDATE as DELETE+ADD: A fundamental difference from MySQL, PostgreSQL's UPDATE operation is implemented as a DELETE followed by an ADD. This means that every time a developer types (triggering frequent file updates), new records are created, and old ones are marked as "dead rows" (tombstones).
• Vacuuming Issues: These dead rows are not immediately reclaimed; a background process called VACUUM is responsible for cleaning them up. If the write workload is too high, or the vacuum process can't keep up, the database size can explode.
• Transaction ID Wraparound: The system also faced issues with transaction IDs and anti-transaction IDs, which, if mismanaged, can lead to database instability or even prevent it from booting.
• 22TB Outage: This culminated in a severe incident where the main indexing database swelled to 22 terabytes (approaching the 64TB RDS limit), causing it to grind to a halt. AWS support could not provide a solution.

Caching, Queuing, and Race Conditions

Another critical incident highlighted the intricate interdependencies and potential pitfalls in distributed systems:

1. DynamoDB Cache Failure: Cursor uses DynamoDB as a cache. A "gnarly bug" in the code meant that files exceeding a certain size were not being cached.
2. Cascading Load: When a large file was not cached, it would bypass the cache and trigger a call to the embedding model. This increased the load on the embedding model.
3. Queuing Race Condition: After chunking and embedding, the system needed to update the Merkle Tree hashes via a queue. A "gnarly race condition" in this queue, particularly affecting large files, prevented these updates from committing successfully.
4. Vicious Cycle: Because the updates weren't committed, the client would repeatedly detect that the large files hadn't been indexed, leading to repeated attempts, cache misses, embedding model calls, and re-triggering the race condition in the queue. This created an uncontrolled feedback loop, where "load just keeps going and going and going," despite normal-looking global error rates. The problem was specific to "load-bearing files" that were disproportionately large.

Inference Infrastructure and Provider Management

Managing AI inference at Cursor's scale involves a complex interplay of self-hosted and third-party resources:

1. Global GPU Fleet: For critical, low-latency tasks like autocomplete, Cursor operates its own geographically distributed GPU infrastructure, utilizing H100s across regions like Phoenix, Virginia, London, and Tokyo to minimize latency for users worldwide.
2. Frontier Model Provider Challenges:

• Reliability: Third-party model providers historically suffer from "terrible reliability." Asif recounted instances where providers would crash at 30-40 million tokens per minute due to unoptimized caching or other issues.
• Cold Start Problem: Providers also face the cold start problem, where scaling up new instances can lead to them being immediately overwhelmed, preventing full recovery.
• Rate Limits and Negotiations: Cursor is often one of the largest users of frontier models, leading to constant "live negotiations" with providers for higher token limits. This mirrors early cloud provisioning challenges but with the added volatility of rapidly scaling AI workloads.
• Multi-Cloud/Multi-Provider Strategy: To mitigate these risks, Cursor employs a multi-cloud and multi-provider strategy, balancing user traffic across different providers (e.g., AWS, Google Cloud, Anthropic) and even different regions to ensure resilience and access to sufficient inference capacity.

1. Traffic Management and Prioritization: During incidents, Cursor implements various strategies to manage traffic, such as prioritizing certain users, applying global traffic kills, or using prefix-based prioritization (similar to WhatsApp's approach) to bring up critical services first.

Security Architecture

Security for user code is paramount:

1. Vector Embedding Encryption: Cursor encrypts all vector embeddings of user code. Crucially, the encryption key resides on the user's device.
2. Defense in Depth: This means that even if Cursor's vector database were compromised (an unlikely event given cloud provider security), the stored embeddings would be unreadable without the client-side key. This provides a strong "defense in depth," acknowledging that while it's "99.99% sure there's no way to go from a vector to code," adding an encryption layer provides peace of mind for sensitive proprietary data.

Demo / Proof of Concept

▶ Watch: Architectural flow: user interaction to inference (5:00)

This talk was a technical deep dive into Cursor's backend infrastructure and operational challenges, rather than a product demonstration. While Cursor itself is a developer tool, the presentation focused on the architectural decisions, database migrations, and incident response strategies that enable its functionality at scale, rather than showcasing its user-facing features.

Defensive Implications

▶ Watch: Compartmentalization for blast radius control and stability (5:30)

The insights from Cursor's scaling journey offer critical lessons for developers, infrastructure engineers, and organizations building AI-native products:

1. Pragmatic Database Selection: Prioritize simplicity and proven reliability in database choices, especially when scaling rapidly. Complex, globally distributed databases like YugabyteDB might offer theoretical advantages but can introduce significant operational overhead and unexpected failure modes. For many workloads, a well-managed relational database like PostgreSQL on a hyperscaler's managed service (e.g., AWS RDS) can be more robust and easier to operate. "Don't choose a complicated database, go with a hyperscaler's... use Postgres, don't do anything complicated."
2. Deep Understanding of Database Internals: Don't just use a database; understand how it works under the hood. PostgreSQL's UPDATE behavior (as DELETE + ADD) and the implications for VACUUMing and disk space consumption are crucial for write-heavy applications. Proactive monitoring of disk usage, dead rows, and vacuum process health is essential to prevent outages.
3. Robust Caching and Queueing Design: Design caching layers and asynchronous queues with extreme care, especially in distributed systems. Monitor cache hit rates meticulously, as a drop can indicate hidden problems and lead to cascading load. Be vigilant for race conditions in queues, which can be notoriously difficult to diagnose but can bring down critical systems. Thorough testing and monitoring of edge cases, particularly with large data payloads, are vital.
4. Multi-Provider Strategy for AI Inference: When relying on third-party AI model providers, assume they will have historically terrible reliability. Implement a multi-provider and multi-cloud strategy to distribute risk, negotiate higher rate limits, and ensure continuous service. Be prepared for "live negotiations" and the need to dynamically balance traffic based on provider performance and availability.
5. Design for Incident Response: Build systems with compartmentalization to limit blast radius. Develop clear incident response protocols, and cultivate a team that thrives under pressure, capable of rapid diagnosis and even on-the-fly refactoring or migration. The ability to "kill traffic" or prioritize users during an outage is a necessary evil.
6. AI for Infrastructure Resilience: While AI infrastructure itself is challenging, AI can also aid in its resilience. Asif mentioned models catching "gnarly bugs" in single-line code changes that human reviewers might miss. Integrating AI-powered code review tools can proactively prevent outages caused by subtle errors.
7. Security by Design: For AI tools handling sensitive data like code, security must be built in from the ground up. Employing encryption for embeddings with client-side keys provides a strong defense-in-depth strategy, even against hypothetical data exfiltration scenarios.

Key Takeaways

• Simplicity Wins in Databases: Opt for simpler, well-understood database solutions like managed PostgreSQL over complex distributed databases in the early stages, even if they seem less scalable on paper. Hyperscalers often provide superior operational reliability for these simpler options.
• Deep Dive into Database Internals: Understand the specific behaviors of your chosen database (e.g., PostgreSQL's UPDATE mechanism) to prevent unexpected scaling issues like runaway disk usage due to dead rows and vacuuming challenges.
• Race Conditions are Insidious: Distributed systems are prone to hard-to-find race conditions, especially in caching and queuing layers. Monitor granular metrics like cache hit rates and design for robust error handling and eventual consistency to mitigate their impact.
• Multi-Cloud/Multi-Provider for AI Resilience: Relying on a single frontier model provider is a single point of failure. Implement a multi-cloud and multi-provider strategy to manage rate limits, ensure redundancy, and navigate the inherent unreliability of external AI services.
• Incident Response is Critical Engineering: High-stakes incidents demand compartmentalization, rapid diagnosis, and a team capable of radical, on-the-fly solutions, including full system refactors and migrations under immense pressure.
• AI Enhances Engineering, Doesn't Replace It: While AI can automate "boring, unimportant work" and even catch subtle bugs, it frees engineers to focus on more creative and complex architectural challenges, leading to more sophisticated systems.

About the Speaker(s)

Sualeh Asif is the CTO and Co-Founder of Cursor, an AI-native development tool. With a background in databases from his studies at MIT, Sualeh possesses a deep understanding of complex data systems and the challenges of operating them at scale. His expertise was evident in his detailed recounting of Cursor's infrastructure journey, particularly the nuanced issues encountered with various database technologies like YugabyteDB and PostgreSQL. As a leader, he embodies a pragmatic, problem-solving approach, trusting his highly talented team to tackle critical infrastructure challenges, even to the point of on-the-fly system rewrites during severe outages. His perspective highlights the blend of theoretical knowledge, practical experience, and a willingness to be "fully unfiltered" about the realities of building cutting-edge AI products.