Starting and Scaling a Platform Engineering Team - Camille Fournier & Ian Nowland

Starting and Scaling a Platform Engineering Team - Camille Fournier & Ian Nowland

Speakers: Camille Fournier, Co-author; Ian Nowland, Former VP at DataDog

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=9lPp-6nJ8bI

Overview

This talk, delivered by Camille Fournier and Ian Nowland at KubeCon EU, delves into the critical strategies and common pitfalls associated with establishing and growing effective platform engineering teams. Drawing from their extensive experience, the speakers provide a foundational understanding of when to initiate such a team, the essential skill sets required, and the operational models that lead to sustainable success. They argue that platform engineering is not merely an evolution of DevOps but a distinct discipline rooted in software engineering principles, operational excellence, and a keen product focus.

The core of the presentation revolves around practical advice for delivering tangible value, managing complexity, and fostering developer productivity within a rapidly scaling organization. Camille Fournier outlines the theoretical framework and common mistakes, emphasizing the necessity of a product-centric approach and continuous stakeholder communication. Ian Nowland then reinforces these concepts through a compelling case study from his tenure as a VP at DataDog, illustrating how a data platform team navigated multiple resets—from reactive operations to failed "build it and they will come" strategies—before finally achieving success through strategic product thinking.

This discussion is particularly relevant in today's cloud-native landscape, where the proliferation of tools and services, while enabling rapid initial development, often leads to an unmanageable sprawl of "glue code" and unmanaged shared infrastructure as companies scale. The speakers offer a clear roadmap for organizations to move beyond this bottleneck, transforming internal infrastructure into a curated, self-service product that empowers application teams and ensures long-term operational health.

Background

▶ Watch: Introduction to platform engineering and talk agenda (0:00)

The rapid evolution of the cloud-native ecosystem, exemplified by the nine-year history of the CNCF and ten years of Kubernetes, has dramatically lowered the barrier to entry for new companies. Developers can easily provision infrastructure in the cloud and quickly deploy applications using YAML and other configurations. This initial agility, however, comes with a hidden cost: the creation of extensive "glue code"—one-off automation, integration code, custom configurations, and administrative tools that bind applications to their underlying building blocks. While manageable for small teams, this glue code becomes a significant hindrance as companies scale.

As an organization grows from, for instance, 50 to 250 engineers, the communal support for shared products like CI/CD pipelines or databases falters. Upgrades to core infrastructure components, such as Kubernetes or EKS, or the implementation of critical security patches, become arduous, slowing down every application team. This is the inflection point, according to Fournier and Nowland, where a dedicated platform engineering team becomes not just beneficial, but essential. Starting such a team prematurely in a small startup, however, is often an unnecessary overhead.

The speakers define a platform as "the foundation of self-service APIs, tools, services, knowledge, and support... curated into a product offering that application teams can use to deliver features more quickly." This definition underscores the team's core mission, which revolves around three primary goals:

1. Manage Complexity: Abstracting the myriad underlying building blocks and diverse architectural choices to simplify the operational landscape for application teams.
2. Create Leverage: Concentrating specialized expertise within the platform team to build robust solutions that would be challenging for individual application teams, thereby enabling the organization to achieve more with fewer resources.
3. Improve Productivity: Empowering application teams to focus on delivering business features rather than wrestling with infrastructure concerns, ultimately enhancing overall organizational efficiency.

These goals necessitate a specific blend of skills and a strategic approach, moving away from traditional infrastructure models to a more product-oriented mindset that directly serves the internal developer customer.

Key Findings

▶ Watch: Ideal company size for initiating a platform team (3:50)

The talk distills several critical findings for successfully building and scaling platform engineering teams, highlighting a blend of technical expertise, operational commitment, and strategic product thinking.

Firstly, platform engineering is fundamentally software engineering. It's not merely an extension of DevOps or automation; it involves building sophisticated abstractions, self-service interfaces, and multi-tenancy solutions. This includes modifying underlying open-source components, enhancing APIs, and implementing robust security and quality guardrails. Without a strong software engineering foundation, platform teams struggle to effectively manage the broad interfaces and inherent complexity of modern infrastructure.

Secondly, operational ownership is non-negotiable. Platform teams must be willing to operate the critical systems they build upon and abstract. This commitment ensures that the team is seen as essential to the business, securing budget and resources. Moreover, operating these systems provides invaluable feedback, leading to the development of more trustworthy, resilient, and scalable foundations. Neglecting operations reduces a platform team to an "optional nice-to-have," making it vulnerable to budget cuts.

Thirdly, a product-focused approach is paramount. Unlike old-school central infrastructure teams that often optimize for cost and deliver offerings developers dislike, platform teams must prioritize developer productivity. This means actively engaging with internal customers to understand their needs and wants, curating platform offerings based on their input, and fostering customer empathy throughout the team. Success is measured not just by adoption, but by the tangible improvements in productivity and business impact.

The speakers also emphasized crucial aspects of the platform execution lifecycle:

• Deliver early wins and earn trust: New platform teams should avoid lengthy, isolated development cycles. Instead, they should focus on delivering incremental value quickly to prove their impact and build credibility within the organization.
• Internal market research is key: Rather than purely innovating, platform teams should identify existing solutions that application teams have already built and proven useful, then productize and generalize them for broader organizational use.
• Avoid complete rewrites: Resist the temptation to rebuild functional systems from scratch just to "scale." Incremental improvements and adaptations are often more effective and less risky.
• Measure beyond adoption: True platform success lies in its impact on productivity, scalability, and other business metrics, not just how many teams adopt it.
• Abstract migration pain: A significant opportunity for platform teams is to absorb and manage the burden of infrastructure migrations (e.g., EKS upgrades, language version changes) rather than pushing this work onto application teams.

Camille Fournier summarized these principles into four critical questions: "Are you writing software?", "Are you operating critical systems?", "Are you talking to your customers?", and "Are you communicating with your stakeholders?".

Ian Nowland's DataDog case study further highlighted these findings through a series of "resets" a data platform team underwent. It demonstrated the failures of:

• Reactive operational models: Merely vending open-source databases (Kafka, Cassandra, ElasticSearch, PostgreSQL) without abstraction led to 50 incidents a week and team burnout.
• Pure consulting: Retreating from operational ownership resulted in a loss of context, worse incidents, and diminished influence.
• SLA/SLO-centric approaches: Over-reliance on documentation and strict service level agreements created bureaucracy and failed to solve underlying problems.
• "Build it and they will come": Developing complex, future-looking solutions (like an in-house multi-region key-value store akin to Spanner) that didn't align with immediate business needs led to irrelevance and a perception of being out of touch.

Ultimately, DataDog's success came from embracing true product thinking, aligning platform health, internal developer needs, and short-term business value. This involved raising abstraction levels for common patterns (e.g., a "caching searchable database" combining ElasticSearch, Redis, and PostgreSQL) and engaging in active partnerships with key business initiatives, all while avoiding "future platform" language that instilled fear of migration.

Technical Deep Dive

▶ Watch: Definition: self-service APIs, tools, services, knowledge, support (4:10)

The technical core of platform engineering, as articulated by Camille Fournier, lies in its strong software engineering foundation. This isn't just about scripting or automation; it's about building robust, reusable systems. Key technical activities include:

• Self-Service Interfaces: Developing user-friendly portals, CLIs, or APIs that allow application teams to provision, configure, and manage their resources without direct intervention from the platform team. This often involves building custom controllers or operators for Kubernetes to abstract underlying complexity.
• Multi-Tenancy: Architecting systems to securely and efficiently serve multiple application teams or business units from shared infrastructure. This requires careful consideration of isolation, resource allocation, and security policies.
• Modifying Open Source: Adapting and enhancing existing open-source projects (e.g., Kafka, Cassandra, ElasticSearch, PostgreSQL as seen in DataDog's case) to fit specific organizational needs, add custom features, or integrate with internal systems. This might involve contributing upstream or maintaining forks.
• Enhanced APIs and Guardrails: Creating higher-level APIs that simplify interactions with complex infrastructure components, and embedding security and quality guardrails directly into the platform to ensure compliance and best practices are followed automatically. This could involve policy engines like Open Policy Agent (OPA).
• Combined Products: Building opinionated, integrated solutions that bundle multiple underlying services into a coherent offering. For example, a "database-as-a-service" might combine a PostgreSQL instance with integrated backup, monitoring, and logging solutions, all exposed through a single, simplified interface.

Operational excellence is another crucial technical pillar. This involves:

• System Skills: Employing DevOps, SRE, or infrastructure engineers with deep expertise in debugging complex distributed systems, optimizing performance, and ensuring high availability and resilience. This includes mastering tools for observability, incident response, and capacity planning.
• On-Call Ownership: The platform team taking direct responsibility for the operational health and on-call rotations of the core platform components. This ensures they feel the pain points directly, driving improvements and fostering a sense of ownership.
• SLOs and Stability: Defining and meticulously tracking Service Level Objectives (SLOs) to ensure the platform meets agreed-upon performance and reliability targets, along with continuous efforts to improve system stability.

A significant technical opportunity for platforms is abstracting migration pain. This involves designing the platform in such a way that it can handle major infrastructure upgrades (e.g., EKS version bumps) or language runtime migrations (e.g., Python 2 to Python 3) internally, minimizing the work required from application teams. This often necessitates robust automation, blue/green deployment strategies, and backward compatibility efforts within the platform itself.

Ian Nowland's case study at DataDog provided concrete technical examples of both challenges and successful strategies:

• Initial Operational Hell: The data platform team was responsible for managing open-source data stores like Kafka, Cassandra, ElasticSearch, and PostgreSQL on Kubernetes across all three major cloud providers. The challenge was exacerbated by Kubernetes' initial poor support for stateful workloads and the subtle differences in behavior across cloud environments under heavy load. Product teams, given broad access to these systems, would inevitably create "hotspots" or "hot shards," leading to 50 incidents per week requiring expert intervention.
• Failed Solutions:
• An attempt to "get out of the game" by pushing teams to use vendor-managed services like RDS failed because cross-cloud inconsistencies and specific workload patterns still led to issues requiring the platform team's expertise.
• The "build it and they will come" approach involved an ambitious plan to build an in-house reduced interface key-value store with global data store capabilities, conceptually similar to Google Spanner or CockroachDB. This was a massive technical undertaking, but it failed because the immediate business need was for standard SQL, and the multi-region requirement was not a pressing concern for most customers.
• Successful Product-Driven Technical Solutions:
• The team developed a specialized global configuration store that met a specific, high-value need.
• They identified a common pattern: application teams often combined ElasticSearch, Redis, and PostgreSQL to create a "caching searchable database." Instead of vending these components separately, the platform team created a higher-level abstraction that integrated these services. This combined offering imposed certain restrictions on usage but delivered significant value by simplifying a complex, recurring problem for application teams, who were willing to accept limitations for the increased productivity. This demonstrated the power of identifying common architectural patterns and productizing them as integrated, opinionated services.

This shift from merely providing infrastructure components to building opinionated, abstracted products that solve specific developer pain points epitomizes the technical evolution required for successful platform engineering.

Demo / Proof of Concept

▶ Watch: Three main goals: manage complexity, create leverage, improve productivity (4:40)

This talk did not feature a live technical demonstration or proof of concept in the traditional sense. Instead, Ian Nowland presented a detailed case study from his experience at DataDog. This case study served as an illustrative "proof of concept" by tracing the journey of a data platform team through various strategic attempts—some failing, some succeeding—to demonstrate the real-world implications of different platform engineering philosophies. The outcomes of these approaches, including the team's initial "operational hell" with 50 incidents a week and its eventual success in gaining headcount and stakeholder approval, acted as empirical evidence for the principles discussed.

Defensive Implications

▶ Watch: Platform engineering is software engineering; building abstractions (6:00)

For organizations, the insights from this talk provide a robust framework for building and operating platform teams that are truly an asset rather than a cost center. These "defensive implications" (or rather, strategic best practices) are crucial for ensuring the platform team's sustainability, impact, and alignment with business goals.

Strategic Team Formation and Mandate:

• Timing is Key: Do not start a platform team too early. The optimal time is when the organization has scaled significantly (e.g., 50 to 250 engineers) and "glue code" or unmanaged shared infrastructure begins to impede developer velocity and cause operational pain (e.g., difficult Kubernetes upgrades or security patches).
• Balanced Skill Set: Build a team that explicitly blends software engineering expertise, system skills (DevOps, SRE, infrastructure engineers), and a product-focused mindset. Neglecting any of these three pillars will lead to an ineffective platform. Software engineers are needed to build robust abstractions, systems engineers for operational excellence, and product thinking to ensure developer needs are met.
• Cultivate Customer Empathy: Even if dedicated product managers are not immediately feasible, instill customer empathy within the platform team, particularly among managers and senior individual contributors. The platform is an internal product, and understanding its users is paramount.

Execution and Delivery Best Practices:

• Prioritize Quick Wins: In the early stages, focus on delivering tangible, high-impact value quickly to build trust and demonstrate the team's worth. Avoid going dark for months or years to build a "perfect" platform that might not meet actual needs upon release.
• Internal Market Research: Instead of purely greenfield innovation, look for existing, proven solutions developed by application teams that can be generalized, productized, and offered as a shared service. This reduces risk and ensures the platform addresses real, demonstrated needs.
• Iterate, Don't Rewrite: Resist the urge to completely rewrite existing, functional internal systems. Focus on incremental improvements, abstractions, and making existing solutions more robust and self-service.
• Measure True Impact: Define success beyond simple adoption metrics. Focus on quantifiable improvements in developer productivity, system stability (e.g., reduced incident count), cost efficiency, and time-to-market for application teams. Communicate these metrics regularly to stakeholders.
• Abstract Migration Pain: A key strategic advantage for platforms is to absorb the burden of significant infrastructure changes (e.g., EKS version upgrades, language runtime migrations like Python 2 to 3). By doing this, the platform team protects application teams from constant, disruptive migration work, significantly boosting overall productivity.
• Provide Robust User Support: Recognize that platform users will have questions beyond critical incidents. Establish clear support channels and processes to handle "how-to" queries, preventing distraction for core developers and ensuring effective platform utilization.

Operational Model and Avoiding Pitfalls:

• Own the Operations: Platform teams must own the operational responsibility and on-call for the critical systems they build and abstract. This is essential for gaining budget, being seen as a critical business function, and ensuring the platform is designed for reliability and maintainability. Merely "vending open source" without operational ownership leads to burnout, blame, and a lack of influence, as seen in DataDog's initial struggles with Kafka, Cassandra, ElasticSearch, and PostgreSQL.
• Avoid the "Consulting Trap": Retreating to a purely consulting role without operational control leads to a loss of context, less effective advice, and diminished influence when actual problems arise.
• Beyond SLAs/SLOs: While important for setting expectations, an over-reliance on SLAs and SLOs as a substitute for direct problem-solving and abstraction can make the platform team appear bureaucratic and unhelpful to application teams.
• Resist "Build It and They Will Come": Do not embark on massive, multi-year projects to build speculative, future-proof systems (e.g., a custom multi-region key-value store like Spanner or CockroachDB) if current business needs are simpler (e.g., standard SQL). This leads to wasted effort, a perception of being out of touch, and a failure to deliver immediate business value. Focus on what the business needs now and in the next 12 months.

Communication and Stakeholder Management:

• Listen to Requirements, Not Solutions: When interacting with customers (application teams), actively listen to their underlying problems and requirements, but be wary of directly implementing their proposed solutions. The platform team's expertise lies in synthesizing diverse requirements into a cohesive, abstracted solution that works for many.
• Proactive Stakeholder Communication: Regularly communicate the value and impact of the platform team's work to key stakeholders (managers, budget holders, heads of product). Articulate this value in terms that resonate with their priorities (e.g., cost savings, increased developer velocity, improved reliability), rather than purely technical terms. This "sales" aspect is crucial for securing headcount and investment.

By adhering to these principles, organizations can establish platform engineering teams that are not only technically proficient but also strategically aligned, customer-centric, and capable of driving significant business value.

Key Takeaways

• Strategic Timing for Platform Teams: Initiate platform engineering when an organization scales beyond 50-250 engineers, and the complexity of "glue code" and unmanaged shared infrastructure becomes a significant bottleneck for developer productivity.
• Three Pillars of Platform Success: A successful platform team inherently blends strong software engineering for building abstractions, operational excellence for managing critical systems, and a product-focused approach to meet internal developer needs.
• Operational Ownership is Crucial: Platform teams must own the operational burden of the systems they provide. This commitment ensures they build trustworthy foundations, gain influence, and are perceived as essential to the business, avoiding the pitfalls of merely vending open-source components.
• Product-Driven Execution: Prioritize delivering quick wins, conducting internal market research to identify existing needs, and abstracting migration pain from application teams. Avoid lengthy, isolated "build it and they will come" projects that do not align with immediate business value.
• Effective Communication is Paramount: Engage continuously with customers, listening to their requirements (not necessarily their proposed solutions), and proactively communicate the platform's value and impact to stakeholders in terms they understand to secure resources and maintain alignment.
• Beware of Common Pitfalls: Avoid simply vending broad open-source interfaces, retreating into a pure consulting role, over-relying on rigid SLAs/SLOs, or pursuing speculative future-state architectures that lack immediate business demand.

About the Speaker(s)

Camille Fournier is a recognized expert in the field of platform engineering and organizational design. She is the co-author of a book on platform engineering, indicating her deep theoretical and practical knowledge in the domain. Her insights in this talk are drawn from extensive experience in building and scaling technology organizations.

Ian Nowland served as a Vice President at DataDog from 2019 to 2023. During his tenure, he gained significant practical experience in leading and transforming engineering teams, particularly in the realm of data platforms. His contribution to the talk, a detailed case study from DataDog, provides a real-world perspective on the challenges and successful strategies involved in starting and scaling platform engineering initiatives.