How Do You Measure Devel... Jennifer Riggins, Cat Morris, Akshaya Aradhya, Laura Tacho & Helen Greul

How Do You Measure Devel... Jennifer Riggins, Cat Morris, Akshaya Aradhya, Laura Tacho & Helen Greul

Speakers: Jennifer Riggins, Cat Morris, Akshaya Aradhya, Laura Tacho, Helen Greul

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=gycxQT3DHIU

Overview

The KubeCon EU panel discussion, "How Do You Measure Devel...", moderated by Jennifer Riggins and featuring industry leaders Cat Morris, Akshaya Aradhya, Laura Tacho, and Helen Greul, delved into the complex and often controversial topic of developer productivity. This talk was not merely an exploration of metrics but a profound discussion on the philosophy, challenges, and strategic importance of understanding how software development teams create value. The panelists, hailing from diverse backgrounds including platform engineering, product management, and CTO roles, converged on a critical insight: measuring developer productivity is less about individual output and more about optimizing the entire software delivery system.

The core of the discussion revolved around shifting the narrative from individual performance to organizational effectiveness, fostering trust, and leveraging data to drive meaningful improvements in developer experience (DevEx). In an era of increasing technological complexity, rapid innovation, and tighter budgets, the ability to accurately assess and enhance how developers work directly impacts a company's bottom line, its capacity for innovation, and its ability to attract and retain top talent. This article dissects the panel's insights, offering a comprehensive look at their perspectives on defining, measuring, and improving developer productivity in a sustainable and human-centric way.

Background

▶ Watch: Introduction and panelist prompts on productivity and questions (0:00)

The quest to quantify developer productivity is as old as software engineering itself, yet it remains one of the industry's most contentious subjects. Historically, early attempts often mirrored manufacturing paradigms, focusing on easily quantifiable but ultimately misleading metrics like lines of code (LOC) or number of commits. As Laura Tacho pointed out, "I can't tell you how many euros my semicolon can sell for," highlighting the fundamental disconnect between physical production and the knowledge work inherent in software development. This reductionist approach not only fails to capture the true value of creative problem-solving, collaboration, and strategic thinking but also fosters a culture of distrust, where developers perceive measurement as surveillance or a precursor to layoffs.

The panelists acknowledged that the problem often arises when product velocity falters, prompting leadership to "dig" into developer productivity from a defensive position. This reactive approach, coupled with a bias towards fixing problems in a counterintuitive way (e.g., adding more engineers to a slow team, which often exacerbates delays), perpetuates the cycle of ineffective measurement. The industry has long been "in search of this one metric that matters," but research, including that behind the DORA metrics, SPACE framework, and DevEx, consistently shows that no single metric can encapsulate the multifaceted nature of software development impact. The need for a more holistic, system-level understanding has driven the development of more sophisticated frameworks, yet their adoption and correct application remain a significant challenge, particularly in bridging the gap between C-suite expectations and developer realities.

Key Findings

▶ Watch: Helen: 'Time to value' and asking about unseen flow hurts (2:59)

The panel discussion illuminated several key findings that redefine the discourse around developer productivity:

1. Redefining Productivity as Time to Impact/Value: The panelists largely agreed that true developer productivity is best understood as time to impact or time to value. Akshaya Aradhya emphasized reducing friction to make developers productive as fast as possible, aligning with customer needs. Helen Greul expanded on this, defining it as a function of efficiency, meaning, satisfaction, and clarity of direction. Cat Morris, drawing on Team Topologies principles, focused on reducing toil to improve flow. Laura Tacho, co-author of the DX Core framework, underscored the need for speed, high quality, great developer experience, and business impact—all working in concert.

1. The Controversy of Measurement and the Trust Deficit: A central theme was the inherent controversy of measuring developer productivity. The panelists highlighted the natural human aversion to being "spied on" and the fear that metrics could be weaponized for individual performance management, especially in economically uncertain times. Laura Tacho explicitly stated that using metrics for stack ranking engineers based on PR output is "weaponizing" them and will cause "long-lasting damage." Trust, therefore, is paramount, and the intention behind measurement must be transparent and focused on system improvement.

1. Organizational and Team-Level Focus, Not Individual: There was a strong consensus that developer productivity should be measured at the organizational or team level, not individually. The term "developer productivity" itself was deemed a "pants name" (in local parlance) by Laura Tacho, as it incorrectly emphasizes the individual. The goal is to improve the system, the "organization as a system," and the "software engineering as a system." This perspective advocates for metrics that assess the collective efficiency, flow, and experience of a team or an entire engineering department.

1. The Power of Small, Incremental Changes: Laura Tacho introduced the concept of asking, "What could we do that would make your life 2% better?" This approach counters the industry's tendency to undervalue small, iterative improvements in favor of grand, often unattainable, "10x" leaps. These 2% changes, accumulated over time, can lead to significant, sustainable improvements in developer experience and overall system performance. This also helps debunk the myth of the "10x developer," instead focusing on enabling "ordinary developers to be really successful" within the organization.

1. Qualitative Feedback is Essential: While quantitative metrics are valuable, the panelists stressed the indispensable role of qualitative feedback. Helen Greul's favorite question, "What is one thing that hurts your day-to-day flow that leadership doesn't see or doesn't realize?", often uncovers surprising insights. Laura Tacho emphasized that "there's no people in your organization that know more about the friction...than the developers." Combining survey data, 1:1 conversations, and pulse checks with workflow metrics provides "directional insight" to prioritize investments effectively.

1. Justifying Investment in Developer Experience (DevEx): A critical challenge is convincing stakeholders that investing in DevEx generates long-term income. Akshaya Aradhya demonstrated this by linking improvements in low cognitive load and fast feedback loops to accelerated delivery timelines. Cat Morris highlighted the cost of delay – the opportunity cost of slow teams – as a powerful argument. Ultimately, connecting DevEx to tangible business outcomes like revenue per engineer or improved retention is key to securing buy-in.

Technical Deep Dive

▶ Watch: Cat: Reduce toil to improve flow; what do you wish I'd asked? (4:30)

The discussion on measuring developer productivity naturally led into a deep dive on various frameworks, tools, and approaches that enable a more nuanced understanding of software development systems. The panelists emphasized moving beyond simplistic metrics to embrace a holistic view.

At the heart of modern developer productivity measurement are frameworks like DORA (DevOps Research and Assessment) metrics, the SPACE framework, and DevEx (Developer Experience), all of which aim to provide a more comprehensive picture than traditional output metrics. Laura Tacho, as the co-author of the DX Core framework, highlighted its role in unifying these concepts. While the talk didn't detail each component of these frameworks, the underlying principle is a multi-dimensional approach:

• DORA metrics typically focus on software delivery performance, including deployment frequency, lead time for changes, mean time to recovery (MTTR), and change failure rate. These provide insights into delivery speed and stability.
• The SPACE framework expands this by considering Satisfaction and Well-being, Performance, Activity, Communication and Collaboration, and Efficiency and Flow. This framework explicitly integrates the human element, acknowledging that developer joy and well-being are critical for sustained productivity.
• DevEx, or Developer Experience, encompasses everything that shapes a developer's interaction with their tools, processes, and environment. A positive DevEx reduces friction, cognitive load, and frustration, directly impacting efficiency and satisfaction.

Panelists referenced specific technologies and concepts that either contribute to or measure DevEx:

• Cratics: Cat Morris's company, Centaso, builds this open-source platform framework designed to reduce cognitive load, emphasizing the role of internal platforms in streamlining developer workflows.
• Backstage: Helen Greul's experience with Backstage at Spotify (an open-source developer portal) underscored the importance of unified internal developer platforms (IDPs) that offer self-service capabilities. These platforms aim to reduce the need for developers to navigate multiple systems or rely on manual handovers, thereby increasing throughput at the system level.
• Helm, Pulumi, Go Deployer: Akshaya Aradhya mentioned these Infrastructure as Code (IaC) tools in the context of improving deployment times. The goal was to enable backend engineers to deploy infrastructure in minutes, rather than hours, by leveraging the infrastructure team's work. This exemplifies how investing in developer tooling and platform capabilities (developers helping developers) can unlock significant organizational value.
• AI/GenAI: The panel discussed the dual impact of AI. Akshaya noted how AI has made cyberattacks "extremely complex," necessitating "good AI agents" to tackle "bad AI agents" in real-time. This highlights the evolving security landscape that developers must contend with. Furthermore, AI can be leveraged for internal productivity, such as using GenAI to triage lengthy pentest reports (e.g., 100 pages long) and even suggest mitigation strategies for "unknown unknowns." This demonstrates AI's potential to reduce cognitive load and improve security posture proactively.

In terms of actionable metrics, the panel advocated for system-level indicators that reflect overall flow and efficiency:

• Self-serve approach: How quickly developers can obtain necessary resources or deploy services without multiple levels of approval or manual intervention.
• Scaffolding new components: The ease and speed with which new projects or components can be initiated using standardized templates and tools.
• Audit log of pipeline analysis: Tracking the efficiency and potential bottlenecks within CI/CD pipelines.
• SLAs uptime/downtime: Monitoring the reliability of internal tools and infrastructure, as downtime directly impacts developer flow.
• Low cognitive load and fast feedback loops: While harder to quantify directly, these are critical outcomes of good DevEx and can be inferred from other metrics and qualitative feedback.
• Revenue per engineer: A higher-level business metric that can correlate with improved engineering efficiency and impact.
• Cost of delay: Quantifying the financial impact of slower delivery, which provides a strong business case for DevEx investments.

Crucially, the panel reiterated that metrics like lines of code or individual Pull Request (PR) output are not suitable for measuring developer productivity. Instead, the focus should be on aggregate, trend-based data that reveals systemic issues and opportunities for improvement across teams, rather than singling out individuals.

Demo / Proof of Concept

▶ Watch: Laura's hot question: 'What could make your life 2% better?' (6:17)

While the KubeCon session was a panel discussion rather than a traditional technical demonstration, the panelists provided compelling real-world examples that served as "proofs of concept" for their philosophies on developer productivity. The most illustrative instance came from Akshaya Aradhya, who recounted a direct experience in justifying investment in developer experience to a skeptical Product Manager (PM).

Akshaya described a situation where their team was facing pressure to ship faster, yet was struggling with underlying friction. The PM initially viewed any investment in developer experience or "tech debt" as a delay to market delivery. Akshaya, however, framed the investment as a strategic move to accelerate future deliveries. The proposition was clear: "Give me a month to basically tackle something... I'm going to speed up your delivery... and push it up to June." The original target for a particular delivery was September.

This was a critical moment where auditable, measurable metrics were crucial for persuasion. Akshaya's argument was built around the tangible benefits of reducing low cognitive load and establishing fast feedback loops for developers. Although the specific technical changes made during that month were not detailed, the outcome served as a powerful validation. While the team didn't hit the ambitious June target, they delivered in August, still a significant acceleration from the original September projection. As Akshaya put it, "a win is a win."

This anecdote functions as a practical proof of concept for demonstrating the business value of DevEx. It shows that by identifying and addressing systemic friction points – through improving tooling, processes, or internal platforms – engineering teams can significantly improve their delivery velocity. The "demo" here was the real-world impact on a product roadmap, directly linking an investment in developer enablement to accelerated time-to-market. It proved that what might initially seem like an internal "cost" can, in fact, be a strategic "investment" with a measurable return, transforming the conversation from "tech debt" to "delivery acceleration."

Defensive Implications

▶ Watch: Debate: No 10x developers, but 10x organizations exist (7:05)

The panel's insights offer critical guidance for both organizations and individual developers navigating the complexities of productivity measurement. For organizations and leadership, the primary defensive implication is the imperative to build and maintain psychological safety and trust. Any attempt to measure developer productivity must be transparent, clearly communicated, and unequivocally focused on improving the system, not on individual performance management. Misapplying metrics for stack ranking will erode trust, foster resentment, and ultimately lead to a less productive and higher-attrition workforce. Leaders must "fine-tune their word smithing," as Laura Tacho suggested, to use language that resonates positively with developers, focusing on "developer experience," "developer thriving," or "system productivity" rather than "developer productivity."

Furthermore, organizations should invest strategically in platform engineering and internal developer platforms (IDPs). The panelists highlighted that reducing cognitive load, improving flow, and enabling self-service capabilities are paramount. This means providing developers with robust tools, clear documentation, and streamlined processes that minimize friction in their day-to-day work. Examples like Backstage or Cratics illustrate how dedicated platform teams can create scaffolding and infrastructure that allow feature teams to move faster and more autonomously. Proactive investment in DevEx, justified by metrics like cost of delay and revenue per engineer, can prevent costly slowdowns and enhance competitive advantage.

From a security perspective, the discussion underscored the need for security to be embedded throughout the CI/CD lifecycle, not as an afterthought. Akshaya Aradhya's experience with complex AI-driven attacks highlights the constant need for vigilance and sophisticated defense mechanisms. Organizations should empower teams with tools, potentially AI-driven, to triage security reports (e.g., pentest reports) and identify vulnerabilities proactively. This integration of security into developer workflows is a critical defensive measure against an evolving threat landscape.

For individual developers, the defensive implications revolve around advocacy and self-empowerment. Developers should feel comfortable speaking up about friction points, inefficiencies, and anything that "hurts their day-to-day flow." If an organization lacks the psychological safety for such feedback, it signals deeper cultural problems that might warrant seeking a different environment. Developers should also understand that investments in DevEx benefit them directly, leading to more sustainable work practices, less toil, and a greater ability to focus on impactful work. By providing constructive feedback and participating in discussions about system improvements, developers become active agents in shaping a more productive and enjoyable work environment.

Key Takeaways

• Shift Focus from Individual to System: Developer productivity is fundamentally about optimizing the entire software delivery system and team performance, not about measuring or comparing individual output. Trust and psychological safety are paramount.
• Embrace Holistic Measurement Frameworks: Move beyond simplistic metrics like lines of code. Utilize multi-dimensional frameworks such as DORA, SPACE, and DX Core to gain a balanced view encompassing speed, quality, developer experience, and business impact.
• Prioritize Developer Experience (DevEx): Invest in platform engineering, internal developer platforms (IDPs), and tools that reduce cognitive load, minimize toil, and create fast feedback loops. This includes self-serve capabilities and streamlined processes.
• Leverage Qualitative and Quantitative Data: Combine hard metrics (e.g., delivery speed, system uptime, cost of delay) with qualitative feedback from developers (surveys, 1:1s, open-ended questions) to understand friction points and guide investment decisions.
• Justify DevEx Investments with Business Value: Frame investments in developer experience not as costs, but as strategic moves that accelerate time-to-market, improve product quality, reduce attrition, and ultimately increase revenue per engineer and lower the cost of delay.
• Embrace Incremental Improvement: Focus on making small, consistent "2% better" changes rather than chasing elusive "10x" improvements. These iterative enhancements accumulate to significant, sustainable gains in productivity and developer well-being.

About the Speaker(s)

The panel was moderated by Jennifer Riggins, who skillfully guided the discussion and engaged with audience questions.

Akshaya Aradhya is a VP of Engineering at Oceler, a company founded by the creator of Kafka. His extensive background includes significant roles at Netflix and GitHub, where he contributed to the platform and infrastructure teams responsible for projects like GitHub Copilot. Akshaya defines developer productivity as "time to impact," focusing on reducing friction and enabling rapid productivity.

Helen Greul serves as the VP of Engineering at Multiverse, a tech company dedicated to closing skill gaps through personalized learning. With a decade of experience working with platform teams, including her tenure with Backstage at Spotify, Helen brings deep expertise in building frameworks for measuring developer productivity. She views productivity as "time to value," a function of efficiency, meaning, satisfaction, and clarity.

Cat Morris is a Product Manager at Centaso, where she works on Cratics, an open-source platform framework aimed at reducing cognitive load. Prior to Centaso, Cat spent five or six years as a platform product manager at Thoughtworks, indicating a long-standing commitment to improving developer flow by eliminating waste and toil.

Laura Tacho is the CTO at DX, a developer intelligence platform that assists companies in measuring and enhancing developer productivity. She is also notably the co-author of the DX Core framework, a unified approach to measuring developer productivity developed in collaboration with researchers behind DORA, SPACE, and DevEx. Laura advocates for a holistic definition of productivity encompassing speed, quality, developer experience, and business impact.