From the Observability TAG: Designing a Common Query... lolita Sharma, Pereira Braga & Chris Larsen

From the Observability TAG: Designing a Common Query... lolita Sharma, Pereira Braga & Chris Larsen

Speakers: Alolita Sharma, Lead Observability Engineering, Apple; Pereira Braga, Observability Steward, Google; Chris Larsen, Observability Engineer, Netflix

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=1iWD14xvBQA

Overview

The KubeCon EU talk "From the Observability TAG: Designing a Common Query..." addresses a critical challenge in modern cloud-native environments: the overwhelming complexity arising from the proliferation of Domain-Specific Languages (DSLs) for querying observability data. As systems become more distributed and the volume and variety of telemetry (metrics, logs, traces, profiles, wide events) explode, practitioners are forced to learn and manage numerous distinct query languages, leading to significant inefficiencies and increased operational burden. This session, presented by leaders from Apple, Google, and Netflix, representing the CNCF Observability TAG, proposes a radical yet pragmatic solution: standardizing on an extended version of SQL as a universal query language for all observability data.

The speakers, Alolita Sharma, Chris Larsen, and Pereira Braga, highlight the shift in observability data consumption from a niche SRE function to a broad organizational need, encompassing developers, platform engineers, security teams, AI engineers, managers, and even external customers. This expanded user base exacerbates the pain points of fragmented query languages, making it difficult to gain a holistic understanding of system behavior and impacting everything from incident response to business intelligence. The talk presents compelling research from Google and Netflix, underscoring the high cost of learning bespoke DSLs and demonstrating the inherent relational nature of observability data, which naturally aligns with SQL's strengths.

Ultimately, the talk is a call to action for the cloud-native community to embrace a common, extensible query standard. By leveraging the familiarity and power of SQL, enhanced with specific extensions for time series and a more intuitive pipe syntax, the aim is to drastically reduce complexity, lower onboarding costs, improve cross-domain correlation, and pave the way for more effective integration of AI and Large Language Models (LLMs) in observability. This standardization effort promises to unlock greater engineering efficiency and foster a more unified approach to understanding and managing complex distributed systems.

Background

▶ Watch: Introduction and talk objective: common query method (0:00)

The landscape of modern software development, particularly within cloud-native architectures, has undergone a profound transformation. Applications are increasingly distributed, composed of numerous microservices running across dynamic infrastructures like Kubernetes. To effectively monitor and manage these complex systems, observability data—encompassing metrics, logs, traces, profiles, and wide events—is crucial. However, the very tools and platforms designed to collect, store, and analyze this data have inadvertently created a new layer of complexity: a fragmented ecosystem of Domain-Specific Languages (DSLs) for querying.

Historically, each type of observability data often came with its own specialized storage solution, and consequently, its own query language. For instance, querying metrics might involve PromQL for Prometheus, Graphite query language, or Flux for InfluxDB. Logs often required learning SplunkQL, ElasticQL for Elasticsearch, or LogQL for Loki. Distributed traces introduced yet more languages like TraceQL or New RelicQL. This proliferation means that an engineer trying to diagnose an issue might need to interact with half a dozen or more distinct query syntaxes, each with its own semantics, functions, and operational nuances.

The problem is compounded by the expanding demographic of observability data consumers. Traditionally, observability was primarily the domain of Site Reliability Engineers (SREs). Today, however, developers are increasingly on-call and responsible for their services end-to-end. Platform engineers need to understand system usage for scaling. Security engineers analyze logs and events for threats. AI engineers require operational data for model training and performance monitoring. Even managers, support engineers, and external customers are demanding insights from observability data to understand system health, customer impact during outages, or product performance. For these diverse groups, learning a multitude of bespoke DSLs presents a significant barrier, increasing onboarding time, hindering cross-domain analysis, and ultimately slowing down incident resolution and innovation.

While projects like OpenTelemetry have made significant strides in standardizing the collection and ingestion of telemetry data, the querying aspect has remained largely unaddressed. The CNCF Observability TAG recognized this gap and initiated a workgroup focused on defining open observability query semantic definitions. This talk directly stems from that effort, presenting research and a proposed direction to overcome the current query language fragmentation, aiming for a unified approach that leverages the widespread familiarity and power of SQL.

Key Findings

▶ Watch: Defining observability data types: metrics, logs, traces, profiles (4:00)

The talk presents several compelling findings derived from extensive research conducted by both the CNCF Observability TAG and internal teams at Google and Netflix, all pointing towards the necessity and feasibility of a standardized query language.

Firstly, the CNCF workgroup interviewed 11 different DSL designers, including those behind prominent observability tools. A key revelation was that the underlying data store is often not as tightly coupled to the query syntax as commonly perceived. Many designers expressed that their languages could be extended to support other telemetry types without fundamental architectural changes. More importantly, these interviews revealed significant commonalities across DSLs, including similar predicates, functions, and aggregation capabilities. While a few (like Tempo's TraceQL) supported graph predicates, the core operational patterns were strikingly similar, suggesting a strong potential for convergence.

Secondly, internal research at Google quantified the hidden costs of DSL proliferation. They found that engineers joining Google, when faced with multiple bespoke query languages for observability, spent an average of two weeks of engineering time just to learn these new syntaxes. This cost is not a one-time expense; individuals who interact with observability data irregularly must constantly refresh their knowledge, leading to recurring inefficiencies. Google's research also highlighted a demographic shift: a majority of observability users are now developers, not just SREs, further amplifying the impact of complex, non-standardized tooling.

Thirdly, Netflix's internal analysis corroborated Google's findings. Chris Larsen noted that the primary source of support burden for Netflix's observability engineering team came from developers struggling with their custom, RPN-based Atlas stack language. This language, while powerful, requires a significant learning investment. The data also showed that beyond developers and SREs, data scientists (10-22% usage in Netflix tools), managers, and even external customers (like ISPs seeking visibility into Netflix's operations) are increasingly using observability data. This broad consumption base underscores the urgent need for an intuitive and widely understood query mechanism.

A crucial insight emerged from analyzing the inherent structure of observability data: it is fundamentally relational. Metrics, logs, and traces are often interconnected through common attributes, tags, or keys and values. This realization, coupled with the observation that cross-correlation between different telemetry types is becoming increasingly vital for root cause analysis, strongly suggests that a language designed for relational data would be the most effective solution.

Finally, the talk directly addresses the role of AI and LLMs in observability. While promising, current LLMs face challenges like hallucinations when tasked with querying diverse, non-standardized data sources. The cost of vectorizing data for LLM analysis at scale is also currently too high to justify the ROI. The speakers conclude that for generative AI to be truly effective in observability, a common data model, a common query standard, and common semantics across all telemetry types are essential. This foundational standardization would reduce LLM hallucinations, improve accuracy, and make AI-driven insights more reliable and cost-effective.

These findings collectively build a compelling case for adopting SQL as the recommended common query language, positioning it as a robust, familiar, and extensible solution to address the current fragmentation and future demands of observability.

Technical Deep Dive

▶ Watch: Complexity of multiple storage and query languages (6:00)

The core technical proposal from the CNCF Observability TAG and Google is to adopt an extended subset of ANSI SQL as the standard query language for all observability data. This choice is rooted in SQL's proven capabilities for handling relational data, its widespread familiarity, and its continuous evolution to support modern data paradigms.

The recommendation specifically focuses on the SELECT part of SQL semantics, explicitly avoiding Data Manipulation Language (DML) operations like INSERT, UPDATE, or DELETE, as these are not relevant for querying observability data. The goal is to define data models by type, ensuring full support for OpenTelemetry models, which are becoming the de facto standard for telemetry collection. This means the query language would operate on standardized representations of metrics, logs, traces, and profiles.

A key aspect of this approach is to focus on relational execution engines. While acknowledging that some existing systems might have different underlying storage and processing models, the working group aims to define a standard that can be mapped to various backends, potentially through an intermediate representation or query plan. This allows for federated queries across diverse data sets—metrics, logs, traces, and even traditional production databases or BI systems—all using a single, consistent language.

To make SQL more suitable for observability, the proposal includes defining standard functions and syntactic sugar to reduce verbosity and align with common observability use cases. This is where the concept of pipe syntax becomes particularly innovative and impactful. Inspired by the Unix philosophy of chaining commands, pipe syntax allows for a more direct and sequential expression of data transformations and aggregations, making queries more intuitive and readable, especially for data exploration.

Pereira Braga demonstrated the power of pipe syntax with a conceptual example of calculating a rate (similar to PromQL's rate function). A traditional SQL query for such a calculation can be very verbose, spanning multiple lines or even pages, with complex nested subqueries and window functions. In contrast, pipe syntax allows for a more streamlined expression:

This example, while illustrative, highlights how pipe syntax reorders the semantic flow to mirror how users typically think about data processing: start with a table, then apply transformations in sequence. Google's internal adoption of this pipe syntax has shown significant and sticky usage growth, indicating its effectiveness in improving engineering efficiency.

The evolution of ANSI SQL itself supports this direction. Recent standards have introduced temporal support (2011), native JSON handling (2016), and even initial graph entities (2023), making it increasingly capable of handling diverse and modern data structures. Furthermore, the trend among NoSQL databases, such as Google's Bigtable, Cassandra, and HBase, to eventually add SQL support demonstrates the language's enduring power and flexibility. Even systems like DuckDB, Databricks Spark, and Firebolt are now integrating pipe syntax, validating its utility beyond Google's internal ecosystem. The open-source version of Google SQL, ZetaSQL, also includes this pipe syntax, making it accessible for broader community adoption and experimentation.

This technical approach aims to combine the robust, well-understood foundation of SQL with modern extensions tailored for observability, creating a powerful and user-friendly standard that can unify querying across the entire telemetry landscape.

Demo / Proof of Concept

▶ Watch: Observability lifecycle: using logs, metrics, traces, profiling (6:30)

While the talk did not feature a live, interactive demonstration, it provided compelling evidence and conceptual examples that serve as a strong proof of concept for the proposed SQL-based approach with pipe syntax extensions.

The most illustrative conceptual demonstration involved contrasting a verbose, multi-page SQL query for calculating a Prometheus-style rate with a significantly more concise and readable version using pipe syntax. As Pereira Braga explained, traditional SQL for complex time-series aggregations can become cumbersome, requiring intricate nesting and a non-intuitive order of operations. The pipe syntax, however, allows users to express their intent in a sequential, Unix-like fashion, making it much easier to compose and understand queries. This conceptual comparison clearly articulated the benefits of the proposed extension in terms of readability and ease of use, directly addressing the verbosity concern often associated with complex SQL queries in observability contexts.

Beyond the conceptual example, the speakers provided concrete evidence of the real-world application and success of this approach within Google. Pereira Braga highlighted the "really, really usage" growth of pipe syntax within Google's internal query engines, such as F1 and BigQuery. This internal adoption, described as "sticky" and "spreading very fast," serves as a powerful validation of the design's effectiveness in a large-scale, production environment with a diverse user base. The fact that Google, a pioneer in distributed systems and data management, has converged on this SQL-based approach internally lends significant credibility to the proposal.

Furthermore, the talk detailed the external availability and growing industry adoption of pipe syntax. It was noted that this functionality is already available in Google BigQuery, Databricks Spark (via Spark support), and was recently launched for Firebolt. The open-sourcing of Google SQL as ZetaSQL, which includes the pipe syntax implementation, provides a tangible platform for the broader community to experiment with and contribute to this evolving standard. This widespread integration in various data platforms demonstrates that the concept is not merely theoretical but is being actively implemented and validated across the industry, reinforcing its viability as a common query method.

The availability of the CNCF draft semantic specification, the ZetaSQL project, and a dedicated paper on pipe syntax (all linked via QR codes in the presentation) further solidifies the proof of concept. These resources allow interested parties to dive into the technical details, understand the underlying semantics, and even begin contributing to the standardization effort. The collective weight of internal success, external product integration, and open-source availability strongly underpins the feasibility and practical benefits of SQL with pipe syntax for unifying observability queries.

Defensive Implications

▶ Watch: Challenges in analyzing observability data and custom needs (8:30)

The push for a common SQL-based query language for observability has profound implications for defensive strategies within organizations, significantly enhancing their ability to detect, investigate, and respond to security incidents and operational issues.

Firstly, by unifying query capabilities across metrics, logs, and traces, security teams can achieve a much more holistic view of system behavior. Currently, correlating events across different telemetry types often requires switching between multiple DSLs and tools, which is time-consuming and prone to human error during high-pressure incidents. A standardized SQL interface allows security analysts to write single queries that can join and filter data from various sources, identifying suspicious patterns that might be invisible when looking at data silos. For example, a single SQL query could correlate an unusual spike in network metrics with specific error logs from an application and suspicious user activity from an audit trace, providing a comprehensive context for anomaly detection and root cause analysis.

Secondly, the reduction in observability query language fragmentation directly translates to a lower cognitive load for security engineers. Learning and maintaining proficiency in numerous bespoke languages consumes valuable time and resources. A common SQL standard, especially one enhanced with intuitive pipe syntax, simplifies onboarding for new security hires and reduces the need for constant re-skilling. This efficiency allows security teams to focus more on threat intelligence, incident response playbooks, and proactive security measures rather than struggling with tool intricacies. The "two weeks of engineering time" saved on learning bespoke languages, as highlighted by Google's research, can be directly reinvested into improving security posture.

Thirdly, the proposed standardization creates a fertile ground for more effective integration of AI and LLMs into security operations. With a common data model and query semantics, AI systems can process and analyze observability data with greater accuracy and fewer "hallucinations." This means more reliable automated threat detection, faster anomaly correlation, and potentially LLM-assisted natural language querying for incident responders, allowing them to ask questions like "Show me all failed logins from outside the US in the last hour correlated with unusual CPU spikes on authentication services" and receive precise, actionable results. This reduces the ROI barrier currently associated with vectorizing data for LLM analysis at scale.

Finally, platform and security engineering teams should actively investigate and advocate for the adoption of SQL-based observability platforms or extensions. This could involve leveraging existing tools that support SQL (like BigQuery, Databricks Spark, Firebolt) or contributing to open-source initiatives like ZetaSQL and the CNCF Observability TAG's work. By providing a consistent query experience, organizations can empower a broader range of personnel—including developers doing security checks, SREs triaging incidents, and data scientists performing forensic analysis—to effectively utilize observability data for defensive purposes, ultimately leading to more resilient and secure systems.

Key Takeaways

• The Proliferation of Observability DSLs is Unsustainable: Modern cloud-native environments are plagued by a multitude of Domain-Specific Languages (DSLs) for querying metrics, logs, and traces, leading to significant complexity, increased onboarding costs (e.g., ~2 weeks of engineering time at Google), and reduced efficiency for a growing user base.
• Observability Data is Fundamentally Relational: Research from both CNCF and internal teams at Google and Netflix confirms that diverse telemetry types are interconnected through common attributes, making SQL a natural and powerful fit for cross-domain correlation and analysis.
• SQL with Extensions Offers a Unified Solution: The proposed standard leverages ANSI SQL, extended with specialized features like time series support and an intuitive pipe syntax, to provide a common, familiar, and highly capable language for all observability queries.
• Pipe Syntax Enhances User Experience: The pipe syntax, adopted internally at Google with significant success and now available in tools like BigQuery, Databricks Spark, and Firebolt, dramatically improves the readability and ease of authoring complex queries by allowing a more direct, sequential expression of data transformations.
• Standardization is Crucial for AI in Observability: A common query standard and data model are essential prerequisites for effective and cost-efficient integration of AI and Large Language Models (LLMs) into observability, reducing hallucinations and improving the ROI of AI-driven insights.
• Wider Adoption is Key: Organizations should advocate for and adopt SQL-based observability solutions, exploring open-source initiatives like ZetaSQL and contributing to the CNCF Observability TAG's work to accelerate the industry-wide convergence on a unified query language.

About the Speaker(s)

Alolita Sharma is a lead in Observability Engineering at Apple and serves as a co-chair for the CNCF TAG Observability. Her work focuses on advancing the state of observability practices and standards within the cloud-native ecosystem. She is a key contributor to the query standardization specification work group within the TAG.

Chris Larsen is an Observability Engineer at Netflix, where he is part of the productivity team under the platform organization, with a remit to make tools easy to use for developers and other internal users. He is an active member of the CNCF TAG's query standardization specification work group, contributing insights from Netflix's extensive experience with observability at scale.

Pereira Braga is an Observability Steward at Google. He has conducted extensive internal research within Google to understand how observability is utilized by various user groups, including developers, and has been instrumental in exploring and advocating for the adoption of SQL extensions, such as pipe syntax, to address query language fragmentation within the company.