Keynote: Into the Black Box: Observability in the Age of LLMs - Christine Yen

Keynote: Into the Black Box: Observability in the Age of LLMs

Speakers: Christine Yen, Co-founder and CEO, Honeycomb

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=R0255efML-I

Overview

In this KubeCon EU keynote, Christine Yen, co-founder and CEO of Honeycomb, addresses a critical challenge facing modern software development: how to maintain reliability and debug systems built with Large Language Models (LLMs). As LLMs become ubiquitous, offering "magical" capabilities through cheap API calls to foundation models, they introduce a new layer of complexity and unpredictability that traditional software development practices are ill-equipped to handle. Yen argues that the very nature of LLMs – their nondeterminism, black-box operation, and unbounded input variations – renders conventional testing, mocking, and debugging strategies largely ineffective.

The talk posits that while the LLM boom feels like a "phase change," the solution isn't to abandon decades of software engineering wisdom. Instead, Yen champions observability as the essential paradigm for navigating this new era. Building on established trends like CI/CD, testing in production, and high-cardinality metadata, observability provides the necessary feedback loops to understand, debug, and iterate on LLM-backed applications in real-time, directly from production environments. This approach allows developers to embrace the inherent chaos of LLMs and user interaction, ensuring that software continues to work as expected, even when operating with unpredictable, emergent behaviors.

Yen's presentation is crucial for any developer, architect, or SRE grappling with the integration of generative AI into their systems. It reframes the challenge from trying to predict and control LLM behavior to effectively observing and reacting to it. By advocating for a systematic approach to tracking inputs, outputs, and contextual metadata using tools like traces and spans, she offers a pragmatic pathway to building robust, user-centric applications in a world increasingly powered by intelligent, yet opaque, black boxes.

Background

▶ Watch: Introduction: LLMs challenge traditional software development (0:00)

The advent of LLMs introduces a profound shift in software development paradigms, challenging long-held assumptions about system predictability and debuggability. Historically, software engineers have relied on structured methodologies for ensuring code correctness: unit tests to validate isolated components with predefined inputs, mocks to simulate external dependencies deterministically, and step-through debuggers to trace logical execution paths. However, LLMs fundamentally disrupt these practices. Their nondeterministic nature means the same input can yield different outputs, rendering deterministic mocks useless. The "full breadth of human expression" they are designed to process creates an "unbounded variation on inputs," making it impossible to define a representative set of test cases. Furthermore, their black-box operation, often involving complex neural networks, lacks simple logical paths for traditional debugging.

This inherent unpredictability extends beyond the code level to product development and release practices. Traditional alpha/beta programs often fail to capture the full range of user behavior and edge cases when dealing with LLM-backed functionality, as users are invited to interact in ways the developers may never anticipate. The speaker emphasizes that there's "no solid sense of correct" to fall back to, and no sterile environment can truly replicate the chaos of real-world user input.

However, Yen asserts that this doesn't mean abandoning all prior knowledge. Instead, she highlights several trends from the cloud-native era that laid the groundwork for addressing such chaos:

• CI/CD (Continuous Integration/Continuous Delivery): Enabled rapid iteration and frequent code shipments, fostering an embrace of change.
• Testing in Production: Acknowledged the limitations of pre-production environments and the necessity of validating behavior with real user input.
• High Cardinality Metadata / High Dimensionality Data: The ability to break down aggregate data by numerous possible related fields, crucial for understanding complex systems and detangling multiple contributing factors, whether for architectural complexity, business impact, or per-user experiences.
• SLOs (Service Level Objectives): Borrowed from SRE, these allowed leveraging existing alerting workflows but anchoring them around fuzzy, qualitative concepts like user experience or service quality.

These trends, collectively, have driven a distinct new approach to understanding software behavior, even before the LLM boom. This approach, centered on observability, provides a model for measuring, debugging, and improving unpredictable, qualitative experiences by comparing expected behavior against what is actually happening in production. The core message is that the challenges posed by LLMs—unpredictable behavior, chaotic user input, and the certainty that "one fix will break something else"—are precisely what observability is designed to address.

Key Findings

▶ Watch: Existing practices (CI/CD, testing in prod) are still relevant (3:29)

The central finding of this talk is that observability is not merely beneficial but absolutely essential for building and maintaining reliable software systems in the age of LLMs. Christine Yen argues that traditional testing and debugging methodologies are fundamentally inadequate for dealing with the nondeterministic, black-box, and unpredictable nature of LLMs. Instead, a proactive and continuous observation of systems in production is the only viable strategy.

A critical component of this strategy is the integration of evals (evaluation tools for LLMs) with observability. While observability focuses on understanding the chaos of production, evals provide a codified definition of "good" and "bad" behavior for LLM outputs, allowing for more flexible definitions of success or failure. The key insight is that these two practices form a powerful feedback loop: observations from production (via observability) reveal how users are interacting with the LLM and where unexpected behaviors occur, which then informs the creation and refinement of new evals. These evals, in turn, help capture intended behavior and flag issues as prompts evolve. This iterative process of "release quickly, observe in the wild, learn, and pull learnings back into the codebase as evals" is presented as the cornerstone of developing robust LLM-backed applications.

Furthermore, Yen emphasizes that effective observability for LLMs hinges on systematically tracking inputs and outputs across the entire request lifecycle. This includes not only the raw user input and the final LLM output but also all intermediate steps: the construction of context, the specific prompt sent to the LLM, metadata like token usage, and any parsing or validation applied to the LLM's response. By capturing this rich, high-cardinality data, engineers can move beyond aggregate statistics to investigate individual outliers, understand the root causes of unexpected LLM behavior, and rapidly iterate towards better prompts and application logic. The talk explicitly states a preference against "siloed or specialized" LLM observability tools, advocating instead for extending existing observability workflows that align with general application logic, ensuring consistency and broader team adoption.

Technical Deep Dive

▶ Watch: Observability: comparing expected vs. actual in production (4:32)

Achieving robust observability for LLM-backed applications requires a detailed and systematic approach to instrumentation, focusing on capturing the complete lifecycle of an LLM interaction. Christine Yen outlines a method that extends existing observability patterns to encompass the unique characteristics of generative AI.

The core principle is to systematically track all inputs and outputs, along with critical contextual metadata, throughout the application's execution path. For a standard web application, this might involve instrumenting to capture HTTP request arguments, application metadata, and return values. When integrating a third-party black-box service, like a payment gateway, the focus shifts to recording what requests the application sends, from which part of the code, on whose behalf, the response time, success/failure status, and any other relevant metadata. This allows reasoning about how application logic and inputs impact the black box's outputs and, ultimately, the end-user experience.

Applying this to an LLM world, the technical strategy becomes more intricate due to the additional "boxes" in the workflow, but the underlying principles remain consistent. Yen explicitly recommends using traces for this purpose, specifically spans, to understand the intricate relationships between the overall end-user experience and its subcomponents. While acknowledging that structured logs can also be used, traces offer a superior way to visualize and correlate sequential and concurrent operations.

A well-instrumented LLM interaction trace should capture the following at a minimum:

1. End-User Experience Span: This top-level span encapsulates the entire user interaction, from their initial raw input to the final output returned.
2. Context Construction Span: Details how the application builds the contextual information that will accompany the user's raw input. This includes capturing the arguments used to construct the context and the time taken.
3. Prompt Generation Span: Records the final prompt that is ultimately passed to the LLM. This is crucial for debugging, as subtle variations in prompt wording can drastically alter LLM behavior.
4. LLM Call Span: This is the direct interaction with the LLM service (e.g., OpenAI, Anthropic). Within this span, it's vital to capture:

• The exact prompt sent.
• Useful metadata such as token usage (input and output tokens), which can have cost and performance implications.
• The raw response received from the LLM.
• Latency of the LLM call.

1. Output Parsing/Validation Span: After receiving the LLM's response, applications often parse, validate, or reformat the output before presenting it to the user. This span should capture the raw LLM output, the results of parsing/validation, and any transformations applied.

The overarching principle is that "criteria for decision-making should be captured in a span." This means any internal logic, conditional branching, or data enrichment that influences the LLM interaction or its subsequent processing should be recorded as distinct spans with relevant attributes. For example, if the application decides to use a different prompt template based on user role, that decision and the chosen template should be explicitly recorded.

This granular instrumentation allows engineers to connect high-level aggregate metrics (like overall latency or user satisfaction graphs) with specific, problematic individual requests. When an aggregate graph shows an outlier—say, a spike in hallucination rates—an engineer can drill down from that aggregate view to an individual trace. Within that trace, they can then examine the specific raw input, the constructed context, the exact prompt sent to the LLM, its token usage, and the subsequent parsing logic. This comprehensive context is indispensable for investigating and iterating towards better prompts and application behavior, especially given the many factors that can lead to an unsatisfactory LLM response. Yen explicitly shows how a "blue row" representing the LLM call span can be investigated by looking at all the preceding spans that built up the prompt context.

The speaker cautions against adopting siloed or specialized LLM observability tools that dictate what metrics to care about. Instead, she advocates for extending existing observability platforms and workflows that engineering teams already use for their overall application logic. This ensures that LLM-specific observability aligns with broader operational practices and reflects what "good looks like" for the organization's unique applications.

Demo / Proof of Concept

▶ Watch: Pairing observability with evals for feedback loops (6:10)

While Christine Yen's keynote did not feature a live, interactive demo of an LLM observability tool, she effectively illustrated the concepts using visual examples and architectural diagrams. The presentation included screenshots and conceptual diagrams that depicted how traces and spans could be structured to observe an LLM-backed application.

Specifically, she showed:

• An aggregate graph displaying metrics like latency and user satisfaction, typical of observability platforms.
• A detailed trace view, illustrating how individual spans are nested and sequenced to represent the various stages of an LLM interaction. This included spans for the end-user experience, context construction, prompt generation, the actual LLM call, and output parsing/validation.
• The ability to click from an aggregate anomaly to a specific outlier trace, then drill down into individual spans to inspect the exact inputs, outputs, and metadata at each stage of the LLM request.

These visual aids served as a powerful conceptual proof of concept, demonstrating how a well-instrumented observability system could provide the necessary visibility to debug and iterate on LLM-driven features. The emphasis was on the type of data to collect and how to visualize its relationships, rather than showcasing a specific tool in action.

Defensive Implications

▶ Watch: Setting up observability tooling for LLM workflows (7:58)

The insights from this talk provide critical guidance for defenders and engineering teams building and operating LLM-backed systems. The core defensive posture shifts from attempting to prevent all unpredictable behavior to effectively detecting, understanding, and rapidly responding to it.

1. Prioritize End-to-End Observability: The most crucial implication is to implement comprehensive, end-to-end observability across all LLM-integrated services. This means instrumenting not just the LLM API calls themselves, but every step of the user's request lifecycle: raw input, context generation, prompt construction, LLM interaction, output parsing, and the final response. Using traces and spans is essential for correlating these distributed operations and understanding their causal relationships.
2. Capture High-Cardinality Context: Defenders must move beyond basic metrics and capture rich, high-cardinality metadata within each span. This includes the full raw user input, the exact prompt sent to the LLM, the LLM's raw response, token usage, and any internal decision-making criteria. This granular data is vital for diagnosing issues like hallucinations, unexpected outputs, prompt injection attempts (if applicable to the system's design), or performance degradation.
3. Integrate Observability with Evals: Proactive defense involves integrating LLM evaluation (evals) into the feedback loop. Observability helps identify what is going wrong in production; evals help codify why it's wrong and prevent regressions. Teams should use production observations to inform and refine their eval suites, ensuring that they capture real-world edge cases and evolving definitions of "good" behavior. This allows for automated validation of prompt changes or model updates against known issues.
4. Embrace "Testing in Production": Given the inherent unpredictability of LLMs, traditional pre-production testing is insufficient. Defenders must adopt a "testing in production" mindset, using observability to continuously monitor real user interactions and quickly identify deviations from expected behavior. This enables rapid iteration and deployment of fixes based on live data, minimizing user impact.
5. Develop Rapid Iteration Capabilities: The talk underscores that LLM-backed systems will require continuous tuning. Defensive strategies should therefore focus on enabling fast, safe iteration. This means having robust CI/CD pipelines, feature flagging capabilities, and, crucially, an observability platform that can immediately highlight the impact of new deployments or prompt changes on user experience and system behavior.
6. Avoid Siloed LLM Tools: Yen advises against specialized, siloed observability tools for LLMs. Instead, integrate LLM observability into existing application observability platforms. This ensures a holistic view of the entire system, prevents operational silos, and leverages familiar workflows for engineering and on-call teams, making it easier to diagnose issues that span traditional application logic and LLM interactions.
7. Shift On-Call Responsibilities: The responsibility for LLM-backed services falls squarely on engineering teams. This implies that developers building these services must be part of on-call rotations and be equipped with the observability tools necessary to diagnose and resolve issues quickly. This fosters a sense of ownership and direct feedback from production back into development.

By adopting these defensive implications, organizations can move from a reactive stance against LLM unpredictability to a proactive one, building resilient, observable, and continuously improving AI-powered applications.

Key Takeaways

• LLMs break traditional testing and debugging: Their nondeterministic nature, black-box operation, and unbounded input variations render unit tests, mocks, and step-through debugging largely ineffective for ensuring correctness.
• Observability is paramount for LLMs: In the face of inherent chaos and unpredictability, continuous observation of LLM-backed systems in production is the only reliable way to understand behavior, debug issues, and ensure user satisfaction.
• Evals and observability form a crucial feedback loop: Observability reveals real-world LLM behavior, informing the creation and refinement of "evals" (LLM evaluation tools) that codify desired and undesired outcomes, leading to iterative prompt and application improvement.
• Systematic tracking of inputs, outputs, and context is key: Comprehensive instrumentation using traces and spans to capture raw user input, context construction, exact prompts, LLM responses, token usage, and output parsing is vital for deep investigation.
• Leverage existing observability practices: Rather than adopting siloed LLM-specific tools, extend current observability workflows (like those for high-cardinality data and SLOs) to encompass LLM interactions, ensuring a holistic view and leveraging familiar engineering practices.
• Embrace chaos and rapid iteration: The future of LLM development involves shipping quickly, observing results in the wild, and continuously learning and adapting based on real-time production data, with developers owning the end-user experience.

About the Speaker(s)

Christine Yen is the co-founder and CEO of Honeycomb, a company specializing in observability for complex, distributed systems. Her work and advocacy have significantly contributed to the industry's understanding of modern software debugging and operational excellence, particularly in cloud-native environments. She is a recognized thought leader in the observability space, often speaking at major conferences like KubeCon. Christine is also an author, having written an O'Reilly report and a book on related topics, reflecting her deep expertise in helping engineering teams understand and manage the behavior of their software in production.