Calibration and Bias in Algorithms, Data, and Models: a tutorial on metrics and plots for measuring calibration, bias, fairness, reliability, and robustness: Calibration and Bias in Algorithms, Data, and Models: a tutorial on metrics and plots for measuring calibration, bias, fairness, reliability, and robustness

Calibration and Bias in Algorithms, Data, and Models: a tutorial on metrics and plots for measuring calibration, bias, fairness, reliability, and robustness

Speakers: Mark Tygert

Conference: ICML 2025

YouTube: https://slideslive.com/39043826

Overview

In this comprehensive tutorial at ICML 2025, Mark Tygert delivers a critical examination of widely used methods for measuring calibration, bias, fairness, reliability, and robustness in machine learning. Tygert, drawing from his extensive background in applied mathematics, statistics, and physics, argues that many standard practices in AI/ML, particularly those involving binning or bucketing, are fundamentally flawed and have been abandoned by other scientific fields decades ago. The talk aims to equip the audience with statistically rigorous alternative methods, primarily rooted in classical cumulative statistics, which offer superior accuracy and interpretability.

The tutorial goes beyond mere criticism, providing a detailed technical deep dive into these robust alternatives, explaining their mathematical foundations, and demonstrating their practical application. Tygert emphasizes that these methods are not only relevant for traditional calibration of probabilistic predictions but also generalize effectively to complex problems like measuring fairness across subpopulations while controlling for multiple confounding variables, and handling weighted datasets common in large-scale industrial settings. By highlighting the pitfalls of current approaches and championing historically proven techniques, Tygert challenges the ML community to adopt more reliable measurement paradigms.

This presentation is crucial for anyone involved in building, evaluating, or deploying machine learning models, especially those concerned with the trustworthiness and ethical implications of AI systems. It offers a clear roadmap for moving beyond misleading metrics towards assessments that are both scientifically sound and practically actionable, ultimately fostering more reliable and fair AI.

Background

▶ Watch: General scope: calibration, bias, fairness, reliability, robustness (0:00)

The pervasive integration of machine learning into critical domains necessitates robust methods for evaluating model performance beyond simple accuracy. Two key areas of concern are calibration and fairness. Calibration refers to the consistency between a model's predicted probabilities (scores) and the actual observed outcomes (responses). For instance, if a model predicts a 70% chance of an event, that event should occur approximately 70% of the time among all instances where the model made such a prediction. Fairness, on the other hand, deals with ensuring that model predictions or outcomes do not disproportionately disadvantage specific demographic or protected groups, often requiring comparisons between subpopulations while accounting for relevant confounding factors.

In the machine learning and artificial intelligence communities, the predominant methods for assessing calibration and, by extension, certain aspects of fairness, have historically relied on binning or bucketing. This involves dividing the range of predicted probabilities or scores into discrete bins and then comparing the average predicted probability within each bin to the average observed outcome. Common visualizations include reliability diagrams, and summary statistics like Empirical Calibration Error (ECE) or Integrated Calibration Index (ICI) are derived from these binned comparisons.

However, as Tygert forcefully highlights, these binning-based methods are fraught with severe limitations. He asserts that they are "widely criticized and known to have all kinds of unbelievable failure modes" and are, in fact, "asymptotically inconsistent." This means that even with infinitely large datasets, these methods do not converge to the true underlying calibration error. Statisticians, medical researchers, and physicists, according to Tygert, largely abandoned these techniques over a century ago in favor of more robust approaches. The fundamental problem lies in an inherent Heisenberg principle or tradeoff: increasing the number of bins to resolve finer variations inevitably amplifies noise, while widening bins to reduce noise sacrifices resolution. This unavoidable tradeoff makes binning-based metrics unreliable and susceptible to manipulation based on arbitrary binning choices, leading to misleading conclusions about model performance. The problem exists because the ML community, in some cases, has not fully integrated the statistical rigor developed in other fields for over a century.

Key Findings

▶ Watch: Introducing methods from other fields to resolve issues (2:50)

The central thesis of Mark Tygert's tutorial is a stark critique of prevalent machine learning practices for evaluating model calibration and bias, coupled with a strong recommendation for alternative, statistically sound methodologies. The key findings and contributions can be summarized as follows:

1. Fundamental Flaws of Binning-Based Metrics: Standard machine learning methods, such as reliability diagrams, Empirical Calibration Error (ECE), and Integrated Calibration Index (ICI), which rely on binning or bucketing predicted scores, are demonstrated to be fundamentally flawed. Tygert asserts these methods are "asymptotically inconsistent," meaning they fail to converge to the correct measure of calibration even with infinitely large datasets. This inconsistency stems from an unavoidable tradeoff between resolving fine variations and averaging away noise, often referred to as a "Heisenberg principle."
2. Superiority of Cumulative Aggregation Methods: Tygert advocates for cumulative aggregation methods, rooted in classical statistics (e.g., Kolmogorov-Smirnov, Kuiper's statistic), as a robust and mathematically sound alternative. These methods avoid the pitfalls of binning by analyzing the cumulative differences between responses and scores, effectively integrating information across the entire range of predictions without arbitrary discretization.
3. Connection to Stochastic Processes: The cumulative difference plots exhibit properties analogous to random walks and Brownian motion, allowing for the construction of statistically rigorous confidence intervals and facilitating a deeper understanding of the deviations.
4. Generalizability to Fairness and Bias Measurement: The cumulative methods extend seamlessly to measuring bias and fairness. Instead of comparing predictions to responses, they can compare the responses of a subpopulation to those of the full population (or between two subpopulations), conditional on matching scores or covariates, thus enabling a "like-with-like" comparison.
5. Handling Multiple Covariates with Space-Filling Curves: A significant contribution to fairness analysis is the introduction of space-filling curves (e.g., Peano or Hilbert curves). These mathematical constructs allow the mapping of multiple, high-dimensional covariates (e.g., age, income, education) into a single scalar value. This scalar can then be used with the cumulative methods to control for complex interactions and ensure fair comparisons.
6. Adaptability to Weighted Sampling and Deterministic Outputs: The proposed methods are flexible, generalizing to datasets with weighted samples (common in industrial settings) and offering strategies to avoid randomization when dealing with degenerate (repeated) scores, providing deterministic results if desired.
7. Practical Implications and Tooling: The talk underscores the practical implications for practitioners, urging a shift away from misleading metrics. It highlights that open-source software, such as MAPIE, has already begun incorporating these more robust cumulative methods, making them accessible to the broader ML community.

In essence, Tygert's key finding is that the ML community has a significant opportunity to improve the reliability and rigor of its model evaluation by adopting time-tested statistical methods, thereby avoiding the inherent limitations of current, widely-used approaches.

Technical Deep Dive

▶ Watch: Two approaches for calibration: cumulative differences vs. binning (3:20)

The core of Mark Tygert's tutorial lies in contrasting two distinct paradigms for assessing model calibration and bias: the widely used but flawed binning approaches, and the statistically robust cumulative aggregation methods.

Defining Calibration

At its most fundamental, calibration refers to the consistency between a model's predicted probabilities (or scores) and the actual outcomes. Tygert defines this rigorously: a prediction S (e.g., a 30% chance of snow) is perfectly calibrated if, across N independent instances where S was predicted, S% of the actual responses R are 1 (e.g., it snowed), and (1-S)% are 0 (it did not snow).

The notation used throughout the tutorial is S for the predicted probability or score (treated as deterministic, conditioned upon), and R for the response or outcome (a Bernoulli random variable, 0 or 1, with E[R] = S). For a set of N predictions, we have paired (S_k, R_k) observations for k = 1, ..., N. A crucial initial step for the cumulative methods is to reorder these pairs such that the scores S_k are non-decreasing. If scores are degenerate (i.e., S_i = S_j for i != j), a slight random perturbation can be applied to ensure strict inequality, though Tygert later shows how to avoid this randomization for deterministic results.

Cumulative Aggregation: The Robust Approach

Tygert champions cumulative aggregation methods as the superior alternative, drawing parallels to classical statistical tests like Kolmogorov-Smirnov and Kuiper's statistic. These methods construct a cumulative difference plot, C_k, defined as:

C_k = (1/N) * sum_{i=1 to k} (R_i - S_i)

where k ranges from 1 to N, and (S_i, R_i) are the sorted pairs. This plot visualizes the accumulated deviation of responses from scores as one traverses the sorted predictions. A perfectly calibrated model would ideally yield a C_k plot that stays very close to zero.

The theoretical elegance of this approach stems from its connection to random walks and Brownian motion. The incremental differences (R_i - S_i) have an expected value of zero if the model is perfectly calibrated. Summing these mean-zero increments over k forms a driftless random walk. In the limit of large N, this random walk converges in distribution to Brownian motion. This connection allows for the construction of statistically rigorous confidence bands (e.g., 4-sigma bands, corresponding to approximately a 95% confidence interval) around the zero line, providing a visual and quantitative measure of significant miscalibration.

From the C_k plot, summary statistics can be derived:

• Kolmogorov-Smirnov statistic: The maximum absolute deviation of C_k from zero, i.e., max_k |C_k|.
• Kuiper's statistic: The range of deviations, max_k C_k - min_k C_k, which Tygert notes has a more intuitive interpretation related to the total miscalibration over any interval.

These statistics provide single-number summaries that are robust and interpretable, unlike their binning-based counterparts.

Binned Responses: The Flawed Approach

In contrast, the widely adopted methods in ML rely on binning. Scores are partitioned into M bins, with N_j samples in the j-th bin. Within each bin, the average score S_tilde_j and average response R_tilde_j are computed.

• Reliability Diagram: This is a plot of R_tilde_j versus S_tilde_j. Perfect calibration is represented by the diagonal line Y=X.
• Empirical Calibration Error (ECE) / Integrated Calibration Index (ICI): These are summary statistics typically calculated as L1 or L2 norms of the deviations between R_tilde_j and S_tilde_j, often weighted by the bin's sample density: ECE = sum_j (N_j/N) * |R_tilde_j - S_tilde_j|.

Tygert's core critique here is that binning introduces an unavoidable tradeoff: decreasing bin width increases resolution but also increases noise, while increasing bin width reduces noise but sacrifices resolution. This is the "Heisenberg principle" of binning. He demonstrates that ECEs, by summing absolute values of deviations across bins, prevent noise from averaging away, making them asymptotically inconsistent. They converge to the wrong answer, and their values fluctuate wildly depending on arbitrary choices like the number of bins or binning strategy.

Generalizations

The power of cumulative methods extends far beyond basic calibration:

1. Fairness and Bias Measurement: To assess fairness, Tygert proposes comparing a subpopulation's responses (R_subpopulation) to the full population's responses (R_full_population), but crucially, conditional on matching scores or covariates. For example, one might compare apartment recommendations for individuals of different skin colors at the same income level. The cumulative difference C_k would then track (R_subpopulation_i - R_full_population_i) for matched individuals.
2. Controlling for Multiple Covariates: When dealing with multiple confounding variables (e.g., age, income, education) that need to be controlled for in fairness analysis, space-filling curves (like Peano or Hilbert curves) offer an elegant solution. These curves provide a continuous, one-dimensional mapping of a multi-dimensional covariate space. By mapping multiple covariates to a single scalar "score" via a space-filling curve, the problem is reduced to a one-dimensional comparison, allowing the direct application of cumulative methods. Tygert emphasizes that while understanding the curve's mapping might be complex for humans, computers can easily provide interactive visualizations to interpret specific points on the curve in the original multi-dimensional space.
3. Weighted Sampling: Many real-world datasets, especially in industrial settings, come with weights (e.g., from inverse propensity scoring). The cumulative formulas can be generalized by replacing N with the sum of weights and adjusting the k/N indexing to reflect cumulative weights.
4. Avoiding Randomization: For degenerate scores (repeated values), instead of randomly perturbing them, one can group identical scores and average their corresponding responses. This produces a deterministic graph with fewer points but preserves the integrity of the cumulative differences.

In summary, Tygert provides a robust, statistically grounded framework for evaluating ML models, advocating for a paradigm shift from simplistic, flawed binning methods to sophisticated cumulative statistical approaches adapted from classical mathematics.

Experimental Setup & Results

▶ Watch: Generalizations: controlling for multiple covariates, space-filling curves (5:00)

As a tutorial, this talk primarily focuses on theoretical exposition and illustrative examples rather than presenting novel experimental results from a specific setup. Mark Tygert uses carefully constructed synthetic data to demonstrate the fundamental flaws of traditional methods and the robustness of cumulative approaches.

For instance, when illustrating the problem with reliability diagrams, Tygert presents a synthetic example where the underlying data has a known calibration. He then shows multiple reliability diagrams generated from the exact same synthetic dataset but using different numbers of bins (e.g., 4, 8, 16, 32 bins). The "results" are strikingly divergent: the plots appear dramatically different, some suggesting good calibration, others poor, and the noise levels vary wildly. This visual demonstration powerfully illustrates the asymptotic inconsistency and sensitivity of reliability diagrams to arbitrary binning choices. He explicitly states, "You don't know which is noise and which is real," and "the same all same data all same method but they give different answers."

Similarly, when evaluating Empirical Calibration Error (ECE) metrics, Tygert plots the ECE value against the number of bins used. Ideally, this plot should be a flat line, indicating that the metric is stable regardless of binning. However, the synthetic example shows the ECE values fluctuating by "an order of magnitude or something," further reinforcing the metric's unreliability. He provocatively suggests that this variability allows practitioners to "fudge these classic statistics as much as you like" to get a desired answer, highlighting a critical ethical and scientific concern.

In contrast, while not explicitly showing "experimental results" for the cumulative methods, Tygert implicitly demonstrates their robustness by describing how they inherently avoid the binning tradeoff. The cumulative difference plots, being based on sorted individual data points rather than aggregated bins, are not subject to the same arbitrary choices and are presented as inherently stable and interpretable through their connection to Brownian motion and associated confidence bands.

Tygert asserts that these issues are not confined to synthetic examples, stating, "I'll show you on real data it's worse. I couldn't even devise a synthetic data set as bad as it is in normal practice." This underscores the real-world severity of relying on flawed binning-based metrics in machine learning.

Practical Implications

▶ Watch: Handling weighted samples for real-world large datasets (6:00)

Mark Tygert's tutorial carries significant practical implications for various stakeholders in the AI/ML ecosystem, urging a fundamental shift in how model calibration, bias, and fairness are assessed.

For Practitioners and Model Builders:

• Abandon Flawed Metrics: The most immediate implication is a call to discontinue reliance on binning-based methods such as reliability diagrams, Empirical Calibration Error (ECE), and Integrated Calibration Index (ICI). These metrics are demonstrated to be unreliable, asymptotically inconsistent, and susceptible to manipulation due to arbitrary binning choices.
• Adopt Cumulative Statistics: Practitioners should instead embrace cumulative aggregation methods derived from classical statistics, such as those related to Kolmogorov-Smirnov and Kuiper's statistic. These non-parametric methods provide robust, interpretable measures of miscalibration and deviation, offering clear visual insights (e.g., through cumulative difference plots with Brownian motion-based confidence bands) and statistically sound summary statistics.
• Enhanced Fairness Analysis: For measuring bias and fairness, the cumulative approach provides a powerful framework for comparing subpopulations conditional on covariates. This allows for "like-with-like" comparisons, addressing complex social issues more rigorously than simple average comparisons.
• Multi-Covariate Control: When dealing with multiple confounding variables, practitioners should explore space-filling curves (e.g., Hilbert curves) to map these high-dimensional covariates into a single scalar. This technique enables the application of one-dimensional cumulative methods to control for complex interactions, offering a sophisticated tool for robust fairness analysis. Interactive visualization tools can aid in interpreting these mappings.
• Handle Weighted Data: For datasets with weighted samples (common in large-scale industrial applications like those at Meta or Google), the generalized cumulative formulas seamlessly incorporate these weights, ensuring accurate analysis even with complex sampling designs.
• Leverage Open-Source Tools: The tutorial highlights that open-source software, such as MAPIE, is already incorporating these robust cumulative methods, making them accessible and easier to adopt for a wide range of ML projects.

For Infrastructure Teams and Deployers:

• Integrate Robust Monitoring: Infra teams responsible for model deployment and monitoring should update their evaluation pipelines to incorporate cumulative statistics. This ensures that models in production are continuously assessed using reliable metrics, providing a more accurate picture of their real-world performance and fairness.
• Support Interactive Analysis: Implementing interactive tools that allow users to explore cumulative difference plots and understand multi-dimensional mappings via space-filling curves would greatly enhance diagnostic capabilities and model governance.

Tradeoffs and Limitations:

• Complexity of Space-Filling Curves: While powerful, the conceptual understanding and practical implementation of space-filling curves for multi-covariate control can be initially challenging. However, Tygert assures that computational tools can make this transparent.
• Continuous vs. Discrete Data: The cumulative methods are best suited for continuous scores and covariates. For inherently discrete data, classical methods like chi-square tests might still be more appropriate, though Tygert advises against artificially discretizing continuous data.
• Randomization vs. Determinism: While random perturbations simplify certain aspects of cumulative methods, Tygert provides alternatives to achieve fully deterministic results, albeit with potentially fewer points visualized on the graph.
• Evolving Field: The field of multi-calibration, while touched upon, is an active area of research, indicating that even more sophisticated methods are on the horizon.

In essence, the practical implication is a call for greater statistical rigor in ML model evaluation. By adopting these proven methods, practitioners can build, deploy, and regulate AI systems with higher confidence in their calibration, reliability, and fairness, moving beyond potentially misleading metrics.

Key Takeaways

• Critique of Standard ML Metrics: Widely used machine learning metrics for calibration and bias, such as Empirical Calibration Error (ECE) and reliability diagrams, are fundamentally flawed, asymptotically inconsistent, and susceptible to manipulation due to arbitrary binning strategies. Other scientific fields abandoned these methods decades ago.
• Embrace Cumulative Statistics: Robust and statistically sound alternatives exist in cumulative aggregation methods (e.g., based on Kolmogorov-Smirnov and Kuiper's statistic). These non-parametric approaches analyze the entire distribution of scores and responses, avoiding the inherent tradeoffs and inconsistencies of binning.
• Connection to Stochastic Processes: Cumulative difference plots exhibit properties akin to random walks and Brownian motion, allowing for the construction of rigorous statistical confidence bands that provide clear, visual indicators of miscalibration.
• Generalizability for Fairness and Bias: These methods are highly generalizable, enabling rigorous measurement of fairness and bias by comparing subpopulations' responses conditional on matched scores or covariates, ensuring "like-with-like" comparisons.
• Multi-Covariate Control with Space-Filling Curves: For complex scenarios involving multiple confounding variables, space-filling curves (e.g., Hilbert curves) can map these high-dimensional covariates into a single scalar, allowing the application of one-dimensional cumulative methods for robust conditional analysis.
• Practical Adoption and Future Directions: Open-source libraries like MAPIE are beginning to incorporate these superior methods, making them accessible to practitioners. The field continues to evolve with ongoing research into areas like multi-calibration.

About the Speaker(s)

Mark Tygert is a distinguished researcher whose expertise spans applied mathematics, statistics, and physics. His diverse background, including a past faculty position at Yale Medical School, provides him with a unique interdisciplinary perspective on machine learning challenges. Currently, he works at Meta, where his research often involves complex data scenarios such as weighted sampling. Tygert's presentation clearly reflects his deep understanding of classical statistical rigor and his commitment to bringing these time-tested methods to the forefront of machine learning practice, challenging the community to adopt more robust and reliable evaluation techniques.