Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity

Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity

Speakers: Ahmed Alaa, Thomas Hartvigsen, Niloufar Golchini, Shiladitya Dutta, Frances Dean, Inioluwa Raji, Travis Zack

Conference: ICML 2025

YouTube: https://slideslive.com/39044041

Overview

In a critical presentation at ICML 2025, Tom Hartvigsen, faculty at the University of Virginia, delivered a compelling position paper on behalf of a large collaborative team from Berkeley and UCSF. The talk, titled "Benchmarks Medical Should Large Prioritize Language Construct Model Validity," or simply "we should care more about what our benchmarks measure," addresses a fundamental challenge in the burgeoning field of medical large language models (LLMs). As the race to deploy LLMs in various high-stakes medical applications accelerates, the community faces significant hurdles in accurately measuring progress and ensuring the reliability of these models.

Hartvigsen and his co-authors argue that the current culture of benchmarking, largely inherited from traditional, narrow-task machine learning, is ill-suited for the general-purpose, open-ended nature of generative medical LLMs. While traditional benchmarks like ImageNet have successfully driven architectural advancements in computer vision, they posit that medical LLM evaluations often lack construct validity—the extent to which a test measures what it claims to measure. This deficiency leads to misleading performance claims, potentially flawed decision-making in model development and deployment, and ultimately, risks patient safety in a domain where errors carry severe consequences. The talk advocates for a paradigm shift, urging the ML community to learn from social science methodologies to design more meaningful and trustworthy benchmarks for medical AI.

The core of the presentation reveals an empirical evaluation demonstrating the disconnect between performance on popular medical LLM benchmarks and real-world clinical data. By comparing the accuracy and relative rankings of leading medical LLMs on a widely used benchmark, MedQA, against actual patient data from UCSF's electronic health records (EHR), the researchers expose a profound lack of correlation and predictive power. This work underscores the urgent need for a re-evaluation of how medical LLMs are assessed, emphasizing that without benchmarks that truly reflect clinical utility and medical knowledge, the pursuit of advanced AI in healthcare may inadvertently lead to the deployment of models that are effective in simulated environments but unreliable in practice.

Background

▶ Watch: Introduction: Prioritizing construct validity in medical LLM benchmarks (0:00)

The rapid advancements in large language models have spurred an intense focus on their application across numerous domains, with medicine emerging as a particularly high-stakes frontier. From assisting with diagnoses to streamlining administrative tasks, the potential for medical LLMs is vast. However, the critical challenge lies in accurately measuring progress and ensuring these models are fit for deployment in a context where errors can have severe, even life-threatening, consequences. This need for robust evaluation has historically been addressed through benchmarks, which serve as standardized tests to track model performance and drive innovation.

In traditional machine learning, particularly for specialized, narrow, and task-specific models, benchmarks have been immensely successful. A prime example is ImageNet, which provided a standardized dataset for image classification, driving significant architectural and algorithmic advances in computer vision for years. For such models, typically classifiers with standardized inputs (e.g., a chest X-ray) and standardized outputs (e.g., "pneumonia" or "no pneumonia"), evaluating performance through simple accuracy metrics is straightforward and effective. The "climbing the hill" analogy holds: improving on ImageNet often translates to better performance on other imaging setups.

However, the landscape for generative medical LLMs is fundamentally different. These models are designed to be general-purpose and open-ended, capable of handling highly flexible inputs (e.g., complex patient instructions, varied queries) and producing equally flexible, descriptive outputs. This inherent flexibility makes the system "highly underspecified" and renders traditional, accuracy-based evaluations incredibly challenging. The community is still grappling with how to measure meaningful progress for models that don't fit the neat input-output paradigms of their predecessors.

To understand the current state of medical LLM evaluation, the research team analyzed the 100 most cited medical language model papers published since 2020. Their findings revealed a fragmented and diverse benchmarking landscape. While approximately 40% of these papers utilized real-world hospital or public data, this data is often not perfectly representative of in-hospital clinical realities. Strikingly, about 60% of the papers relied on medical exams for evaluation. Initially surprising, as medical exams are a relatively small component of a doctor's overall journey, this trend makes sense from an evaluation perspective: exams, particularly multiple-choice question-answering problems, reduce output flexibility, allowing for easy, quantifiable accuracy measurement.

The most popular example of such an exam-derived benchmark is MedQA, which is based on the US Medical Licensing Exam (USMLE). MedQA consists of made-up vignettes describing patient histories and other properties, followed by multiple-choice questions. Models are prompted with these questions and provide an A-E answer, allowing for straightforward accuracy computation. The critical issue, as highlighted by Hartvigsen, is that MedQA was "designed for people" to assess their knowledge on their path to certification, with "nothing to do with large language models." Despite this fundamental mismatch, MedQA has become a de facto benchmark, driving "really, really bold claims" about LLM capabilities, such as models "outperforming doctors" or "passing the USMLE." These claims, the authors argue, are not true and are starting to influence crucial decision-making regarding which models are developed and deployed.

This predicament begs the question of whether improving models on such exams is actually meaningful. To address this, the authors draw a crucial parallel to social science methodologies. Social scientists have spent decades tackling the problem of measuring "latent constructs"—unobservable properties or capabilities of individuals, such as depression or empathy. They design "instruments" (tests or surveys) to gauge these constructs and then validate whether these instruments truly measure what they intend to. This concept is central to construct validity, which asks: "Does my benchmark measure what I think it measures?" By adopting this social science perspective, the ML community can rigorously assess whether medical LLM benchmarks are genuinely measuring relevant medical knowledge or merely an artifact of their design.

Key Findings

▶ Watch: Analysis of current medical LLM benchmarks: prevalence of exams (2:20)

The central contribution of this work is an empirical evaluation that exposes the fundamental limitations of current medical LLM benchmarks, particularly concerning their construct validity and relevance to real-world clinical practice. The researchers conducted a retrospective analysis, comparing model performance on the popular MedQA benchmark with their performance on a matched set of real-world patient data.

Their key findings can be summarized as follows:

1. Significant Accuracy Drop on Real-World Data: The study took seven top-performing medical LLMs from a MedQA leaderboard and reproduced their reported accuracy on the benchmark. Subsequently, these same models were tested on real patient cases derived from UCSF's extensive electronic health records (EHR) that closely matched the MedQA vignettes. The "unsurprising finding," as Hartvigsen noted, was a substantial drop in accuracy when models transitioned from the synthetic MedQA environment to real-world patient data. While the exact percentage drop isn't specified in the transcript, it's described as "a bunch," indicating a practically significant difference that would matter in clinical settings. This drop highlights a critical distribution shift between benchmark data and real-world scenarios.

1. Lack of Criterion Validity: Beyond raw accuracy, the study investigated criterion validity, which assesses how well a test (MedQA) predicts performance on an external criterion (real-world patient data). Specifically, they calculated the probability that a model would be correct on a real-world case, given that it was correct on the corresponding MedQA question. The results showed that these probabilities were "much lower" than what one would expect if MedQA were a reliable predictor of real-world performance. This indicates that excelling on MedQA does not strongly correlate with or predict a model's ability to perform accurately on actual patient data, further undermining its utility as a proxy for clinical competence.

1. Dramatic Shifts in Model Rankings (Lack of Construct Validity): Perhaps the most striking finding related directly to construct validity. The researchers observed how the relative ranking of models changed dramatically between the MedQA benchmark and the real-world EHR evaluation. For instance, GPT-4, which was ranked second on MedQA, dropped to sixth place when evaluated on actual patient data. Conversely, Llama 3, which initially ranked sixth on MedQA, soared to first place on the real-world task. This radical reordering of model performance clearly demonstrates that MedQA does not effectively measure the same underlying "medical knowledge ability" or "latent construct" that is required for real-world clinical tasks. The benchmark, therefore, fails to capture a model's true medical capability relevant to practice.

In summary, the empirical evidence strongly suggests that current popular medical LLM benchmarks like MedQA, while offering low friction and reproducibility, are fundamentally flawed. They fail to accurately reflect real-world performance, predict clinical utility, or reliably rank models based on their true medical knowledge. This calls for a significant rethinking of how medical LLMs are evaluated to ensure that advancements are genuinely meaningful for patient care.

Technical Deep Dive

▶ Watch: Critique of MedQA: Misleading claims from medical exam benchmarks (3:00)

The technical core of this position paper revolves around its empirical methodology, which critically evaluates the construct validity of medical LLM benchmarks. Drawing inspiration from social science, the authors conceptualize medical LLM evaluation as measuring a "latent construct"—a potential property of model capability, such as "communicating empathetically" or possessing "medical knowledge." The benchmark itself is viewed as an "instrument" designed to test whether a model (the "subject") possesses this capability.

The primary benchmark under scrutiny is MedQA, a widely adopted dataset derived from the US Medical Licensing Exam (USMLE). MedQA comprises a collection of vignettes, which are fabricated patient scenarios detailing medical history and other properties. Each vignette is accompanied by multiple-choice questions (MCQs), typically with A through E options. The inherent design of MedQA, with its standardized MCQ format, makes it easy to compute accuracy scores, which has contributed to its popularity. However, a critical characteristic is that these vignettes are "totally made up by some board" and explicitly "not designed for large language models."

To assess the real-world relevance of MedQA performance, the researchers utilized a large, real-world dataset: UCSF's electronic health records (EHR). This dataset contains comprehensive, de-identified patient information, representing actual clinical encounters. The innovative aspect of their experimental setup was the process of bridging these two distinct data sources. For each vignette and question in the MedQA benchmark, the team undertook a meticulous process to "find the patients that match the questions really, really well" within the vast UCSF EHR dataset. This involved identifying real patient cases whose medical histories and conditions closely mirrored those described in the synthetic MedQA vignettes. While the exact matching algorithm or criteria are not detailed in the transcript, the objective was to create a parallel evaluation scenario where the models faced clinically analogous situations, but with the nuances and complexities inherent in real patient data. The speaker acknowledges that despite best efforts, real patient histories would be "slightly different" from the made-up vignettes, reflecting the inherent variability of clinical practice.

The evaluation process involved:

1. Model Selection: Seven top-performing medical LLMs were selected from a public MedQA leaderboard. While the transcript specifically mentions GPT-4 and Llama 3 as examples, the study encompassed a broader set of leading models.
2. Baseline Reproduction: The reported accuracy of these models on the standard MedQA benchmark was first reproduced to establish a baseline.
3. Real-World Evaluation: Each model was then tasked with answering questions pertaining to the matched real-world patient data from the UCSF EHR. This involved presenting the models with the real patient information and posing questions analogous to those in MedQA.
4. Metric Analysis: The comparison was structured around three key aspects of validity:

• Direct Accuracy Comparison: A straightforward comparison of accuracy scores between MedQA and the real-world EHR cases. A significant drop in accuracy on the real-world data would indicate a lack of generalizability.
• Criterion Validity: This was assessed by calculating the conditional probability: P(correct on real-world | correct on MedQA). A low probability would imply that MedQA performance is not predictive of real-world success.
• Construct Validity (Ranking Analysis): This was the most critical aspect, examining how the relative rankings of the seven models shifted when moving from MedQA evaluation to the real-world EHR evaluation. A substantial change in rankings (e.g., GPT-4 moving from 2nd to 6th, Llama 3 from 6th to 1st) directly demonstrates that the two evaluation settings measure different underlying capabilities, thus indicating a failure of construct validity for MedQA. The implication is that MedQA does not reliably encode or reflect "medical knowledge ability" in a way that is relevant to actual clinical application.

The methodology, by systematically comparing synthetic benchmark performance against real-world clinical performance using a matched patient cohort, provides a robust empirical foundation for the argument that medical LLM benchmarks must prioritize construct validity. It moves beyond qualitative arguments to offer concrete evidence of the disconnect between current evaluation practices and the true demands of the medical domain.

Experimental Setup & Results

▶ Watch: Applying social science principles to evaluate LLM benchmarks (4:10)

The experimental setup designed by Hartvigsen and his collaborators aimed to empirically challenge the construct validity of widely used medical LLM benchmarks. The core idea was to compare model performance on a synthetic, exam-based benchmark with their performance on real-world patient data.

Datasets:

1. MedQA Benchmark: This dataset, derived from the US Medical Licensing Exam (USMLE), serves as the primary example of a popular, yet problematic, medical LLM benchmark. It consists of multiple-choice questions based on synthetic patient vignettes. These vignettes describe medical histories and conditions, crafted by a board, and are inherently "made up" rather than reflecting actual patient encounters. The format is standardized, allowing for easy, automated accuracy calculation.
2. UCSF's Electronic Health Records (EHR) Dataset: This represents the "real-world" ground truth. It is a massive dataset comprising de-identified patient records from the University of California, San Francisco. This rich source contains authentic medical histories, diagnoses, treatments, and outcomes, providing a high-fidelity representation of clinical reality.

Baselines and Models:

The study utilized "seven top models off of a medical language MedQA model leaderboard." While the specific names of all seven models are not enumerated in the transcript, GPT-4 and Llama 3 are explicitly mentioned as examples. These represent state-of-the-art large language models that have demonstrated strong performance on various benchmarks, including MedQA.

Experimental Procedure:

The crucial step involved meticulously matching MedQA vignettes to real patient cases within the UCSF EHR. For each MedQA question, the researchers identified patients in the EHR whose medical history and characteristics "match the questions really, really well." This matching process, though not detailed algorithmically, was central to creating a comparable real-world evaluation scenario. After matching, the models were:

1. Tested on the original MedQA questions to reproduce their leaderboard accuracy.
2. Tested on the questions derived from the matched real-world patient data, effectively asking the same clinical question but in the context of an authentic patient record.

Hardware:

The transcript does not specify the hardware used for model inference or evaluation, but it can be inferred that standard GPU or TPU infrastructure for LLM deployment would have been employed.

Metrics:

The evaluation focused on three primary metrics to assess different aspects of validity:

1. Absolute Accuracy: Direct comparison of the percentage of correct answers on MedQA versus the real-world EHR data.
2. Criterion Validity (Predictive Power): Measured as the conditional probability P(correct on real-world case | correct on MedQA question). This quantifies how well MedQA performance predicts real-world clinical accuracy.
3. Construct Validity (Ranking Stability): Assessed by comparing the relative rankings of the seven models between the MedQA task and the real-world EHR task. A significant change in rank order would indicate that the benchmarks are not measuring the same underlying construct.

Headline Numbers and Results:

• Accuracy Drop: The "unsurprising finding" was that model accuracy "drops a bunch" when evaluated on real-world data compared to MedQA. This drop was described as being "a big lot in practice," implying a significant reduction in performance that would have serious implications in a clinical setting.
• Low Predictive Power: The probabilities calculated for criterion validity (P(correct on real-world | correct on MedQA)) were found to be "much lower" than expected. This explicitly indicates that MedQA performance is "not that predictive of" or "not that correlated with" performance on actual patient cases.
• Dramatic Rank Changes: This was the most compelling result for construct validity.
• GPT-4, which held the second position on the MedQA leaderboard, plummeted to sixth place on the real-world EHR evaluation.
• Llama 3, initially ranked sixth on MedQA, ascended to the first position when faced with real patient data.

These results unequivocally demonstrate that, for the models tested and the specific benchmarks evaluated, MedQA performance is not indicative of real-world clinical capability. The radical shift in model rankings particularly highlights that MedQA does not effectively measure the "medical knowledge ability" required for practical application, thereby suffering from a critical lack of construct validity.

Practical Implications

▶ Watch: Defining construct validity: Does the benchmark measure its claims? (5:15)

The findings presented in this talk carry profound practical implications for anyone involved in the development, evaluation, or deployment of large language models in the medical domain. The core message is clear: reliance on current, easily quantifiable benchmarks like MedQA can lead to a dangerously misleading perception of model capabilities.

For practitioners and infra teams considering the integration of LLMs into clinical workflows, the research serves as a stark warning. Models that perform exceptionally well on synthetic medical exams may exhibit significant and unpredictable failures when confronted with the complexity, variability, and nuance of real patient data. Deploying such models based solely on benchmark performance could lead to misdiagnoses, inappropriate treatment recommendations, or other adverse patient outcomes. The "high-stakes setting" of medicine means that simply "deploying these things and seeing how well they work" is not an acceptable strategy. The observed accuracy drops and rank changes underscore the need for extreme caution and rigorous, clinically relevant validation before any LLM is integrated into patient care.

For model builders and deployers, the talk highlights critical tradeoffs. Current static benchmarks (like MedQA) offer low friction, making them easy to use, and they enhance reproducible research at scale. However, their demonstrated detachment from real-world use cases means they "risk being low relevance." Conversely, human evaluations by medical professionals offer high relevance, as they directly assess performance in a clinically meaningful context. But they come with "very high friction"—they are slow, expensive, and difficult to scale, especially given the scarcity of qualified medical evaluators. The challenge is to bridge this gap, designing evaluation methods that combine the scalability of benchmarks with the relevance of human expert assessment.

The authors advocate for a community-wide effort to rethink and redesign medical LLM benchmarks. One concrete suggestion is to "stick with multiple-choice questions but redesign these tests" with a focus on calibration. The goal is to ensure that "when you get better and better [on the benchmark] it also is indicative of real improvements on real tasks." This implies a need for benchmarks that are carefully constructed and validated against clinical ground truth, potentially involving more nuanced question types, richer contextual information, and a systematic process to ensure their alignment with actual medical practice.

The talk also explores several alternative views and their associated limitations:

• Static benchmarks are obsolete because medical knowledge changes too fast: While it's true that medical knowledge evolves rapidly (e.g., new randomized control trials), the authors suggest that some fundamental capabilities can still be robustly measured. The challenge is in designing benchmarks that can be updated or are robust to minor knowledge shifts.
• Dynamic human evaluation is the only path forward: While highly relevant, this approach sacrifices the advantages of scalable, reproducible benchmarks, making it difficult to track incremental progress across a wide range of models.
• Combining many different benchmarks is sufficient: This approach, while seemingly comprehensive, introduces the notoriously difficult problem of rank aggregation across disparate tasks, making it hard to derive a single, meaningful measure of overall model capability.

Ultimately, the practical implication is a call for a fundamental shift in mindset: moving beyond simple performance metrics to prioritize validity in all its forms, especially construct validity. This means asking, "Does my benchmark truly measure what I think it measures, and is that 'what' relevant to real-world medical practice?" Without this shift, the exciting promise of medical AI risks being undermined by a foundation of unreliable evaluation.

Key Takeaways

• Current Medical LLM Benchmarks Lack Construct Validity: Popular benchmarks like MedQA, designed for human medical licensing exams, do not accurately measure the underlying medical knowledge or capabilities relevant for real-world clinical applications by large language models.
• Performance Does Not Translate to Real-World Tasks: Models that perform exceptionally well on synthetic, multiple-choice medical exam benchmarks experience significant accuracy drops and exhibit poor predictive power when evaluated on authentic patient data from electronic health records.
• Model Rankings Are Unreliable: The relative ranking of leading medical LLMs can change drastically between synthetic benchmarks and real-world evaluations (e.g., GPT-4 dropping from 2nd to 6th, Llama 3 rising from 6th to 1st), indicating that these benchmarks measure different constructs.
• Need for Redesigned, Calibrated Benchmarks: The community must collaborate to design new benchmarks that are specifically engineered for LLMs, prioritize construct validity, and are calibrated to ensure that improvements on the benchmark are truly indicative of enhanced performance on real clinical tasks.
• Learning from Social Science is Crucial: The field of machine learning can benefit significantly from social science methodologies in measurement theory, particularly in understanding and assessing "latent constructs" and instrument validity.
• High-Stakes Medical Domain Demands Rigorous Evaluation: Given the critical nature of healthcare, it is imperative to move beyond "low friction" but "low relevance" benchmarks towards evaluation strategies that are robust, clinically meaningful, and truly reflect a model's fitness for deployment.

About the Speaker(s)

Thomas Hartvigsen is a faculty member at the University of Virginia. He presented this position paper on behalf of a large collaborative team that included researchers from the University of California, Berkeley, and the University of California, San Francisco (UCSF). His work, as demonstrated in this talk, focuses on critical evaluation methodologies for advanced AI systems, particularly within high-stakes domains like medicine. The comprehensive nature of the research presented highlights his expertise in bridging the gap between theoretical machine learning advancements and their practical, ethical implications in real-world applications.