LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models

Speakers: Parshin Shojaee, Ngoc Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa Doan, Chandan Reddy

Conference: ICML 2025

YouTube: https://slideslive.com/39044076

Overview

This talk, presented by Chandan Reddy from Virginia Tech on behalf of his PhD students Parshin Shojaee and Ngoc Hieu Nguyen and co-authors, introduces LLM-SRBench, a novel benchmark designed to push the boundaries of Large Language Models (LLMs) in the domain of scientific equation discovery, also known as symbolic regression. The central premise is to move beyond mere data fitting and towards leveraging the vast scientific knowledge embedded within LLMs to uncover new, explainable mathematical hypotheses that accurately describe observed phenomena. The talk highlights a critical limitation of existing benchmarks: their susceptibility to LLM memorization, which hinders true scientific discovery.

The motivation for LLM-SRBench stems from the recognition that while traditional data-driven symbolic regression methods have advanced, they largely operate as "black boxes," unable to incorporate rich scientific literature and domain expertise. LLMs, with their advanced reasoning capabilities and extensive knowledge base, offer a promising avenue to bridge this gap. However, existing benchmarks, often derived from well-known scientific laws like those of Feynman, are easily "solved" by LLMs through memorization rather than genuine inference or discovery. LLM-SRBench aims to address this by providing a challenging, non-memorizable set of problems that demand true scientific reasoning and hypothesis generation from LLMs, complete with a robust evaluation framework that emphasizes both data fidelity and the symbolic accuracy of the discovered equations, particularly in out-of-distribution scenarios.

This work is significant because it lays the groundwork for evaluating and developing LLMs that can act as true scientific assistants, capable of proposing novel, explainable hypotheses grounded in scientific principles, rather than simply rediscovering known facts. By introducing datasets that are inherently difficult to memorize and a sophisticated evaluation metric that correlates with generalization, LLM-SRBench sets a new standard for assessing the scientific discovery potential of advanced AI models. It fosters research into agentic LLM paradigms that iteratively refine hypotheses, offering a pathway towards accelerating the scientific process in various domains.

Background

▶ Watch: Introduction to LLM-SRBench and scientific equation discovery (0:00)

The pursuit of scientific understanding fundamentally revolves around the process of empirical observation followed by the formulation of hypotheses, often expressed as mathematical equations. As exemplified by Kepler's laws of planetary motion, handcrafted centuries ago based on meticulous astronomical data, these equations provide a rigorous and predictive framework for understanding the physical world. This endeavor, broadly termed equation discovery or symbolic regression (SR), seeks to find compact, interpretable mathematical expressions that best fit observed numerical data.

For several decades, symbolic regression has evolved from manual derivation to more compute-driven and data-driven approaches. Early methods often relied on genetic programming, an evolutionary algorithm that searches for mathematical expressions by iteratively combining and modifying symbolic building blocks. More recently, with the advent of powerful neural architectures, transformer-based methods have been applied, framing the problem as a sequence-to-sequence task where numerical data is mapped to equation strings. These data-driven techniques excel at fitting complex functions to data, treating the problem as a sophisticated form of regression.

However, a significant limitation of these purely data-driven approaches is their inability to leverage the vast repository of human scientific knowledge. Fields like physics, chemistry, and biology possess centuries of accumulated literature, textbooks, and research findings that encode fundamental principles, relationships, and domain-specific constraints. Traditional SR methods typically operate from first principles, attempting to derive equations solely from the provided numerical data, effectively ignoring this rich contextual information. This often leads to the discovery of "black-box" functions that, while accurate in data fitting, lack the interpretability, scientific grounding, or generalizability expected of a true scientific hypothesis. The challenge has thus been to incorporate this invaluable domain information into the equation discovery process, a task that remained largely elusive until the recent surge in capabilities of Large Language Models.

Large Language Models (LLMs) have emerged as a transformative technology, demonstrating remarkable abilities in scientific reasoning, problem-solving, and the assimilation of knowledge from diverse textual sources. Trained on colossal datasets encompassing scientific literature, research papers, and textbooks, LLMs possess an inherent understanding of scientific concepts, relationships, and mathematical structures. This makes them uniquely positioned to overcome the limitations of traditional symbolic regression. Instead of merely fitting data, LLMs can potentially synthesize new hypotheses by drawing upon their internal knowledge base, guided by the empirical data. This paradigm shift—from purely data-driven fitting to knowledge-informed discovery—promises to accelerate scientific advancement by enabling AI systems to build upon existing principles, much like human scientists do, rather than starting from scratch.

Key Findings

▶ Watch: How LLMs leverage scientific knowledge for discovery (2:50)

The core contribution of this work is the introduction of LLM-SRBench, a novel and challenging benchmark specifically designed to evaluate the scientific equation discovery capabilities of Large Language Models, addressing critical limitations of existing datasets. The primary findings revolve around its construction, evaluation methodology, and the insights gained from benchmarking current LLMs:

1. Addressing Memorization: LLM-SRBench directly tackles the pervasive issue of memorization in LLMs when applied to symbolic regression. Unlike previous benchmarks (e.g., the Feynman benchmark) which are easily "solved" by LLMs due to their exposure to well-known scientific laws during pre-training, LLM-SRBench comprises problems that require genuine reasoning and discovery. This is achieved through two distinct variants:

• LSR-Transform: This variant takes established scientific laws (e.g., Feynman equations) and applies symbolic transformations (e.g., rearranging variables, changing the target). For instance, transforming E = mc² into m = E/c² creates a problem that is semantically related but not directly memorized by LLMs, forcing them to derive the relationship.
• LSR-Synth: This variant introduces entirely novel synthetic equations derived from real-world scientific domains (chemistry, biology, physics, materials science). These equations incorporate complex interaction terms and components that have never existed in standard textbooks or scientific literature, thus ensuring that LLMs cannot simply recall them. Human expert validation confirms the feasibility and scientific relevance of these synthetic equations.

1. Comprehensive Dataset: LLM-SRBench provides a rich collection of 239 datasets derived from these two variants, covering a wide array of scientific domains. This extensive dataset serves as a robust testbed for evaluating the generalizability and domain adaptability of LLM-based equation discovery methods.

1. Robust Evaluation Testbed with LLM as Judge: A significant innovation is the proposed LLM as judge evaluation mechanism, leveraging powerful models like GPT-4o. This judge model is tasked with comparing the generated equation (represented as code or an expression tree) against the ground truth, assessing their mathematical equivalence and similarity. This overcomes limitations of traditional metrics like exact match or expression tree distance, which can be overly strict or fail to capture semantic equivalence. The LLM judge demonstrates a remarkable 95% correlation with human agreement, validating its effectiveness as a reliable and nuanced evaluator.

1. Challenging Benchmark for Current LLMs: Initial benchmarking of four state-of-the-art LLM-based symbolic regression methods (ranging from Llama 3 [8B parameters] to GPT-4o) reveals that LLM-SRBench is indeed highly challenging. The maximum symbolic accuracy observed across all methods is approximately 30%. This low performance, especially compared to the near-perfect scores on memorizable benchmarks, underscores the difficulty of the new problems and highlights the significant room for improvement in current LLM reasoning and discovery capabilities.

1. Correlation with Out-of-Distribution (OOD) Generalization: The study identifies a strong and crucial correlation between the symbolic accuracy (as measured by the LLM judge) and Out-Of-Distribution (OOD) generalization. This is a pivotal finding for scientific discovery: an equation that is symbolically accurate and well-formed is more likely to generalize to unseen data outside the training range. This emphasizes that the quality of the symbolic representation is not merely an aesthetic preference but a critical indicator of a hypothesis's scientific validity and predictive power.

In summary, LLM-SRBench represents a significant leap forward in evaluating AI's potential for scientific discovery. It provides a non-memorizable, diverse, and challenging suite of problems, coupled with an advanced evaluation methodology that prioritizes explainability, symbolic accuracy, and OOD generalization, thereby setting a new agenda for research in LLM-driven scientific exploration.

Technical Deep Dive

▶ Watch: Iterative workflow of LLM-driven hypothesis generation (3:50)

The technical core of LLM-SRBench and the LLM-driven equation discovery paradigm lies in its iterative, agentic workflow and the sophisticated construction of its benchmark datasets and evaluation metrics.

Agentic LLM Workflow for Equation Discovery

The proposed methodology for LLM-driven scientific equation discovery operates as an agentic paradigm, where the LLM acts as an expert agent, iteratively refining hypotheses based on feedback. The workflow unfolds as follows:

1. Prompting and Context Provision: The process begins by prompting the LLM to act as an expert in a specific scientific domain, explicitly stating the equation discovery task. Crucially, the LLM is provided with scientific background, including a problem statement, definitions of variables, and their descriptive context. This domain knowledge is vital for guiding the LLM's reasoning. Additionally, numerical data collected from experiments is supplied.

1. Hypothesis Generation: Based on the instructions, scientific context, and data, the LLM proposes a hypothesis. This hypothesis takes the form of a mathematical equation, which can be represented in various modalities: a symbolic expression (e.g., y = a*x^2 + b*x + c), an expression tree (a parse tree representing the equation's structure), or a piece of executable code (e.g., a Python function that computes y given x). The LLM leverages its code generation capabilities for this step.

1. Self-Verification and External Evaluation: Before external evaluation, the LLM can perform self-verification to check for basic correctness, such as code compilation or syntactic validity of the equation. Subsequently, an external evaluator tool is employed. This tool takes the generated code representing the hypothesis and executes it against the provided numerical data. The evaluation mechanism assesses multiple criteria:

• Data Fitting Criteria: How well the equation fits the observed data (e.g., using metrics like Mean Squared Error).
• Explainability: How simple and interpretable the equation is, often favoring parsimonious models.
• Symbolic Accuracy: How closely the generated equation matches the true underlying scientific law in its symbolic form.

1. Iterative Refinement and Memory: This is where the "agentic" nature becomes prominent. The evaluation results provide feedback to the LLM. Promising candidates (hypotheses that performed well on the evaluation metrics) are saved and can be used as in-context examples for subsequent iterations. Less promising candidates are discarded. This iterative cycle continues over potentially hundreds or thousands of attempts, allowing the LLM to learn from its past proposals and progressively optimize for both data fidelity and the scientific soundness/explainability of the equation. This process mirrors Darwinian "survival of the fittest," where the LLM plans, generates, verifies, evaluates, and refines.

LLM-SRBench Dataset Construction

The benchmark is meticulously constructed to ensure non-memorability and scientific relevance, comprising two main variants:

1. LSR-Transform:

• Source: This variant begins with well-known scientific equations, primarily from the Feynman benchmark, which are commonly found in textbooks and scientific literature.
• Transformation: Instead of presenting the original equation, a symbolic transformation is applied. For example, if the original equation is E = m * c^2, a transformed problem might ask for m given E and c, expecting m = E / c^2. This transformation changes the target variable and rearranges the equation.
• Challenge: LLMs, while familiar with the original form, often do not have these specific transformed representations explicitly stored in their knowledge base. This forces the LLM to perform mathematical reasoning and symbolic manipulation rather than simple recall.
• Process: The process involves selecting a pivot variable, applying symbolic transformations, checking for solvability, sampling a dataset based on the transformed equation, and reformulating the problem statement for the LLM.

1. LSR-Synth:

• Source: This variant focuses on generating novel synthetic equations that are grounded in real-world scientific domains but have not been published or widely disseminated.
• Novelty: These equations are designed to incorporate complex interaction terms and components that are typically discovered through advanced data-driven approaches or specific experimental observations, not readily available in general scientific literature or textbooks.
• Domains: Equations are constructed for diverse fields such as chemistry, biology, physics, and materials science, ensuring broad applicability.
• Validation: The synthetic equations are subject to human expert validation to confirm their feasibility, scientific plausibility, and representativeness of actual scientific phenomena.
• Dataset Sampling: Once validated, datasets are sampled from these novel equations across various ranges and conditions.

Evaluation Metrics and LLM as Judge

A crucial aspect of LLM-SRBench is its robust evaluation methodology, particularly the innovative LLM as judge metric:

1. Traditional Metrics:

• Data Fidelity: Measured by metrics like Normalized Mean Squared Error (NMSE) and accuracy, quantifying how well the predicted values from the discovered equation match the true data points.
• Out-of-Distribution (OOD) Testing: Emphasized as critical for scientific hypothesis validation. An equation should not only fit the training data but also generalize well to data points outside the observed range, indicating true predictive power.
• Symbolic Accuracy: Historically measured by exact match (equation strings must be identical) or expression tree distance (structural similarity of parse trees). These metrics can be overly rigid, failing to account for mathematically equivalent but syntactically different expressions (e.g., x + y vs y + x).

1. LLM as Judge (GPT-4o):

• Concept: To overcome the limitations of traditional symbolic accuracy metrics, LLM-SRBench proposes using a powerful LLM (specifically GPT-4o) as an intelligent judge.
• Functionality: The LLM judge receives both the generated hypothesis (as code or an expression tree) and the ground truth equation. It then evaluates the mathematical equivalence and similarity between the two, considering different forms of modalities.
• Advantages: This approach allows for a more nuanced assessment of symbolic correctness, recognizing equivalent expressions despite syntactic variations. It leverages the LLM's deep understanding of mathematics and programming logic.
• Validation: The LLM judge's evaluations demonstrate a high 95% correlation with human agreement, establishing its reliability as a surrogate for human expert judgment in comparing complex mathematical expressions.
• Importance: This metric is not just about semantic similarity; it has been shown to correlate strongly with OOD generalization. A high symbolic accuracy score from the LLM judge implies an equation that is more trustworthy for predicting outcomes in unseen scientific contexts, making it a powerful indicator of a truly robust scientific hypothesis.

By combining an agentic LLM workflow with carefully constructed non-memorizable datasets and a sophisticated, human-validated LLM-as-judge evaluation, LLM-SRBench provides a comprehensive and challenging framework for advancing AI-driven scientific discovery.

Experimental Setup & Results

▶ Watch: LLM's agentic paradigm for planning and optimization (5:00)

The experimental setup for LLM-SRBench focused on benchmarking existing LLM-based symbolic regression methods against the newly created challenging datasets and evaluating them with the proposed metrics.

Datasets and Domains

The benchmark utilizes its two primary dataset variants:

• LSR-Transform: Derived from transformed Feynman equations, designed to prevent memorization by LLMs.
• LSR-Synth: Comprising novel synthetic equations with interaction terms, constructed from real-world scientific principles across multiple domains.

In total, 239 distinct datasets were generated, spanning a wide range of scientific disciplines, including:

• Chemistry
• Biology
• Physics
• Material Science

Each dataset includes numerical observations, descriptions of variables, and the problem statement, designed to be fed into an LLM-based agent. Importantly, some datasets also incorporate noise in the observed data, similar to the original SRBench, though the primary results highlighted in the talk focus on cleaner data to isolate the symbolic discovery challenge.

Baselines and Models Tested

The study evaluated the performance of four different LLM-based methods that were published in the preceding year for equation discovery. While specific method names were not detailed in the presentation, these represent the current state-of-the-art in applying LLMs to symbolic regression. The underlying LLMs used for these methods ranged in sophistication:

• Starting from standard open-source models like Llama 3 (8 billion parameters).
• Extending to more powerful, closed-source models such as GPT-4o.

This range allowed for an assessment of how model scale and general intelligence impact performance on the novel benchmark.

Metrics

The primary metrics used for evaluation were:

• Symbolic Accuracy: This is the headline metric, reflecting how well the LLM recovers the exact symbolic form of the ground truth equation. Crucially, this was assessed using the LLM as judge (GPT-4o), which provides a more nuanced evaluation of mathematical equivalence than strict string matching.
• Fidelity Metrics: These include Normalized Mean Squared Error (NMSE) and other accuracy measures, quantifying the goodness of fit of the discovered equation to the numerical data.
• Out-of-Distribution (OOD) Generalization: This metric assesses the equation's performance on data points outside the range observed during the discovery process, a critical indicator of a true scientific hypothesis. The talk highlighted a strong correlation between symbolic accuracy and OOD generalization.

Headline Results

The core finding regarding the performance of current LLMs on LLM-SRBench is striking:

• The maximum symbolic accuracy observed across all tested methods and models was approximately 30%. This indicates a significant drop in performance compared to benchmarks susceptible to memorization (e.g., Feynman benchmark), where LLMs can achieve much higher, often near-perfect, scores.
• For instance, on the original Feynman benchmark, models like Llama 3 (8B) could recover equations with very low numerical and symbolic error within a few attempts, suggesting memorization. In stark contrast, even powerful models like GPT-4o struggled to exceed 30% symbolic accuracy on LLM-SRBench.

Comparative Analysis

• LSR-Transform vs. Feynman: When comparing equations of similar complexity, the symbolic accuracy of reconstructing equations in the LSR-Transform space was significantly lower and more challenging than in the original Feynman space. This empirically validates the benchmark's design goal of preventing memorization.
• Correlation of Symbolic Accuracy and OOD Generalization: A strong positive correlation was observed between the symbolic accuracy (measured by the LLM judge) and OOD generalization. This reinforces the value of the LLM as judge metric, as it effectively identifies equations that are not only symbolically correct but also robust and generalizable, a key characteristic of scientifically meaningful hypotheses.
• Comparison with Non-LLM Methods (PySR): In the Q&A session, it was noted that purely data-driven symbolic regression packages like PySR (Python Symbolic Regression) were also tested. While PySR might achieve slightly better fidelity metrics for some datasets by fitting the data very closely, it often fails to provide explainable or domain-driven equations. The LLM-based agentic paradigms, despite their lower symbolic accuracy on the new challenging benchmark, still hold a distinct advantage when the goal is to obtain an explainable, domain-grounded symbolic representation, which is paramount in scientific discovery. If the objective is solely data fitting without regard for equation structure or interpretability, other methods might be competitive or even superior in fidelity.

The results clearly establish LLM-SRBench as a "fertile ground for future research," demonstrating that current LLMs, despite their impressive capabilities, are still far from mastering genuine scientific discovery in non-memorized contexts. The benchmark effectively highlights the gap between recalling information and truly reasoning to formulate novel, explainable scientific hypotheses.

Practical Implications

▶ Watch: Problem with old, saturated symbolic regression benchmarks (6:00)

LLM-SRBench carries profound practical implications for practitioners, infrastructure teams, model builders, and deployers working at the intersection of AI and scientific research.

For Practitioners and Model Builders:

1. New Frontier for LLMs in Science: This benchmark signals a significant shift in how LLMs are evaluated for scientific applications. It moves beyond simple question-answering or text generation to genuine hypothesis formulation. For model builders, this means developing LLMs that can perform deeper scientific reasoning, symbolic manipulation, and integrate domain knowledge more effectively than current models. The low symbolic accuracy (30%) on LLM-SRBench indicates that there's a vast open research area for improving LLM architectures and training methodologies for scientific discovery.

1. Emphasis on Explainability and Symbolic Representation: The benchmark inherently prioritizes the discovery of explainable mathematical equations over opaque black-box models. For practitioners in scientific domains (e.g., drug discovery, materials science, climate modeling), this is crucial. An interpretable equation allows for deeper scientific insight, hypothesis testing, and the ability to control and predict phenomena, which is often more valuable than a slightly better fit from an uninterpretable model. Model builders should focus on generating robust symbolic outputs rather than just optimizing for numerical accuracy.

1. Importance of Out-of-Distribution (OOD) Generalization: The strong correlation between symbolic accuracy and OOD generalization is a critical takeaway. In scientific contexts, a hypothesis is only truly valuable if it generalizes to unseen conditions. This implies that AI systems for scientific discovery must be evaluated not just on their ability to fit observed data but on their predictive power beyond the training distribution. This informs dataset design, validation strategies, and the objective functions used during model training.

1. LLM as a Robust Evaluation Tool: The "LLM as judge" concept offers a powerful and scalable way to evaluate the symbolic correctness of generated equations. This can save significant human effort in validating complex mathematical expressions. Infrastructure teams can integrate such LLM-based evaluators into their CI/CD pipelines for scientific model development, ensuring that generated hypotheses are mathematically sound and equivalent to ground truth, even with syntactic variations.

For Infrastructure Teams and Deployers:

1. Agentic Paradigms for Scientific Workflows: The iterative, agentic LLM workflow demonstrated by LLM-SRBench provides a blueprint for building autonomous scientific discovery agents. Infra teams will need to design systems that support such iterative processes, including robust tool-use capabilities for LLMs (e.g., external evaluators, code execution environments), efficient memory management for storing promising hypotheses, and mechanisms for handling hundreds or thousands of LLM interactions.

1. Computational Demands: Running iterative LLM-based agents, especially with powerful models like GPT-4o, can be computationally intensive. Deployers must consider the resource requirements (GPU/TPU, memory) for such workflows and potentially optimize for cost-effective inference strategies or explore smaller, specialized LLMs for certain sub-tasks.

1. Data Management for Scientific Benchmarks: The provision of 239 datasets across diverse scientific domains highlights the need for robust data management and versioning systems. Deployers and researchers will benefit from well-structured, accessible data repositories for benchmarking and further research.

Tradeoffs and Limitations:

1. Performance vs. Explainability: There's an inherent tradeoff. Purely data-driven methods (e.g., PySR) might achieve higher data fidelity for certain problems but yield less interpretable equations. LLMs, while aiming for explainable equations, currently show lower symbolic accuracy on novel discovery tasks. The choice depends on the specific scientific goal: is a perfect fit more important, or is a slightly less perfect but highly interpretable equation preferred?

1. LLM Capability Gap: The 30% symbolic accuracy on LLM-SRBench clearly shows that current LLMs, despite their vast knowledge, struggle with genuine creative scientific discovery. They are prone to memorization and lack the robust symbolic reasoning and mathematical intuition needed for novel hypothesis generation. This indicates that LLMs need significant advancements in their internal reasoning mechanisms, perhaps through integration with symbolic AI or more advanced mathematical solvers.

1. Cost and Accessibility of Powerful LLMs: Using models like GPT-4o for evaluation or as the primary discovery agent can be costly and may not be accessible to all research groups. Future work might involve developing more efficient or open-source LLM judges and discovery agents.

In essence, LLM-SRBench pushes the AI community towards building more sophisticated, scientifically grounded LLMs that can genuinely contribute to the discovery process. It provides a clear roadmap for what needs to be improved in LLM capabilities and how such systems should be evaluated for real-world scientific impact.

Key Takeaways

• LLM-SRBench is a novel benchmark designed to evaluate Large Language Models (LLMs) for scientific equation discovery (symbolic regression), specifically addressing the pervasive issue of LLM memorization.
• The benchmark comprises two challenging variants: LSR-Transform (transformed well-known equations like Feynman's) and LSR-Synth (novel synthetic equations with interaction terms from various scientific domains like chemistry, biology, physics, and materials science), totaling 239 datasets.
• A significant innovation is the "LLM as judge" evaluation metric, utilizing powerful models like GPT-4o to assess the mathematical equivalence and similarity of generated equations, demonstrating a high 95% correlation with human expert agreement.
• Current state-of-the-art LLM-based symbolic regression methods achieve only a maximum of approximately 30% symbolic accuracy on LLM-SRBench, highlighting the benchmark's difficulty and the substantial room for improvement in LLM reasoning for novel scientific discovery.
• The study reveals a strong and crucial correlation between symbolic accuracy (as measured by the LLM judge) and Out-Of-Distribution (OOD) generalization, underscoring the importance of symbolically correct and explainable equations for scientific validity.
• The benchmark emphasizes an agentic LLM paradigm that iteratively refines hypotheses, prioritizing scientific discovery, explainability, and OOD generalization over mere data fitting, and provides both the datasets and codebase for public access.

About the Speaker(s)

The talk was presented by Chandan Reddy, a Professor of Computer Science at Virginia Tech. He delivered the presentation on behalf of the primary contributors, Parshin Shojaee and Ngoc Hieu Nguyen, who are PhD students and were unable to attend due to visa issues. The research also involved contributions from co-authors Kazem Meidani, Amir Barati Farimani, and Khoa Doan. Professor Reddy's role as a computer science professor at Virginia Tech indicates his expertise in AI/ML and his leadership in guiding advanced research in this domain.