VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data

VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data

Speakers: Thomas Zeng, Shuibai Zhang, Shutong Wu, Christian Classen, Daewon Chae, Ethan Ewer, Minjae Lee, Heeju Kim, Wonjun Kang, Jackson Kunde, Ying Fan, Jungtaek Kim, HYUNG IL KOO, Kannan Ramchandran, Dimitris Papailiopoulos, Kangwook Lee

Conference: ICML 2025

YouTube: https://slideslive.com/39044028

Overview

In the rapidly evolving landscape of artificial intelligence, Large Language Models (LLMs) have demonstrated remarkable capabilities in understanding and generating human-like text. However, their inherent probabilistic nature often leads to non-deterministic outputs, meaning the same prompt can yield different, sometimes incorrect, answers. This variability poses significant challenges for deploying LLMs in applications requiring high accuracy and reliability, particularly in complex reasoning tasks. The talk by Thomas Zeng and his collaborators introduces VersaPRM, a novel Process Reward Model (PRM) designed to address these limitations by evaluating the correctness of LLM reasoning steps across multiple domains, moving beyond the current domain-specific confines of existing PRMs.

The core innovation of VersaPRM lies in its ability to generalize across diverse subject matters, such as math, law, philosophy, and biology, a significant leap from prior PRMs predominantly focused on mathematical or coding tasks. This generalization is achieved through a data-centric approach, leveraging synthetic reasoning data generated and annotated by LLMs themselves. By creating a diverse, large-scale dataset, the researchers effectively overcome the data scarcity and cost issues typically associated with high-quality, human-labeled reward model training data. VersaPRM thus represents a crucial step towards building more robust and universally applicable AI systems, enabling LLMs to provide not just answers, but verifiable, step-by-step reasoning that can be trusted across a broader spectrum of real-world applications.

The importance of VersaPRM cannot be overstated for the future of reliable AI. By providing fine-grained feedback on each step of an LLM's reasoning process, it offers a powerful mechanism to enhance the accuracy and trustworthiness of AI-generated responses. This is particularly critical in fields where erroneous outputs can have severe consequences. The open-sourcing of VersaPRM's data annotation pipeline, dataset, and model further underscores its significance, fostering a collaborative environment for continued research and development in multi-domain reward modeling. This work paves the way for a new generation of LLM-powered applications that are not only intelligent but also demonstrably logical and reliable across a wide range of human knowledge.

Background

▶ Watch: Introduction and LLM error problem (0:00)

The journey to VersaPRM begins with understanding the inherent challenges of Large Language Models (LLMs). While LLMs excel at generating coherent and contextually relevant text, their responses are often probabilistic. When prompted with a question, an LLM decodes a sequence of tokens, and due to the inherent randomness in this decoding process, multiple inferences can be sampled from the same query. This can lead to a distribution of potential answers, where the correct answer might not always be the most frequently generated one. For instance, a toy arithmetic example showed an LLM having only a 45% chance of generating the correct answer, even if that answer was still more likely than any other single incorrect response.

One initial attempt to leverage this distribution for improved accuracy is majority voting. The concept is simple: instead of relying on a single LLM response, sample multiple responses, tally the occurrences of each answer, and select the answer that appeared most frequently. This technique proves effective when the correct answer is indeed the most likely output from the LLM's distribution. However, its utility diminishes significantly when an incorrect answer dominates the LLM's output distribution, leading majority voting to consistently select the wrong answer. This highlights the need for an external mechanism to assess the plausibility or correctness of LLM-generated responses.

This need gave rise to the concept of Reward Models (RMs). In the literature, two primary types of RMs are discussed:

1. Outcome Reward Model (ORM): This model outputs a single scalar score, typically between zero and one, indicating the overall correctness of the final answer of an LLM response.
2. Process Reward Model (PRM): This is the central object of the VersaPRM paper. A PRM operates at a more granular level, mapping each reasoning step within an LLM's response to a value between zero and one. A score closer to one signifies a correct and logical step, while a score closer to zero indicates an incorrect or invalid step. This vector of step-level scores can then be aggregated (e.g., by taking the minimum coordinate value) to derive an overall plausibility score for the entire reasoning trace. Empirically, PRMs have been shown to work better than ORMs due to their fine-grained feedback, which is particularly useful in complex tasks requiring step-by-step validation.

The efficacy of PRMs has been demonstrated in specific domains. A seminal paper from OpenAI roughly two years prior showcased the power of their proprietary PRM, significantly improving performance on college-level math questions compared to both single LLM responses and majority voting. This established that PRMs could indeed enhance LLM accuracy in real-world scenarios.

However, a critical limitation emerged when attempting to apply existing open-source PRMs to domains beyond mathematics. Testing these models on the MMLU-Pro dataset, which comprises multiple-choice questions across diverse subjects like math, law, philosophy, and biology, revealed a stark disparity. While PRMs yielded substantial performance improvements (e.g., about 48% increase over majority vote) on the math subset, their impact on other domains like law, philosophy, and biology was negligible, often showing improvements that were an order of magnitude smaller. This observation forms the foundational problem VersaPRM seeks to address: the lack of multi-domain generalizability in existing PRMs, largely attributed to their training data being heavily skewed towards math and coding tasks. This domain specificity necessitated a new approach to PRM development, one that could span the breadth of human knowledge.

Key Findings

▶ Watch: Introducing the Process Reward Model (PRM) (2:10)

The research leading to VersaPRM unveiled several critical findings that not only diagnose the limitations of existing Process Reward Models (PRMs) but also propose a robust solution for achieving multi-domain generalization.

The first main finding is that existing open-source PRMs are severely limited in their applicability, primarily working only for specific domains, predominantly math and coding. While previous work, such as OpenAI's proprietary PRM, demonstrated impressive performance gains on math problems, this efficacy does not translate to other subject areas. Experiments on the MMLU-Pro dataset clearly showed that while PRMs significantly boosted performance on math questions (e.g., ~48% improvement over majority voting), their impact on domains like law, philosophy, and biology was negligible, often yielding improvements an order of magnitude smaller. This revealed a significant gap in the current PRM landscape, indicating that the promise of PRMs for enhancing Large Language Model (LLM) reliability was largely unfulfilled beyond narrow, specialized tasks.

The researchers then probed the underlying reason for this domain specificity. They considered two primary hypotheses: an "algorithmic or architectural problem" or a "data problem." While algorithmic issues could contribute, the analysis pointed strongly towards the latter. Existing PRMs are overwhelmingly trained on datasets derived from math and coding questions. Consequently, their strong performance on math tasks is largely an "in-distribution" success, whereas their failure to generalize to other domains is a predictable outcome of this data imbalance. This finding reframed the challenge from primarily an architectural one to a data-centric one, suggesting that a more diverse training dataset was the key to unlocking broader PRM utility.

Building on this insight, the second key finding is the successful development of VersaPRM, the first multi-domain PRM capable of working effectively across a wide array of subject domains. This was achieved by adopting a novel data-centric perspective centered on two core ideas:

1. Creation of a diverse dataset: The researchers prioritized building a training dataset that spans multiple subject domains, ensuring broad coverage of the distribution the PRM needs to learn. This approach posits that even if the underlying learning algorithm or model architecture is not perfectly optimized, a sufficiently diverse dataset can act as a "big enough hammer" to induce generalization.
2. Leveraging LLMs for synthetic data labeling: To overcome the prohibitive costs and time associated with manually annotating large volumes of high-quality, diverse data, VersaPRM employs LLMs to synthetically label the correctness of reasoning steps. This innovative use of LLMs for data generation and annotation at scale proved instrumental in creating the necessary training corpus without incurring massive expenses.

The experimental results confirm the success of this approach. When trained on the synthetically generated, multi-domain dataset, VersaPRM demonstrated significant performance improvements not only in math but also in law, philosophy, and biology, effectively fulfilling the dream of a PRM that generalizes beyond its initial training domain. This provides empirical evidence that the data-centric strategy, combined with LLM-powered synthetic data generation, is a viable and effective path towards building versatile and robust reward models.

Finally, a crucial contribution of this work is the open-sourcing of the entire multi-domain data annotation pipeline, the resulting dataset, and the VersaPRM model itself. This commitment to open science, reflected in the 20k+ downloads on Hugging Face, encourages further research and application of PRMs in domains beyond math and coding, accelerating the development of more general-purpose AI systems.

Technical Deep Dive

▶ Watch: How PRMs enhance majority voting (3:30)

The technical foundation of VersaPRM is rooted in the architecture and training methodology of Process Reward Models (PRMs), augmented by a novel data generation strategy to achieve multi-domain generalization. At its core, a PRM is designed to provide fine-grained feedback on the reasoning process of a Large Language Model (LLM). When an LLM generates a response to a user question, this response typically involves a series of intermediate reasoning steps leading to a final answer. A PRM takes both the user question and the LLM-generated response, then maps each individual reasoning step to a numerical value, typically between zero and one. A score closer to one indicates that the PRM deems the step correct and logically sound, while a score closer to zero suggests an incorrect or invalid step.

This vector of step-level scores is then converted into a single "plausibility score" for the entire response. While various aggregation methods exist, a simple and effective approach mentioned is to take the minimum coordinate of the PRM score vector. This means the overall plausibility of the reasoning trace is only as strong as its weakest link, a conservative but robust measure.

The utility of a PRM becomes evident when integrated with techniques like majority voting. In a scenario where an LLM might frequently generate an incorrect answer, traditional majority voting would fail. However, by first using the PRM to grade multiple LLM responses and then performing a weighted count based on these plausibility scores, the system can effectively downweight incorrect but frequently generated answers. For example, if an incorrect answer appears most often but receives low PRM scores, its effective count is reduced, allowing a less frequent but highly plausible correct answer to "win out" in the weighted tally. This modified majority voting scheme, powered by PRMs, significantly enhances the accuracy of LLM outputs.

The critical challenge addressed by VersaPRM is the observed domain specificity of existing PRMs. These PRMs are typically transformer-based models trained via supervised fine-tuning. The root cause of their limitation, as identified by the researchers, is a data problem. Prior PRMs were predominantly trained on datasets generated from math and coding questions. This created an inductive bias, making them highly effective for in-distribution math tasks but incapable of generalizing to out-of-distribution domains like law or philosophy. The existing literature, while rich in PRM research, has largely focused on these narrow domains or on preference-based tasks (e.g., writing good emails), rather than subject-domain generalization.

VersaPRM's technical solution hinges on a two-pronged data-centric strategy:

1. Diverse Dataset Creation: The first key idea is to construct a training dataset that is as diverse as possible, spanning multiple subject domains. The philosophy here is that broad data coverage can compensate for potential suboptimality in learning algorithms or model architectures, enabling the PRM to generalize effectively.
2. LLM-Powered Synthetic Labeling: To overcome the prohibitive cost and effort of manual annotation, especially for detailed step-by-step correctness labels, VersaPRM leverages LLMs to synthetically label the training data. This is a crucial innovation that enables the creation of large-scale, high-quality, multi-domain datasets at a fraction of the traditional cost.

The construction of this PRM dataset requires three key ingredients:

• Questions: Diverse questions are sourced from existing public datasets that cover a wide range of subjects.
• Reasoning Trace and Answer: For each question, an LLM is prompted to generate its reasoning trace and corresponding answer. This provides the raw material that the PRM will evaluate.
• Correctness Labels for Each Reasoning Step: This is the most challenging component to acquire manually. VersaPRM tackles this by employing an LLM to annotate the correctness of each individual step in the generated reasoning traces. While the talk "glosses over" the specific technicalities of this LLM-based annotation process, it implies a sophisticated prompting and validation mechanism to ensure high-quality synthetic labels. This could involve self-consistency checks, multiple LLM passes, or even using a stronger, more capable LLM for annotation than the one generating the initial responses.

Once these three ingredients are assembled, a comprehensive multi-domain dataset of questions, LLM-generated reasoning steps, and their corresponding synthetic correctness annotations is formed. The VersaPRM model itself is then created by fine-tuning an existing PRM architecture (likely a smaller transformer-based model, given its role as a reward model rather than a generative LLM) on this newly compiled multi-domain dataset. This fine-tuning process adapts the PRM's internal representations and scoring mechanisms to recognize logical consistency and correctness across a much broader range of subject matters, effectively instilling the "versatile" capability implied by its name. The success of this approach validates the hypothesis that the generalization bottleneck for PRMs was primarily a data problem, addressable through strategic and scalable synthetic data generation.

Experimental Setup & Results

▶ Watch: PRM performance across multiple domains (5:00)

The experimental evaluation of VersaPRM was designed to rigorously test its ability to generalize beyond math, directly addressing the identified limitations of existing Process Reward Models (PRMs). The core of the evaluation centered on the MMLU-Pro dataset, a well-established benchmark comprising multiple-choice questions across a diverse range of subject domains. This dataset includes categories such as math, law, philosophy, and biology, making it an ideal choice for assessing multi-domain performance.

The experimental setup involved comparing the performance of different approaches to enhancing Large Language Model (LLM) accuracy:

1. Single LLM Response: This served as the baseline, representing the raw performance of an LLM without any post-processing or reward model integration.
2. Naive Majority Voting: This baseline involved sampling multiple responses from the LLM for each question and selecting the answer that appeared most frequently. This demonstrates the upper bound of improvement achievable purely through statistical aggregation of LLM outputs.
3. Existing Open-Source PRMs: Several existing PRMs, primarily trained on math and coding datasets, were evaluated. These models were applied to LLM responses, and their scores were used for weighted majority voting. This established the performance ceiling for domain-specific PRMs when faced with out-of-distribution data.
4. VersaPRM: The proposed multi-domain PRM, trained on the synthetically generated diverse dataset, was then tested using the same weighted majority voting mechanism.

The key metric used for comparison was the "performance increase" that a given method provided on top of naive majority voting. This metric clearly highlights the incremental value added by PRMs.

The initial results for existing PRMs strikingly confirmed their domain specificity:

• Math Subset: On the math subset of MMLU-Pro, existing PRMs demonstrated significant performance improvements, comparable to previous findings. For example, a performance increase of approximately 48% over majority voting was observed. This reaffirmed that PRMs are highly effective when applied to domains similar to their training data.
• Other Domains (Law, Philosophy, Biology): In stark contrast, when existing PRMs were applied to non-math domains, the performance improvements were an order of magnitude smaller, often barely exceeding 0%. This unequivocally showed that these PRMs were "very unhelpful" beyond math, failing to provide any meaningful boost to LLM accuracy in diverse subject areas.

These results provided strong empirical evidence for the first main finding: existing PRMs indeed only work well for math (and implicitly, coding tasks), and their lack of generalization across domains is a critical limitation.

Following this diagnostic, the VersaPRM model was evaluated. The results for VersaPRM were transformative:

• Multi-Domain Performance: VersaPRM achieved substantial performance improvements not only in the math domain but also across all other tested domains, including law, philosophy, and biology. The presentation explicitly states that "our dream came true and our PRM works" in these other domains, indicating significant and consistent gains comparable to those seen in math. While specific headline numbers for each non-math domain are not detailed in the transcript, the visual representation implied that the performance increase was no longer negligible but rather substantial and positive across the board.

These findings validate VersaPRM's core hypothesis: by employing a data-centric approach and leveraging LLMs for synthetic data labeling to create a diverse, multi-domain training dataset, it is possible to train a PRM that generalizes effectively across a wide range of subject matters. The success on MMLU-Pro demonstrates that VersaPRM effectively overcomes the primary limitation of prior PRMs, offering a truly versatile solution for enhancing LLM reliability.

The researchers also briefly mention that their paper explores "other test-time methods" in addition to weighted majority voting, inviting readers to consult the full paper for further details on these complementary techniques. However, the primary focus of the talk and the headline results revolve around the direct comparison of PRM effectiveness via weighted majority voting across domains.

Practical Implications

▶ Watch: Key finding: Existing PRMs are domain-specific (6:00)

The development of VersaPRM carries substantial practical implications for various stakeholders involved in the deployment and utilization of Large Language Models (LLMs), marking a significant step towards more reliable and versatile AI systems.

For practitioners and application developers, VersaPRM enables the deployment of LLMs in a much broader array of high-stakes applications. Previously, the non-deterministic nature and occasional incorrectness of LLM outputs, coupled with the domain-specific limitations of existing Process Reward Models (PRMs), restricted their use in fields requiring rigorous accuracy and verifiable reasoning. With VersaPRM, it becomes feasible to build applications that not only provide answers but also offer a robust, step-by-step verification of the LLM's thought process across diverse domains like legal analysis, scientific inquiry, or complex problem-solving. This opens doors for LLMs in areas where trust and explainability are paramount, moving beyond mere content generation to reliable reasoning assistance. The ability to perform weighted majority voting using PRM scores also means that applications can achieve higher accuracy than simple LLM sampling or unweighted voting, leading to more dependable user experiences.

Infrastructure teams and MLOps engineers will find that VersaPRM's approach highlights the increasing importance of sophisticated data pipelines. The reliance on LLMs for synthetic data labeling necessitates robust systems capable of generating, validating, and managing large volumes of high-quality synthetic data across multiple domains. This includes developing efficient prompting strategies for annotation LLMs, implementing quality control mechanisms to prevent the propagation of errors or biases from the annotating LLM, and managing the storage and accessibility of these diverse datasets. The open-sourced data annotation pipeline provided by VersaPRM offers a valuable starting point and blueprint for such infrastructure, reducing the barrier to entry for teams looking to build their own multi-domain reward models.

For model builders and researchers, VersaPRM underscores a crucial shift in focus: from solely pursuing architectural innovations to recognizing the paramount importance of data diversity and quality. The finding that existing PRM limitations were primarily a "data problem" rather than an "algorithmic or architectural one" suggests that significant gains in generalizability can be achieved through intelligent data strategies. This encourages further research into advanced synthetic data generation techniques, prompt engineering for LLM-based annotation, and methods for effectively integrating diverse data sources. It also reinforces the value of fine-grained feedback models like PRMs, moving beyond simple outcome prediction to understanding and evaluating the underlying reasoning process, which is critical for complex cognitive tasks.

However, there are also tradeoffs and limitations to consider:

• Reliance on LLMs for Synthetic Labeling: While cost-effective, using LLMs to label data introduces a dependency on the quality and capabilities of the annotating LLM. If the annotating LLM itself has biases or generates incorrect labels, these errors can propagate into the PRM's training data, potentially leading to a less reliable reward model. Robust validation and human-in-the-loop checks might still be necessary for critical applications.
• Complexity of Integration: Integrating a PRM and weighted majority voting into an LLM deployment pipeline adds complexity compared to simply querying an LLM once. This involves managing multiple LLM inferences, querying the PRM for each reasoning step, and then performing the weighted aggregation. This increased computational overhead and latency might be a consideration for real-time, high-throughput applications.
• Scalability to All Domains: While VersaPRM demonstrates success across several diverse domains, scaling to all possible domains and handling highly niche or rapidly evolving knowledge areas might still present challenges. Continuous data generation and model retraining will likely be necessary to maintain broad applicability.
• "Technicalities" of Annotation: The talk acknowledges "technicalities" in the LLM-based annotation process that were glossed over. These details could hide significant engineering challenges related to prompt design, few-shot learning, handling ambiguous steps, and ensuring consistency in synthetic labeling.

Despite these considerations, VersaPRM offers a compelling path forward. Its ability to provide reliable, step-by-step reasoning evaluation across multiple domains represents a significant leap for LLM utility and trustworthiness. By open-sourcing its components, it actively fosters collaboration and accelerates the development of more general-purpose, intelligent AI systems.

Key Takeaways

• Existing PRMs are Domain-Specific: Current open-source Process Reward Models (PRMs), largely trained on math and coding data, show significant performance improvements only in these specific domains, failing to generalize to others like law, philosophy, or biology.
• Data Diversity is Key to Generalization: The primary limitation of existing PRMs is identified as a "data problem," not an architectural one. Achieving multi-domain generalization for PRMs requires training on a diverse dataset that spans a wide array of subject matters.
• LLMs Enable Scalable Synthetic Data Generation: VersaPRM leverages Large Language Models (LLMs) to synthetically generate and annotate high-quality, step-by-step correctness labels for reasoning traces across multiple domains, overcoming the cost and scarcity of manual annotation.
• VersaPRM Achieves Multi-Domain Performance: By fine-tuning on this synthetically generated, diverse dataset, VersaPRM demonstrates strong performance improvements over majority voting across math, law, philosophy, and biology, fulfilling the promise of a truly versatile reward model.
• PRMs Enhance LLM Reliability: Integrating PRMs with weighted majority voting significantly boosts the accuracy of LLM outputs by evaluating and downweighting incorrect reasoning steps, making LLM responses more reliable than single samples or naive voting.
• Open-Source Contribution Accelerates Research: The project open-sourced its multi-domain data annotation pipeline, dataset, and the VersaPRM model, encouraging further research and application of PRMs in domains beyond traditional math and coding tasks.

About the Speaker(s)

The talk on VersaPRM was presented by Thomas Zeng, representing a collaborative effort involving researchers from a consortium of esteemed institutions and companies. The extensive list of collaborators includes individuals from the University of Wisconsin, Korea University, Furiosa AI, UC Berkeley, and Grafton. This diverse group of contributors highlights a multidisciplinary approach to tackling complex challenges in AI and machine learning. While the talk primarily features Thomas Zeng as the presenter, the collective expertise of the team, spanning various academic and industrial backgrounds, underscores the depth of research and engineering that went into developing VersaPRM. Their work focuses on enhancing the reliability and generalizability of large language models, particularly through the innovation of process reward models and synthetic data generation techniques.