rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

Speakers: Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, Mao Yang

Conference: ICML 2025

YouTube: https://slideslive.com/39044027

Overview

The talk introduces rStar-Math, a novel framework designed to empower small Language Models (LLMs) with advanced mathematical reasoning capabilities through a process of self-evolved deep thinking. Presented at ICML 2025, this work addresses a critical challenge in the field of AI: the inherent difficulty for auto-regressive LLMs to perform complex, multi-step reasoning, particularly in mathematics. Unlike human cognition, which often employs a deliberate, "System 2" thinking mode for intricate problems, LLMs typically operate in a fast, "System 1" manner prone to errors and hallucinations, a significant drawback when precision is paramount, as in mathematical problem-solving.

Inspired by the success of AlphaGo's deep reasoning, rStar-Math leverages a Monte Carlo Tree Search (MCTS)-based approach. The core innovation lies in its ability to generate high-quality, step-by-step reasoning trajectories and continuously refine both a policy LLM and a process reward model through a self-evolutionary loop. This allows even computationally constrained 7-billion and 15-billion parameter models to achieve reasoning prowess comparable to or even surpassing much larger, state-of-the-art models, effectively demonstrating that deep thinking, not just scale, can unlock superior reasoning in LLMs. The research highlights a significant step towards building more reliable and intelligent AI systems capable of tackling complex, structured problems.

Background

▶ Watch: rStar-Math: Leveraging MCTS for human-like math reasoning (0:40)

Mathematical reasoning stands as one of the most structured and challenging problem domains for large language models. The inherent nature of auto-regressive generation in LLMs often leads to what cognitive psychologists term "System 1 thinking" – a fast, intuitive, and often error-prone mode. While efficient for many tasks, this rapid generation can result in logical inconsistencies, factual errors, and hallucinations, especially when faced with complex math problems where a single incorrect step invalidates the entire solution. This contrasts sharply with human problem-solving, which frequently employs "System 2 thinking" – a slower, more deliberate, and deeper analytical process that carefully evaluates each step.

Prior attempts to improve LLM reasoning have explored various methods, including Chain-of-Thought (CoT) prompting, which encourages LLMs to articulate intermediate reasoning steps. However, even CoT can suffer from the same System 1 limitations, potentially generating erroneous steps that LLMs struggle to self-correct. The scarcity of high-quality, step-by-step labeled math reasoning data further exacerbates this problem. Existing datasets are either too small, or synthetically generated data often contains subtle errors in reasoning steps, making them unreliable for training robust models. For process reward models, which evaluate the quality of intermediate steps, the problem is even more acute, as step-by-step human labeling is prohibitively expensive and automatic annotation tends to introduce noise.

The inspiration for rStar-Math comes from Monte Carlo Tree Search (MCTS), a powerful search algorithm popularized by its success in AlphaGo. MCTS simulates future moves to build a search tree, balancing exploration of new possibilities with exploitation of promising paths. It involves four key steps: selection (choosing the most promising node), expansion (adding new nodes/steps to the tree), roll-out (simulating a complete path to an outcome), and back-propagation (updating values of parent nodes based on roll-out results). While MCTS offers a framework for deep, human-like reasoning, adapting it to LLMs presents challenges, particularly in generating reliable trajectories and obtaining accurate step-level evaluations for training the policy and reward models necessary for guided search. The rStar-Math work specifically addresses these data quality and model training challenges within the MCTS paradigm for LLMs.

Key Findings

▶ Watch: Understanding the MCTS process: selection, expansion, backpropagation (2:40)

rStar-Math presents several key innovations that collectively enable small language models to master complex mathematical reasoning. The central finding is that through a self-evolved deep thinking recipe, LLMs can significantly enhance their reasoning capabilities, moving beyond the limitations of purely auto-regressive generation.

Firstly, the introduction of step-by-step verified trajectory generation with code augmentation is a major contribution. By integrating Python code generation alongside natural language chain-of-thought, the framework can objectively verify the correctness of each reasoning step. Invalid code execution immediately flags an erroneous step, allowing for its discard and ensuring that only high-quality, verifiable reasoning paths are used for training. This significantly addresses the issue of unreliable synthetic data.

Secondly, rStar-Math innovates in the training of its process reward model (PPM). Recognizing the inherent noisiness and imprecision of assigning exact scores to intermediate steps, the system trains a preference model instead. This model learns to compare and rank step-level reasoning quality using pairwise ranking loss, which is a more robust and effective approach than direct score regression. This preference-based PPM is crucial for guiding the MCTS search more accurately.

Finally, the framework's four-round self-evolutionary process demonstrates that models can continuously improve their own training data and capabilities. Starting with basic terminal-guided Q-value annotations and progressively incorporating the more accurate PPM-augmented MCTS, the system iteratively generates increasingly challenging and high-quality math problems and solutions. This self-improvement loop allows small 7-billion parameter models to surpass the performance of larger models like GPT-o1-mini on math reasoning benchmarks. Furthermore, a 15-billion parameter Qwen model, when equipped with rStar-Math, achieved performance on par with the strong o1-preview model, and significantly outperformed reinforcement learning-based methods like DeepSeek-R1 for similar model sizes. These results underscore the effectiveness of combining MCTS with robust data generation and preference learning for advancing LLM reasoning.

Technical Deep Dive

▶ Watch: Innovation: Step-by-step verified trajectories using code augmentation (4:10)

The technical foundation of rStar-Math is built upon an adaptive Monte Carlo Tree Search (MCTS) framework, specifically tailored for enhancing LLM mathematical reasoning. The system comprises two core components: a policy LLM responsible for generating candidate reasoning steps (nodes) and a process reward model (PPM) that evaluates the quality of these steps and trajectories.

The MCTS process in rStar-Math follows the standard four steps but with LLM-specific adaptations:

1. Selection: The policy LLM, guided by the PPM's scores, selects the most promising node (a partial reasoning trajectory) to expand further. Initially, without a trained PPM, this selection might be more random.
2. Expansion: From the selected node, the policy LLM generates multiple candidate next steps, forming new child nodes. This is where the innovation of code augmentation comes into play. Each generated step includes not only natural language chain-of-thought but also executable Python code.
3. Roll-out: The system then attempts to execute the generated Python code. If the code runs successfully and logically contributes to the problem-solving, the step is deemed valid. If the code fails or produces an incorrect intermediate result, that particular node and its subsequent trajectory are discarded. This step-by-step verification is crucial for filtering out erroneous reasoning paths, a common weakness in traditional CoT methods.
4. Back-propagation: Once a trajectory reaches a terminal answer node (or is discarded due to an error), a quality value (Q-value) is assigned. Initially, this Q-value is derived from comparing the final answer to the ground truth (terminal-guided Q-value annotation). This Q-value is then back-propagated up the tree, updating the scores of all parent nodes along the successful trajectory.

A significant challenge arises from the scarcity of high-quality, step-level labeled data for training both the policy LLM and the PPM. rStar-Math addresses this through two key innovations:

1. Step-by-Step Verified Trajectory Generation with Code Augmentation:

The core idea here is to make the reasoning process self-verifiable. When the policy LLM generates a reasoning step, it's prompted to produce:

• Natural language chain-of-thought explaining the step.
• Corresponding Python code that executes the mathematical operation described.

The benefit of this code augmentation is that the system can run the generated Python code. If the code executes successfully and yields a reasonable intermediate result, the step is considered valid. If the code fails (e.g., syntax error, runtime error) or produces an obviously incorrect value, the step is flagged as low-quality and the associated node is discarded. This robust verification mechanism ensures that the generated trajectories used for training are of high fidelity, avoiding the propagation of subtle errors often found in purely natural language synthetic data.

2. Preference-based Process Reward Model (PPM) Training:

Instead of training the PPM to predict an exact quality score for each step, which is inherently noisy and imprecise, rStar-Math trains a preference model. This model learns to compare pairs of reasoning steps or trajectories and determine which one is "better." The Q-values derived from the MCTS roll-outs (both terminal-guided and later PPM-augmented) are used to construct these step-level preference pairs. For example, if trajectory A has a higher Q-value than trajectory B, then all steps in A are preferred over corresponding steps in B, or specific steps within a trajectory can be compared. The PPM is then trained using a pairwise ranking loss, which is more stable and effective for learning relative quality judgments than trying to regress to absolute, noisy scores.

Self-Evolved Deep Thinking Recipe:

rStar-Math adopts a four-round self-evolutionary process to continuously improve the policy LLM and PPM:

• Rounds 1 & 2 (Terminal-Guided MCTS): In the initial rounds, with no strong PPM available, the MCTS relies on terminal-guided Q-value annotation. This means the Q-value for a trajectory is determined solely by the correctness of the final answer against the ground truth. Node selection is less informed, often involving more random exploration. The verified trajectories from these rounds are used to bootstrap the training of the first PPM.
• Rounds 3 & 4 (PPM-Augmented MCTS): Once a preliminary PPM is trained, it's integrated into the MCTS. Now, the RPM-augmented Q-value annotation allows the MCTS to select promising nodes based on the PPM's learned preferences. This guides the search more efficiently towards high-quality reasoning paths, enabling deeper and more effective exploration. The data generated in these rounds, guided by an increasingly sophisticated PPM, becomes more challenging and refined, further improving both the policy LLM and the PPM in an iterative feedback loop.

This self-evolutionary process allows the models to generate increasingly complex and correct problem-solving trajectories, effectively learning to think "deeper" and more accurately over time, even with limited initial resources (e.g., a single A100 GPU for inference).

Experimental Setup & Results

▶ Watch: rStar-Math's four-round self-evolved deep thinking recipe (6:10)

The rStar-Math framework was evaluated using "two small language models" of 7 billion parameters and a 15-billion parameter Qwen model. The primary hardware constraint mentioned was inference on "one A100 GPU," highlighting the efficiency considerations for resource-limited environments. While specific dataset names for math reasoning benchmarks were not explicitly stated in the transcript, the evaluation focused on standard math reasoning tasks.

The experimental results demonstrated a significant improvement in mathematical reasoning capabilities for these smaller LLMs through the rStar-Math approach:

• Performance of 7B Models: The 7-billion parameter models, after undergoing the four rounds of self-evolution, showed remarkable performance. They were able to outperform models like GPT-o1-mini on math reasoning benchmarks. This is a substantial finding, indicating that deep thinking and robust training methodologies can compensate for a smaller parameter count, achieving superior reasoning quality compared to potentially larger, less specialized models.

• Performance of 15B Qwen Model: When the rStar-Math method was applied to a 15-billion parameter Qwen model, its reasoning capabilities reached a level on par with the o1-preview model. This further reinforces the scalability and effectiveness of the rStar-Math framework, showing that it can elevate the performance of moderately sized LLMs to compete with very strong baselines.

• Comparison with Reinforcement Learning (RL) Methods: The talk also presented a direct comparison with existing reinforcement learning-based approaches, specifically mentioning DeepSeek-R1.
• For the 15-billion parameter models, rStar-Math significantly achieved better performance than DeepSeek-R1. This suggests that the MCTS-based self-evolution, coupled with code augmentation and preference learning, provides a more robust and effective strategy for math reasoning than traditional RL methods in this size category.
• For the 7-billion parameter models, rStar-Math achieved comparable performance to DeepSeek-R1, indicating its competitive edge even at the smallest scales tested.

• Self-Evolution Trajectory: The results table presented during the talk (though not explicitly detailed in the transcript) visually confirmed that in "each round they did get stronger and better performance in math reasoning benchmarks." This empirically validated the core hypothesis of the self-evolutionary process, demonstrating that the iterative refinement of the policy LLM and process reward model through MCTS-generated data leads to continuous and tangible improvements in reasoning quality.

The use of code augmentation proved critical for generating high-quality training data, mitigating the issues of spurious Q-value assignments and poor data quality that can arise from the weaker capabilities of smaller language models. This allowed the training set to evolve beyond mainly simple questions to include more complex and challenging math problems, thereby fostering deeper and more generalized reasoning abilities.

Practical Implications

▶ Watch: Key results: Small LLMs achieve state-of-the-art math reasoning (7:00)

The rStar-Math framework holds significant practical implications for the development and deployment of LLMs, particularly for tasks requiring robust, multi-step reasoning like mathematics.

Firstly, the most direct implication is the democratization of advanced reasoning capabilities. By enabling small LLMs (7B and 15B parameters) to achieve performance comparable to or surpassing much larger models, rStar-Math drastically reduces the computational resources required for complex reasoning. This means that organizations with limited budgets or smaller compute infrastructures can still deploy highly capable reasoning agents, making advanced AI more accessible and cost-effective. Running inference on "one A100 GPU" for these models is a testament to this efficiency.

Secondly, the enhanced reliability and reduced hallucination through step-by-step verified trajectory generation is a game-changer for critical applications. By integrating executable code and using its success/failure as an objective verification mechanism, rStar-Math provides a robust guardrail against the propagation of errors. This is crucial for domains where correctness is paramount, such as scientific computation, engineering design, or financial modeling, where erroneous LLM outputs can have severe consequences. Practitioners can build greater trust in the outputs of such models.

Thirdly, the self-evolved deep thinking recipe offers a pathway for continuous model improvement without constant human intervention. The iterative process of generating data, training models, and using improved models to generate even better data creates a virtuous cycle. This reduces the dependency on expensive human annotation for complex step-by-step reasoning, which is a major bottleneck in developing advanced AI. Infrastructure teams can design systems that periodically run these self-evolution rounds to keep models up-to-date and continuously enhance their reasoning abilities.

However, there are also practical considerations and limitations. The initial computational cost of performing extensive MCTS roll-outs, even with small LLMs, can still be substantial, especially for generating the initial high-quality trajectories. The reliance on code execution for verification means that the framework is best suited for problems where steps can be unambiguously translated into executable code, which might not cover all forms of abstract or qualitative reasoning. The transcript also mentions that "insufficient tree exploration can lead to spurious Q-value assignment," highlighting a need for careful hyperparameter tuning and sufficient exploration budget during the MCTS phase. Furthermore, the "weaker capabilities of smaller language models" can initially lead to "poor quality training data" and a bias towards "simple questions" if not carefully managed by the self-evolution process.

For model builders and deployers, rStar-Math suggests a shift in focus from simply scaling up model size to investing in more sophisticated reasoning architectures and training paradigms. The emphasis on robust data generation, preference learning, and self-improvement offers a blueprint for building more intelligent and reliable AI systems that can reason deeply and effectively in structured problem domains.

Key Takeaways

• Small LLMs Can Master Math Reasoning: rStar-Math demonstrates that 7B and 15B parameter models can achieve state-of-the-art math reasoning performance, outperforming larger baselines like GPT-o1-mini and competing with o1-preview, disproving the notion that only massive models can handle complex reasoning.
• Self-Evolved Deep Thinking (MCTS-based): The core innovation is a four-round self-evolutionary process inspired by AlphaGo's MCTS, which iteratively refines a policy LLM and a process reward model, leading to continuous improvement in reasoning capabilities and data quality.
• Code-Augmented Step-by-Step Verification: Critical to the framework is the generation of executable Python code alongside natural language chain-of-thought for each reasoning step. This allows objective verification and discarding of erroneous steps, ensuring high-quality training data and reducing hallucination.
• Preference-based Process Reward Model: Instead of directly scoring steps, rStar-Math trains a robust preference model using pairwise ranking loss. This model learns to compare and rank reasoning quality, effectively guiding the MCTS search and improving its accuracy.
• Superiority over RL Methods: rStar-Math significantly outperforms reinforcement learning methods like DeepSeek-R1 for 15B models and achieves comparable performance for 7B models, showcasing the effectiveness of its MCTS-inspired architecture for structured reasoning tasks.
• Reduced Computational Cost and Enhanced Reliability: The framework's ability to achieve high performance with smaller models and its built-in verification mechanisms mean more accessible, cost-effective, and reliable LLM deployments for mathematical and other structured reasoning problems.

About the Speaker(s)

The talk on rStar-Math was presented by Yupan Huang on behalf of the authors, who were unable to join the conference in person. The research team behind rStar-Math comprises Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. While their specific titles were not detailed in the transcript, the concluding remarks, which included an invitation to reach out to Lyna Zhang and a recruitment announcement for Microsoft Research, strongly suggest that the authors are affiliated with Microsoft Research. They are actively seeking full-time researchers, engineers, and interns, indicating a vibrant research environment focused on advancing frontier reasoning models.