Machine Learning meets Algebraic Combinatorics: A Suite of Datasets Capturing Research-level Conjecturing Ability in Pure Mathematics

Machine Learning meets Algebraic Combinatorics: A Suite of Datasets Capturing Research-level Conjecturing Ability in Pure Mathematics

Speakers: Herman Chau, Helen Jenne, Davis Brown, Jesse He, Mark Raugas, Sara Billey, Henry Kvinge

Conference: ICML 2025

YouTube: https://slideslive.com/39044078

Overview

This talk, presented by Herman Chau, introduces a novel collection of datasets designed to advance the application of AI in pure mathematics, specifically focusing on the often-overlooked aspects of mathematical discovery: intuition, exploration, and conjecture generation. Developed by a large interdisciplinary team of AI researchers and mathematicians, the initiative addresses critical gaps in existing AI for math datasets, which predominantly focus on known mathematics and proof generation. The presented suite of nine datasets, rooted in algebraic combinatorics, aims to provide machine learning models with raw mathematical data corresponding to both open problems and foundational results, thereby enabling AI to assist mathematicians in forming new hypotheses and uncovering hidden patterns.

The core motivation behind this work is to equip AI with the capacity to engage with mathematics in a way that mirrors a human mathematician's workflow, moving beyond mere problem-solving to active discovery. By reframing complex mathematical challenges as ML-friendly classification or regression tasks, the researchers demonstrate that even off-the-shelf models can achieve high accuracy, suggesting that these models learn deeply interesting mathematical information. This pioneering effort positions AI not just as a computational assistant but as a "telescope" for mathematicians, expanding their intuition and providing new toeholds for achieving novel research results by navigating vast, enumerable combinatorial spaces.

The significance of this contribution lies in its potential to unlock new frontiers in mathematical research. By providing structured, scalable datasets derived from live, open problems, the project invites the ML community to tackle challenges that are at the forefront of mathematical inquiry. It emphasizes the need for interpretability in AI models, as extracting the learned mathematical insights becomes paramount. Ultimately, this work represents a crucial step towards a future where AI actively participates in the creative and exploratory phases of mathematical discovery, fostering a symbiotic relationship between artificial intelligence and pure mathematics.

Background

▶ Watch: Introduction: Gaps in existing AI for math datasets (0:00)

The landscape of AI applications in mathematics has seen significant growth, yet it has traditionally focused on areas where the answers or proofs are already known. Widely available datasets typically involve grade-school to graduate-level mathematics, presenting natural language questions that require either a direct answer or a formal proof. While valuable for training models in logical deduction and problem-solving, this approach misses crucial elements of a mathematician's actual workflow. Mathematicians spend considerable time gaining intuition, exploring examples, and discovering patterns—a process often leading to the generation of conjectures, where the answer is fundamentally unknown. This exploratory phase, driven by open problems, is a cornerstone of mathematical research that existing datasets largely fail to capture.

A typical mathematical workflow, as described by Chau, is highly iterative and often begins with vague questions discussed informally with colleagues. The initial step frequently involves computing examples and playing with concrete instantiations of abstract concepts to build intuition. This experimental phase is followed by literature searches and, if lucky, attempts at formal proof. This cycle of exploration, intuition building, and rigorous validation can repeat many times before a new result is established. The lack of datasets supporting this early, intuitive phase of discovery prompted the creation of this new suite.

The researchers chose algebraic combinatorics as the domain for these datasets for several strategic reasons. Algebraic combinatorics is a field that studies discrete structures arising from abstract algebra, encompassing areas such as representation theory and algebraic geometry. This domain is particularly well-suited for ML applications because:

1. Problem Stating: The problems can often be stated with a relatively small barrier of knowledge, making them accessible to a broader audience.
2. Computer-Friendliness: The discrete nature of the structures (e.g., graphs, permutations) makes them inherently amenable to computational approaches.
3. ML Community Experience: The ML community already has considerable experience with many combinatorial objects, such as graphs and permutations.
4. Open-Source Libraries: Robust open-source libraries exist for generating the necessary data, facilitating the creation of large datasets.
5. Data Volume: It is often possible to generate vast quantities of data, which is crucial for training effective machine learning models, in contrast to some areas of mathematics where examples might be scarce.

These characteristics make algebraic combinatorics an ideal testing ground for developing AI systems capable of mathematical conjecture generation, bridging the gap between computational power and the human-driven process of mathematical discovery.

Key Findings

▶ Watch: Introducing new datasets in Algebraic Combinatorics (2:00)

The central contribution of this work is the development of a unique collection of nine datasets, specifically curated to capture research-level conjecturing ability in pure mathematics. These datasets are distinct in that six correspond to open problems in algebraic combinatorics—problems for which no known solution or proof exists—while three relate to foundational results, providing a rich ground for both discovery and validation.

A key innovation in the dataset design is their parameterized nature. Each dataset is not static but rather a series of datasets controlled by an integer parameter, n. For instance, a dataset concerning permutations might allow n to represent the number of elements being permuted. This parameter acts as a "knob," enabling researchers to generate datasets of varying sizes and complexities, from smaller, more manageable sets to much larger ones. While the project has not yet provided the absolute largest possible datasets, the underlying code for generation is open-sourced, empowering users to scale the data as needed. The data types within these datasets are diverse, ranging from lattice paths and matrices to directed graphs, integer partitions, permutations, and Young tableaux, reflecting the rich variety of objects studied in algebraic combinatorics. Crucially, each problem is accompanied by contextual information and motivation, guiding users on its significance within the broader mathematical landscape.

The researchers demonstrated the utility of these datasets by reframing complex mathematical questions into standard ML-friendly tasks, such as classification or regression. The premise is that if a model can perform well on these tasks using only raw combinatorial data, it must have learned something profoundly interesting and mathematically relevant. A compelling example presented was the problem of structure constants for Schubert polynomials. Despite decades of study, the combinatorial interpretation of these constants remains an open problem. By feeding an off-the-shelf ML model purely combinatorial data (permutations alpha, beta, and gamma), the model achieved a remarkable 99.8% accuracy in predicting these structure constants.

This high accuracy, achieved without any specialized geometric or algebraic knowledge encoded into the model, suggests that the models were able to extract deep, underlying combinatorial relationships. This finding is not merely a successful prediction but a significant indicator that AI can learn and represent complex mathematical structures from raw data. The implication is profound: the challenge shifts from achieving high performance to interpreting what the models have learned. This opens up exciting opportunities for the interpretability community within ML, as extracting these latent mathematical insights from the models becomes the next critical step. The project thus frames AI as a "telescope" for mathematicians, a tool that can expand their intuition, help them identify patterns in vast combinatorial spaces, and provide the initial "toehold" necessary for achieving new mathematical results. The ability to generate and optimize both models and datasets in a data-rich environment further underscores the potential for accelerating mathematical discovery.

Technical Deep Dive

▶ Watch: Suite of nine parameterized datasets: open and foundational problems (2:50)

The technical foundation of this work lies in the meticulous design and curation of the dataset suite, specifically tailored for machine learning engagement with open problems in pure mathematics. The collection comprises nine distinct datasets, each representing a unique challenge or concept within algebraic combinatorics. Crucially, six of these datasets are derived from active open problems, reflecting areas where mathematical understanding is incomplete, while the remaining three are based on foundational results, which can serve as benchmarks or for validating learned representations.

A core technical feature is the parameterization of each dataset by an integer n. This means that for a given mathematical problem, the dataset isn't a single fixed collection of examples, but rather a family of datasets. For instance, if a problem involves permutations, n might denote the number of elements in the permutation. This design provides a "knob" that allows researchers to control the scale and complexity of the data, generating smaller instances for initial exploration or massive datasets for more intensive training. The raw mathematical data types within these datasets are diverse, encompassing lattice paths, matrices, directed graphs, integer partitions, permutations, and Young tableaux. These are the fundamental objects of study in algebraic combinatorics, and their direct representation in the datasets ensures that the ML models are operating on the primary mathematical entities.

The methodology for transforming these mathematical problems into ML-friendly tasks involves reframing them as either classification or regression problems. The underlying philosophy is that if an ML model can achieve high performance on such a task using only the raw combinatorial input, it strongly suggests that the model has learned a significant, underlying mathematical structure. This approach avoids explicitly encoding complex mathematical theories into the model, instead allowing the model to discover these relationships autonomously.

A prime example detailed in the talk is the problem of structure constants for Schubert polynomials. Schubert polynomials are a remarkable family of polynomials in variables x1, x2, ..., parameterized by permutations. They are significant because they encode deep geometric information related to Schubert calculus in cohomology rings of flag varieties, and also find applications in theoretical physics and representation theory. When two Schubert polynomials are multiplied, the product can be expressed as a linear combination of other Schubert polynomials in the same family. The coefficients of this linear combination are known as the structure constants, denoted C_alpha,beta^gamma, where alpha, beta, and gamma are the permutations parameterizing the polynomials involved. For over two decades, it has been conjectured that these C values count something combinatorial, likely related to the permutations alpha, beta, and gamma, but a precise combinatorial interpretation remains an open problem.

To tackle this, the researchers designed an ML task where an "off-the-shelf" model was provided solely with the combinatorial data (the permutations alpha, beta, gamma). The model's task was to predict the corresponding structure constant C_alpha,beta^gamma. The success of this approach—achieving 99.8% accuracy—is highly indicative. It demonstrates that the model, without explicit instruction on the intricate geometry or algebra underlying Schubert polynomials, was able to discern the combinatorial rules governing these constants. This implies that the model has internally constructed a representation that captures the essence of this complex mathematical relationship. The challenge then transitions from mere prediction to model interpretability: how can we extract these learned combinatorial rules from the model and translate them back into human-understandable mathematical conjectures or proofs? This problem highlights a rich avenue for future research at the intersection of ML interpretability and pure mathematics, where the AI serves as a powerful instrument for uncovering hidden mathematical truths.

Experimental Setup & Results

▶ Watch: Reframing mathematical problems for ML-friendly use (4:00)

The experimental setup for this project centers around the nine distinct datasets generated from problems in algebraic combinatorics. Each of these datasets is designed to be parameterized by an integer n, which allows for a flexible generation of data instances. For example, in a dataset concerning permutations, n would represent the number of elements being permuted, allowing the creation of data for permutations of 3, 4, 5, or more elements. This inherent scalability means that while the researchers provided initial datasets, the underlying code for data generation is also made available, empowering other researchers to create larger or more specific instances as needed. The data types range widely, including lattice paths, matrices, directed graphs, integer partitions, permutations, and Young tableaux, each representing fundamental objects in combinatorial mathematics.

For the initial exploration and demonstration of the datasets' utility, the team utilized "off-the-shelf models." The transcript does not specify particular model architectures (e.g., convolutional neural networks, transformers, graph neural networks) or frameworks (e.g., TensorFlow, PyTorch), nor does it detail specific hardware used for training. The emphasis was deliberately placed on the datasets themselves and the inherent mathematical information they contain, rather than on optimizing the ML models. The rationale was that if even basic, readily available models could perform well, it would strongly validate the premise that the datasets capture meaningful mathematical structure.

A critical headline result came from the Schubert polynomial structure constants dataset. In this experiment, the task was to predict the structure constants C_alpha,beta^gamma given only the combinatorial input of the three permutations alpha, beta, and gamma. This was framed as a classification or regression task, depending on the nature of the constants. The "off-the-shelf models" achieved an impressive 99.8% accuracy on this problem. This high level of performance is significant because it was achieved without explicitly encoding any of the deep geometric or algebraic theory known about Schubert polynomials into the model. It suggests that the model was able to infer complex combinatorial relationships solely from the raw permutation data.

Further supporting the efficacy of this approach, the models demonstrated a capacity for extrapolation. In a specific instance, by holding out larger n values (i.e., testing on permutations of more elements than seen during training), the models were able to generalize and, in one notable case, "rediscovered a few theorems" previously known in the theory of quiver mutation. This ability to extrapolate and uncover known mathematical truths from unseen data is a strong indication that the models are learning generalizable mathematical principles, rather than merely memorizing training examples or identifying spurious correlations. The researchers acknowledge that future experimental setups should rigorously test this generalization by consistently holding out larger n sizes for evaluation. The success of these initial experiments, particularly the high accuracy on open problems and the ability to extrapolate, firmly establishes the potential for these datasets to drive significant advancements in AI-assisted mathematical discovery.

Practical Implications

▶ Watch: Concrete example: Schubert polynomials structure constants (4:15)

The introduction of these novel datasets has profound practical implications for various stakeholders within the AI and mathematics communities.

For mathematicians, these datasets represent a powerful new tool, akin to a "telescope" for exploring vast mathematical landscapes. The ability of AI models to process enumerable amounts of combinatorial "gadgets" and identify patterns far beyond human capacity offers an unprecedented advantage in generating intuition and formulating conjectures. This can provide the crucial "toehold" needed to tackle long-standing open problems, accelerating the pace of mathematical discovery. Instead of spending months or years manually exploring examples, mathematicians could leverage AI to quickly identify promising avenues of research or even suggest new theorems, thereby augmenting human creativity and intuition.

For ML practitioners and researchers, this work opens up a rich, data-rich domain for applying and developing advanced machine learning techniques. The challenge shifts from simply achieving high predictive accuracy to solving the critical problem of interpretability. If a model can achieve 99.8% accuracy on a complex combinatorial problem, the next step is to understand what it has learned and how. This necessitates new research in explainable AI (XAI) to extract the underlying mathematical rules, patterns, or algorithms discovered by the models and present them in a human-understandable format. Furthermore, the parameterized nature of the datasets provides a unique environment for studying model scalability, generalization to unseen n values, and the robustness of learned representations across different problem sizes. The project also highlights opportunities for generative AI, where models could potentially generate new mathematical objects or even entire conjectures.

For infrastructure teams and model builders, the emphasis on generating large, scalable datasets means that robust data pipelines and computational resources will be essential. The ability to generate "lots and lots of data" in algebraic combinatorics presents an exciting challenge for distributed training, efficient data storage, and the development of specialized hardware/software stacks optimized for combinatorial data types. There's also an implicit opportunity for dataset optimization, where instead of just making models better, researchers can focus on making the datasets themselves better suited for eliciting specific mathematical insights.

However, there are also important tradeoffs and limitations. While high accuracy is a promising first step, it is explicitly stated as "the beginning of the problem," not the end. The real challenge lies in interpreting the model's knowledge, which is a non-trivial task. The initial experiments used "off-the-shelf models," implying that the full potential of more sophisticated architectures (e.g., specialized graph neural networks, transformer models for sequences of mathematical objects) remains to be explored. The success of these models is contingent on the careful reframing of mathematical problems into ML tasks, which still requires significant mathematical insight. Moreover, the field of algebraic combinatorics, while rich, is only one area of pure mathematics; extending this approach to other complex domains will require similar careful dataset design and problem formulation. The project's success underscores the need for deep interdisciplinary collaboration between mathematicians and AI researchers to bridge the conceptual gaps and translate AI's computational power into meaningful mathematical progress.

Key Takeaways

• New Datasets for Mathematical Discovery: A suite of nine novel datasets (six open problems, three foundational results) in algebraic combinatorics addresses a critical gap in AI for math, focusing on intuition, exploration, and conjecture generation rather than just known solutions.
• Algebraic Combinatorics as an Ideal ML Domain: The field's discrete, computer-friendly nature, existence of open-source data generation libraries, and capacity for generating vast amounts of data make it highly suitable for machine learning applications.
• Parameterized and Scalable Data: Each dataset is parameterized by an integer n, allowing researchers to generate varying sizes and complexities of data, providing a flexible "knob" for experimentation and scalability studies.
• High Accuracy on Open Problems: Off-the-shelf ML models achieved 99.8% accuracy on predicting Schubert polynomial structure constants using only raw combinatorial data, suggesting models learn deep underlying mathematical relationships without explicit encoding of complex theories.
• Shift to Model Interpretability: The primary challenge now transitions from achieving predictive performance to extracting and interpreting the mathematical insights learned by the models, opening new avenues for explainable AI research in mathematics.
• AI as a "Telescope" for Mathematicians: AI can serve as a powerful tool to augment human intuition, identify patterns in vast combinatorial spaces, and provide initial hypotheses, thereby accelerating the process of mathematical discovery.

About the Speaker(s)

Herman Chau is the primary presenter of this talk, representing a large and diverse interdisciplinary team. This team comprises both dedicated AI researchers and accomplished mathematicians, as well as individuals who proficiently navigate both fields. The collective expertise of this group, including Helen Jenne, Davis Brown, Jesse He, Mark Raugas, Sara Billey, and Henry Kvinge, is focused on the burgeoning field of using AI to advance research mathematics. Their work aims to bridge the gap between advanced computational methods and the intricate, often intuitive, processes of mathematical discovery.