AdaParse: An Adaptive Parallel PDF Parsing and Resource Scaling Engine

AdaParse: An Adaptive Parallel PDF Parsing and Resource Scaling Engine

Speakers: Carlo Siebenschuh, Kyle Hippe, Alexander Brace, Arvind Ramanathan, Ian Foster, Robert Underwood

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

In the rapidly evolving landscape of large language models (LLMs), the quality and scale of pre-training data are paramount. This talk introduces AdaParse, an innovative adaptive parallel PDF parsing and resource scaling engine designed to unlock vast repositories of scientific literature for LLM training. Presented by Carlo Siebenschuh from the University of Chicago and Argonne National Laboratory, AdaParse addresses the critical bottleneck of efficiently and accurately extracting textual content from PDF documents, particularly in the scientific domain. The overarching goal is to enable the creation of "science-savvy" foundation models capable of understanding and reasoning over complex scientific communication.

The significance of AdaParse stems from the sheer volume of scientific information locked within PDF files. With estimates suggesting Google Scholar indexes potentially over half a billion PDFs, the challenge of converting this unstructured data into a usable pre-training corpus for LLMs is immense. Existing parsing tools often present a stark trade-off between throughput and accuracy, with none offering a universally optimal solution. AdaParse tackles this by intelligently selecting the best parsing strategy for each individual PDF, dynamically optimizing for both quality and computational cost. This meta-strategy is crucial for researchers aiming to leverage potentially trillions of tokens, transforming what was once an intractable HPC problem into a manageable and scalable data preparation pipeline for the next generation of scientific AI.

Background

▶ Watch: Introduction to AdaParse and LLM pre-training challenges (0:00)

The problem of extracting text from PDF documents, especially complex scientific papers, has long plagued the machine learning and systems communities. Traditional approaches fall into a few categories, each with inherent limitations. The simplest involves extraction-based parsers, which essentially "yank" text tokens directly from the PDF structure. While incredibly fast, these tools are prone to subtle errors, such as whitespace injection, corruption of SMILE strings (used in chemistry), URLs, or even the complete omission of domain-specific words. For scientific applications, where precision is paramount, such errors can be catastrophic, leading to misinterpretations of critical data.

On the other end of the spectrum are OCR (Optical Character Recognition) based parsers. These systems typically employ recurrent neural networks (RNNs) or, more recently, Vision Transformers, to decode textual tokens from an image representation of the page. While potentially more accurate, especially for scanned or image-heavy documents, OCR is computationally intensive. When applied to an entire corpus, auto-regressively generating token by token, it constitutes a massive inference operation even before LLM pre-training begins, presenting a significant high-performance computing (HPC) challenge. More sophisticated transformer-based architectures, like NUGAT or Marker, are generally considered high-quality but can suffer from severe failures, such as dropping entire pages—an unacceptable flaw for scientific integrity.

The core dilemma, as highlighted by Siebenschuh, is the vast disparity in throughput and accuracy across these tools. Benchmarking reveals up to four orders of magnitude difference in processing speed between the fastest and slowest parsers. Crucially, while some high-quality parsers are exceedingly slow, many fast extraction tools are often competitive in accuracy with their slower, more complex counterparts for a significant portion of documents. This observation led to the hypothesis: what if, for any given PDF, we could statistically infer the most appropriate parser to use, balancing its accuracy against its computational cost? This forms the basis of AdaParse, aiming to bridge the gap between the speed of basic extraction and the precision of advanced, resource-intensive methods, without incurring the prohibitive costs of applying the latter universally.

Key Findings

▶ Watch: Analysis of parser throughput and accuracy differences (4:00)

AdaParse's core innovation lies in its meta-strategy that adaptively orchestrates various base parsers, ultimately performing better than any individual component. The system statistically infers the optimal parser for a given PDF, a decision driven not by simple metadata, but by an analysis of the document's initial extracted text. This intelligent selection process allows AdaParse to achieve superior accuracy metrics—such as BLEU score, ROUGE score, and acceptable token frequency—compared to leading reference models, including those it selectively invokes.

A critical finding is AdaParse's remarkable robustness. Through simulations involving scanned or scrambled documents, AdaParse demonstrated significantly less performance degradation compared to other parsers. This resilience stems directly from its hybrid approach, which leverages fast extraction for initial assessment and then selectively deploys more robust, transformer-based parsing only when necessary. This adaptive mechanism ensures high-quality output even for challenging inputs that would typically stump simpler tools.

The research also definitively disproved the efficacy of simple heuristics for parser selection. Initial hypotheses suggested that metadata like publication year, publisher, or scientific domain might guide parser choice. However, empirical evaluation showed that such heuristics performed no better than random selection. Instead, the study revealed the indispensable role of LLMs in making informed parsing decisions. By providing the LLM with the first page of extracted text, AdaParse achieved close to 50% accuracy in predicting the optimal parser's BLEU score via regression. Furthermore, the application of Direct Preference Optimization (DPO) boosted this by an additional percentage point, demonstrating the subtle yet profound impact of preference-based fine-tuning.

Perhaps the most impactful finding from a systems perspective is the dramatic throughput improvement. When deployed on over 500 GPUs on the Polaris supercomputer at ALCF, AdaParse achieved up to 17 times higher throughput than NUGAT while simultaneously delivering higher accuracy. This performance gain is pivotal, as it transforms the previously intractable problem of processing hundreds of millions of scientific PDFs into a feasible pre-training data pipeline, proving that a few carefully selected preference data points can indeed steer high-performance computing workloads with remarkable efficiency. The regularization aspect of DPO was also noted as a key benefit, helping to manage the inherent noise in this multivariate regression problem.

Technical Deep Dive

▶ Watch: AdaParse's core idea: adaptive, statistically inferred parser selection (5:40)

At its heart, AdaParse formulates the problem of optimal PDF parsing as a constrained optimization problem. The objective is to identify the best parser for a given PDF, maximizing accuracy, while adhering to a predefined computational budget (memory and compute). The fundamental challenge is that the ground truth text, which would reveal the "best" parser, is unobserved. Therefore, AdaParse must statistically infer this optimal choice.

Instead of relying on often misleading PDF metadata (which the authors found to be ineffective), AdaParse leverages an ingenious predictive signal: the text already extracted by a fast, CPU-based parser. This initial extraction is "almost for free" in terms of computational cost, providing a rapid preliminary understanding of the document's content. This extracted text then serves as input to a smaller, specialized LLM, specifically Cybert. Cybert was chosen for its compact size and its pre-training on a vast corpus of scientific tokens, making it adept at understanding the nuances of scientific language.

The Cybert model is tasked with inferring how other, potentially more resource-intensive, parsers would perform on the given document. This decoupled approach is crucial for resource management: the initial extraction happens on the CPU, while the selective invocation of transformer-based architectures (which primarily consume GPU resources) only occurs when the LLM predicts a significant accuracy gain is computationally worthwhile. This minimizes GPU idle time and prevents resource contention.

The training of the Cybert model involves a two-stage process. Initially, it undergoes Empirical Risk Minimization (ERM), minimizing the L2 loss against the BLEU score, effectively learning to predict parser accuracy. However, a critical observation was that even when two parsers had similar objective accuracy scores (e.g., BLEU), domain experts often held strong, consistent opinions on which was superior. These "minuscule differences" could have profound scientific implications (e.g., "hyper" vs. "hypo," correct pH capitalization, precise mathematical terminology). To capture these nuanced expert preferences, the model is subsequently fine-tuned using Direct Preference Optimization (DPO). DPO allows AdaParse to align its parser selection with human judgment, making it "aware" of scientifically critical distinctions that traditional accuracy metrics might overlook. This DPO-driven fine-tuning also provides a valuable regularization effect, helping to stabilize the learning process in a noisy multivariate regression environment.

The resulting system operates as a hybrid engine. For every PDF, a rapid extraction is performed. The extracted text is fed into the DPO-fine-tuned Cybert model, which predicts the potential accuracy gains from using more advanced, GPU-intensive parsers. Based on this prediction and the available computational budget (controlled by a user-tunable alpha parameter, representing the maximum frequency of transformer-based inference calls), AdaParse decides whether to proceed with the fast extraction or to selectively spin up a transformer-based parser. This adaptive strategy ensures that high-quality, resource-intensive parsing is only applied to documents where it yields a significant, cost-effective improvement, thereby maximizing overall throughput without sacrificing critical accuracy.

Experimental Setup & Results

▶ Watch: Leveraging domain expert preferences with Direct Preference Optimization (9:00)

The evaluation of AdaParse began with the creation of a robust and diverse benchmark dataset of scientific documents. The team curated ground truth text for these documents, which is a significant undertaking, and then sorted the PDFs by difficulty to thoroughly assess parser performance across a spectrum of challenges. This rigorous setup allowed for a comprehensive analysis of both throughput and accuracy.

Several metrics were employed to gauge AdaParse's performance:

• Word-level accuracy: BLEU score and ROUGE score, commonly used in neural machine translation, quantify the quality of extracted text against ground truth.
• Character-level accuracy: Character Accuracy Rate (CAR) provides a fine-grained measure of text fidelity.
• Preference-based metrics: Win Rate and Frequency of Accepted Tokens reflect the alignment with domain expert preferences, particularly after DPO.

AdaParse was benchmarked against a range of existing parsing tools, including fast extraction tools and more comprehensive transformer-based architectures like NUGAT. The results consistently demonstrated AdaParse's superior performance. In terms of BLEU, ROUGE, and acceptable tokens, AdaParse either matched or surpassed the best individual parsers, effectively acting as a "meta-strategy" that leverages the strengths of its components.

A key experimental phase involved testing AdaParse's robustness. By simulating challenging conditions such as scanned documents or those with scrambled text, the researchers found that AdaParse exhibited significantly less performance degradation compared to other parsers. This confirmed the benefit of its hybrid strategy, allowing it to fall back on robust methods when initial extraction proved insufficient.

The study also included a critical ablation to justify the complexity of using an LLM and DPO for parser selection. Simple heuristics, such as using publication year or scientific domain, were tested and found to perform no better than random selection. Even providing text-based features like the paper's title only yielded blur scores in the high 40s. However, when an LLM was used for regression on the blur score, taking the first page of extracted text as input, performance approached 50%. The subsequent application of DPO further increased this by an additional percentage point, which, given the saturation point of around 56% for even an unattainable "blur optimal selector," represents a significant gain and robustly justifies the LLM and DPO inference.

From a throughput perspective, AdaParse was deployed on Polaris, an ALCF supercomputer, utilizing over 500 GPUs. While pure extraction tools remained unrivaled in speed (as AdaParse itself uses extraction as a first step), AdaParse demonstrated performance characteristics closer to these fast extraction tools than to the slower, comprehensive parsing pipelines. Specifically, AdaParse achieved an impressive 17 times higher throughput than NUGAT while simultaneously offering superior accuracy. This translates into a substantial reduction in data curation time, making the goal of training LLMs on hundreds of millions of scientific PDFs feasible. Resource utilization analysis showed that while AdaParse's workload is heterogeneous due to varying PDF complexities, its utilization profile was comparable to other parsers, albeit with a weak correlation between utilization and overall throughput.

Practical Implications

▶ Watch: AdaParse's superior performance and robustness demonstrated (11:00)

AdaParse represents a significant leap forward for anyone involved in large-scale text data preparation, particularly within the scientific domain. For practitioners and model builders, AdaParse offers an unprecedented ability to construct massive, high-quality pre-training corpora for LLMs. The previous bottleneck of intractable parsing times and inconsistent accuracy is largely overcome, opening the door for truly "science-savvy" foundation models trained on trillions of tokens from scientific literature. This means LLMs can be exposed to the full breadth and nuance of scientific communication, from complex equations to precise terminology, leading to more accurate and reliable scientific AI applications.

For infrastructure teams and those responsible for deploying ML systems, AdaParse provides an intelligent, adaptive solution for managing heterogeneous workloads. The system's ability to statistically infer the optimal parser for each document allows for dynamic resource allocation, selectively spinning up GPU-intensive transformer models only when their accuracy benefits outweigh their computational cost. This optimizes the utilization of both CPU and GPU resources, preventing unnecessary expenditure on computationally expensive methods for simple documents, while ensuring high-quality parsing for complex ones. The presence of a tunable alpha parameter further empowers infra teams to balance throughput and accuracy according to specific project requirements and budget constraints, enabling a customized Pareto frontier exploration.

While AdaParse offers substantial advantages, it's important to acknowledge its practical tradeoffs and limitations. It is inherently slower than a pure, unoptimized extraction method, as it involves an initial extraction step for every document and selective invocation of more complex parsers. However, this slight overhead is demonstrably justified by the significant gains in overall accuracy and the dramatic throughput improvements over high-quality baselines. The system requires initial configuration, but once set up, it is designed for ease of use, making it accessible to the broader community. Future work could explore the integration of diffusion-based large language models for text generation and evaluation, potentially further enhancing accuracy in challenging scenarios, and a more detailed exploration of the Pareto frontier between throughput and accuracy across different alpha values.

Key Takeaways

• AdaParse is an Adaptive Meta-Strategy: It intelligently orchestrates various base PDF parsers, outperforming any single component in both accuracy and throughput by adaptively selecting the best tool for each document.
• LLMs are Crucial for Parser Selection: Simple metadata heuristics are ineffective; instead, a specialized LLM (Cybert) fine-tuned with Direct Preference Optimization (DPO) accurately infers optimal parser choice based on initial extracted text.
• DPO Enhances Scientific Accuracy: DPO makes the model sensitive to minuscule but scientifically critical textual differences, capturing expert preferences that traditional metrics might miss and providing valuable regularization for noisy data.
• Dramatic Throughput and Robustness Gains: AdaParse achieves up to 17x higher throughput than NUGAT with superior accuracy, especially on challenging documents (e.g., scanned/scrambled text), making large-scale scientific data curation feasible.
• Enables Massive Scientific LLM Pre-training: By efficiently and accurately processing potentially trillions of tokens, AdaParse removes a critical bottleneck, paving the way for the development of "science-savvy" foundation models.
• Optimized HPC Workloads: The system demonstrates how a few preference data points can efficiently steer complex HPC workloads, optimizing resource utilization by selectively deploying GPU-intensive parsing only when necessary.

About the Speaker(s)

The lead speaker, Carlo Siebenschuh, is affiliated with both the University of Chicago and Argonne National Laboratory. He is part of a collaborative team, with most members also hailing from these institutions, united by a shared passion for AI for science. Their collective work involves identifying and addressing critical problems at the intersection of AI and scientific research. A recurring theme in their findings is the indispensable role of High-Performance Computing (HPC) in advancing AI for science, even extending to fundamental tasks like data creation, underscoring the necessity of systems like AdaParse to bridge the gap between scientific ambition and computational capability.