Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection

Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection

Speakers: Zhijing Wan, Zhixiang Wang, Zheng Wang, Xin Xu, Shin'ichi Satoh

Conference: ICML 2025

YouTube: https://slideslive.com/39044011

Overview

In an era defined by the exponential growth of data, the deep learning paradigm has witnessed unprecedented advancements. However, the prevailing wisdom that "more data equals better performance" is increasingly being challenged. This presentation, delivered on behalf of Zhijing Wan from Wuhan University, delves into this critical issue, exploring the diminishing returns of endlessly expanding datasets due to escalating costs, redundancy, noise, and data imbalance. The core of the talk centers on subset selection, an elegant strategy designed to identify the most informative samples within a large dataset, thereby enabling efficient model training without compromising performance.

The work specifically investigates the one-shot subset selection paradigm, a highly efficient and scalable approach that selects a data subset once, based on a pre-trained model, for subsequent target model training. Traditional one-shot methods suffer from dataset dependency, necessitating costly retraining when data distributions shift. This talk explores the transformative potential of Foundation Models (FMs) like DINOv2 and CLIP as robust, dataset-independent information extractors. Through a comprehensive empirical study, the authors reveal that while FMs offer significant advantages for fine-grained data, their performance on coarse-grained datasets can be unstable. This nuanced understanding paves the way for a novel multi-model framework, RAM-APL, which strategically combines the strengths of multiple FMs to achieve state-of-the-art performance in fine-grained one-shot subset selection.

This research is highly significant for the ML community as it addresses a fundamental bottleneck in scaling deep learning applications. By providing a deep empirical analysis of FM efficacy in subset selection and proposing an innovative multi-FM solution, the talk offers actionable insights for optimizing data utilization, reducing computational overhead, and enhancing the flexibility of ML workflows. It challenges the conventional view of FMs as universal panaceas, instead advocating for a more sophisticated, context-aware approach, particularly in the critical domain of data efficiency.

Background

▶ Watch: Introduction: The challenge of diminishing returns from data (0:00)

The relentless pursuit of performance in deep learning has historically been synonymous with the acquisition and utilization of ever-larger datasets. Benchmarks like ImageNet have demonstrated a clear inverse relationship between dataset size and error rates, suggesting that more data leads to better models. However, this trend has begun to plateau. As datasets expand beyond a certain point, the marginal gains in performance diminish significantly, while the associated costs—storage, computational resources, and human annotation—skyrocket. This creates a critical bottleneck, making the traditional "data-hungry" approach unsustainable and inefficient.

The problem with simply accumulating more data is multifaceted. Beyond the direct financial and infrastructural burden, larger datasets frequently contain high degrees of redundancy, where many samples offer little new information to the learning process. They are also prone to noise, which can mislead models and degrade generalization capabilities. Furthermore, imbalanced sampling, where certain classes or features are over- or under-represented, can hinder a model's ability to learn robust and fair representations. These factors collectively transform what should be an asset (more data) into a liability, actively impeding effective learning.

To counter these challenges, subset selection has emerged as a crucial strategy. The fundamental objective is to identify and retain only the most informative and representative samples from a large dataset, thereby creating a smaller, higher-quality subset. This allows for more efficient model training—reducing training time, energy consumption, and memory footprint—without a significant compromise in the final model's performance. For instance, studies on datasets like CIFAR-10 have shown that training with as little as 60% of the data can yield performance comparable to using the entire dataset, underscoring the potential of intelligent data curation.

Subset selection methodologies broadly fall into two paradigms: adaptive and one-shot. Adaptive subset selection is an iterative process. It involves repeatedly selecting a subset, training the target model on it for a few epochs, and then using the updated model's state to inform the selection of the next subset. This creates a dynamic feedback loop between selection and training, continuing until model convergence. While adaptive methods offer flexibility and can potentially fine-tune the subset based on evolving model knowledge, they are computationally expensive, require continuous access to the full dataset, and involve complex orchestration of training and selection steps.

In contrast, one-shot subset selection is a single-pass process designed for efficiency and scalability. In this paradigm, a representative subset is chosen once at the outset, typically based on a pre-trained model, and the final target model is then trained exclusively on this fixed subset from scratch. This approach is highly appealing because it eliminates the iterative overhead of adaptive methods and, crucially, does not require access to the full dataset after the initial selection phase. For these compelling reasons, the presented work focuses exclusively on the one-shot setting.

The traditional implementation of one-shot subset selection relies on an Information Extractor (IE). This IE is typically a model pre-trained on the full target dataset. Its role is to generate data representations (embeddings or features) that guide the subset selection algorithm. While effective, this traditional approach suffers from a critical limitation: it is inherently dataset-dependent. If the target dataset is updated or a new dataset is introduced, the IE must be fully retrained, incurring significant computational and time costs. This lack of flexibility severely limits the scalability of traditional IE-based methods in dynamic real-world environments.

Recent advancements in deep learning, particularly the advent of Foundation Models (FMs), offer a promising alternative to traditional IEs. FMs, such as DINOv2, CLIP, SigLIP, and EVA-CLIP, are neural networks pre-trained on massive, diverse datasets (often internet-scale image-text pairs or self-supervised vision tasks). Their strength lies in their remarkable ability to generalize well across a wide array of downstream tasks and domains without extensive task-specific fine-tuning. By leveraging FMs as drop-in IEs, the dataset dependence of traditional methods can be largely eliminated. FMs require zero retraining cost for new datasets or tasks, as they are used directly to extract features, bringing unprecedented scalability and flexibility to the one-shot subset selection process.

However, despite their theoretical advantages, current FM-based subset selection methods presented in research often exhibit two key limitations. First, many studies tend to rely predominantly on a single FM, most commonly DINO, potentially overlooking the diverse strengths of other FMs. Second, these methods are frequently validated on "ideal" datasets that may not accurately reflect the complexities of real-world data. In practical scenarios, datasets are often fine-grained (with subtle distinctions between classes), noisy (containing mislabeled or extraneous samples), and imbalanced. Furthermore, practitioners have access to a spectrum of FMs, each with distinct inductive biases and representational capabilities. This context naturally leads to the core research question addressed by this work: Can FM-based one-shot subset selection truly and consistently outperform traditional IE-based methods across a diverse range of data conditions, especially those encountered in real-world fine-grained applications?

Key Findings

▶ Watch: Limitations of traditional extractors; foundation models as solution (6:00)

The comprehensive empirical study undertaken in this work yielded several critical findings that refine our understanding of Foundation Models' utility in one-shot subset selection and pave the way for more effective data efficiency strategies.

The initial broad finding, somewhat surprisingly, revealed that Foundation Models do not universally outperform traditional Information Extractors (IEs). While FMs offer undeniable advantages in terms of scalability and zero retraining cost, their performance in subset selection is not always superior to IEs meticulously pre-trained on the full target dataset, especially in certain data regimes. This nuance challenges the assumption of FMs as a panacea for all data-related problems.

Delving deeper, the study identified a clear distinction in FM performance based on data characteristics:

1. FMs significantly and consistently outperform traditional IEs for subset selection on fine-grained datasets, irrespective of whether the data is clean or noisy. This is a crucial discovery, indicating that for tasks requiring discerning subtle differences between classes, FMs excel at identifying the most informative samples. Examples cited included datasets like CIFAR-100, Pet, and PetN (likely noisy Pet).
2. Conversely, FMs show limited and/or unstable advantages on coarse-grained datasets, particularly when noise is present. On datasets like CIFAR-10 (a coarse-grained dataset) and CIFAR-10N (a noisy version of CIFAR-10), FMs struggled to consistently outperform traditional IEs, and in some cases, traditional methods were clearly superior. This suggests that the generalization capabilities of FMs, while broad, may not always translate into optimal feature extraction for subset selection in simpler, less detailed classification tasks, especially when confronted with noise.

Beyond this primary distinction, two other critical insights emerged regarding the nature of Foundation Models themselves:

1. Different FMs perform differently across datasets, implying that no single foundation model works best everywhere. This highlights the need for careful selection of FMs or, more powerfully, a strategy to combine their strengths. The choice of FM should be informed by the specific characteristics of the downstream task and dataset.
2. A Foundation Model's superior performance on a downstream classification task does not guarantee its superiority for subset selection. This is a subtle but profound observation. An FM might be excellent at classifying images, but its internal representations might not be optimally structured for identifying the most representative or least redundant samples for training a new model from scratch. The objectives of direct classification and feature extraction for efficient subset selection are distinct.

These findings collectively motivate the next logical step: if no single FM is consistently optimal and FMs show particular promise on fine-grained data, can we leverage the diverse strengths of multiple FMs to push the boundaries of FM-based subset selection? This question led to the development of a novel multi-FM approach.

The result of this exploration is the proposed RAM-APL framework, a multi-FM-based subset selection method specifically tailored for fine-grained tasks. RAM-APL demonstrates state-of-the-art performance on fine-grained datasets, effectively combining the diverse feature extraction capabilities of models like CLIP and DINOv2. The study further investigated the optimal combination of FMs, revealing that while combining multiple FMs can yield better performance than any single model, there is a sweet spot. Specifically, using DINOv2 and CLIP together achieved the best trade-off between performance and efficiency, suggesting a practical two-model ensemble for this task.

In summary, the key contributions and findings are:

• A comprehensive empirical analysis comparing FMs and traditional IEs for one-shot subset selection, revealing nuanced performance depending on data granularity and noise.
• The demonstration that FMs are particularly effective for fine-grained subset selection.
• The insight that no single FM is universally superior for subset selection, and strong classification performance doesn't directly translate to strong subset selection performance.
• The proposal of RAM-APL, a novel multi-FM framework that achieves state-of-the-art results for fine-grained one-shot subset selection by strategically combining feature diversity.
• Identification of DINOv2 and CLIP as a highly effective pair for multi-FM subset selection, offering an optimal balance of performance and efficiency.

Technical Deep Dive

▶ Watch: Key conclusion: FMs excel on fine-grained, not coarse-grained data (14:00)

The core technical challenge addressed by this work lies in effectively leveraging the diverse representations generated by multiple Foundation Models for one-shot subset selection, particularly within the context of fine-grained datasets where FMs demonstrate significant potential. The goal is to select a subset of data that is maximally informative, representative, and minimizes redundancy, enabling a target model to be trained efficiently without performance degradation.

Traditional feature-based subset selection methods typically operate by extracting features from a single Information Extractor (IE) and then applying various criteria (e.g., core-set selection, diversity maximization) to these features. However, when working with multiple FMs, a fundamental issue arises: these models are pre-trained independently on different datasets with varying architectures and objectives, resulting in unaligned feature spaces. Simply concatenating features or applying a single scoring mechanism across these disparate spaces is suboptimal. Moreover, existing feature-based methods often focus either on the intra-data geometry (how samples relate to each other within their class or across the dataset) or inter-class decision boundaries (how distinct different classes are in the feature space). A holistic approach that integrates both perspectives is crucial for robust subset selection, especially in fine-grained scenarios where subtle distinctions matter.

To address these challenges, the authors introduce RAM-APL, a novel multi-FM-based subset selection framework. RAM-APL is designed to systematically integrate insights from multiple Foundation Models by considering both intra-class representativeness and inter-class ambiguity. The framework comprises two key strategies: Rank Aggregation Mapping (RAM) and Average Pseudo-Accuracy Loss (APL), which are then combined to derive a final importance score for each sample.

Rank Aggregation Mapping (RAM)

RAM is designed to capture the intra-class representativeness of a sample across multiple FM feature spaces. The core idea is to transform potentially unaligned feature vectors from different FMs into a uniform ranking space. For each Foundation Model $FM_k$, and for each class $c$, RAM first computes a representativeness score for every sample $x_i \in c$. This representativeness could be based on various metrics, such as the sample's distance to the class centroid in the FM's feature space, or its density within the class cluster. More central or dense samples are considered more representative.

Once individual representativeness scores are computed for each sample within each class for each FM, these scores are converted into ranks. For instance, within a specific class and for a given FM, the most representative sample might receive rank 1, the second most representative rank 2, and so on. This ranking process normalizes the scores across different FMs, effectively mapping them into a comparable space, irrespective of the underlying feature vector magnitudes or distributions.

Finally, RAM aggregates these ranks across all utilized FMs. This aggregation can be done using various rank aggregation techniques, such as Borda count, or simply by averaging the ranks. The aggregated rank then serves as the intra-class representativeness score for each sample, indicating how consistently a sample is deemed representative by the ensemble of FMs. Samples with lower aggregated ranks (i.e., consistently ranked as highly representative) are considered more valuable for inclusion in the subset.

Average Pseudo-Accuracy Loss (APL)

While RAM focuses on intra-class representativeness, APL addresses the inter-class ambiguity by measuring how well a sample can be classified by simple classifiers trained on features from multiple FMs. The concept of "pseudo-accuracy" is central here. For each Foundation Model $FM_k$, a lightweight classifier (e.g., a linear probe or k-NN classifier) is trained on the FM's features extracted from the entire dataset. This classifier then predicts the "pseudo-label" and associated confidence for each sample.

APL measures the "pseudo-accuracy" of a sample across these multiple FM feature spaces. Specifically, for each sample $x_i$, APL calculates how often $x_i$ is correctly classified by the pseudo-classifiers derived from each FM. A sample that is consistently misclassified or classified with low confidence across multiple FMs indicates high inter-class ambiguity or lies close to a decision boundary. Conversely, a sample that is consistently correctly classified with high confidence across FMs is unambiguous.

To derive a "loss" or "ambiguity" score, APL typically computes the average loss (e.g., cross-entropy loss) or one minus the average pseudo-accuracy for a sample across all FMs. Samples with higher APL scores are those that are more ambiguous or harder to classify, suggesting they might be crucial for refining decision boundaries and improving generalization, especially in fine-grained tasks where class distinctions are subtle.

Final Importance Score

The final importance score for each sample in RAM-APL is a weighted combination of its RAM score (intra-class representativeness) and its APL score (inter-class ambiguity). The weighting allows the framework to balance these two critical aspects of data informativeness. For example, a sample that is both highly representative within its class (low RAM score) and moderately ambiguous (moderate APL score) might be considered highly valuable. The specific weights can be tuned based on validation performance or heuristic understanding of the dataset.

The intuition behind combining RAM and APL is that a good subset should ideally contain samples that are:

1. Representative: Anchoring the core of each class's distribution (captured by RAM).
2. Challenging but informative: Helping the model learn to distinguish between similar classes (captured by APL).

By integrating these two perspectives, RAM-APL aims to select a subset that not only covers the main data distribution but also includes crucial boundary-case examples, leading to more robust and accurate target models.

Multi-FM Strategy and Optimal Combination

The framework's ability to leverage multiple FMs is critical. The talk highlights that different FMs (e.g., CLIP, DINOv2, SigLIP, EVA-CLIP) capture diverse semantic and visual features due to their distinct pre-training objectives and architectures. For instance, CLIP is trained on image-text pairs, excelling at zero-shot generalization and high-level semantic understanding, while DINOv2 is a self-supervised vision transformer, focusing on dense local features and robust visual representations. Combining such diverse models provides a richer, more comprehensive understanding of the data.

The study also investigated the optimal number and combination of FMs. While using more FMs can generally lead to better performance, there's a point of diminishing returns in terms of efficiency. The empirical results showed that a combination of DINOv2 and CLIP together achieved the best trade-off between performance gain and computational efficiency. This suggests that these two specific FMs offer a complementary set of features that are particularly effective for fine-grained subset selection using RAM-APL, making them a default recommendation for practical implementation.

In essence, RAM-APL provides a principled methodology to navigate the complexities of multi-FM feature spaces, enabling the construction of highly effective and efficient subsets for fine-grained tasks by considering both the typical and challenging aspects of data distribution.

Experimental Setup & Results

▶ Watch: Motivation for multi-FM approach; no single model is best (16:00)

The research conducted a comprehensive empirical study to rigorously evaluate the performance of Foundation Models and the proposed RAM-APL framework for one-shot subset selection. The evaluation spanned a diverse set of conditions to provide robust insights into the applicability and limitations of these approaches.

Datasets

The evaluation utilized five distinct image datasets, carefully chosen to represent a wide range of data characteristics, particularly focusing on granularity and noise levels:

• CIFAR-10: A coarse-grained dataset with 10 classes of common objects.
• CIFAR-10N: A noisy version of CIFAR-10, used to assess robustness to label noise in coarse-grained settings.
• CIFAR-100: A fine-grained dataset with 100 classes, representing more subtle distinctions between categories.
• Pet (Oxford-IIIT Pet Dataset): A fine-grained dataset for pet breed classification, known for its visual similarities between classes.
• PetN: A noisy version of the Pet dataset, used to test performance on fine-grained data with label noise.

This selection allowed for a direct comparison of subset selection strategies across varying levels of class granularity and data quality.

Information Extractor (IE) Types

Three main categories of Information Extractors were tested to benchmark different approaches:

1. TD (Traditional IEs): These are models pre-trained specifically on the full target dataset. For example, a ResNet trained on the complete CIFAR-10 dataset would serve as the TD for CIFAR-10. This represents the conventional, dataset-dependent approach.
2. TIN (Tiny ImageNet models): These models are pre-trained on a large external dataset, specifically Tiny ImageNet. This category serves as an intermediate baseline, demonstrating the performance of models pre-trained on a general, but not massive, external dataset.
3. Single FM (Foundation Models): This category includes several state-of-the-art Foundation Models, used directly as feature extractors without any task-specific fine-tuning. The FMs evaluated were:

• DINOv2
• CLIP
• SigLIP
• EVA-CLIP

Subset Selection Methods and Sampling Rates

For each dataset and each extractor type, four generic feature-based subset selection methods were applied. While the specific names of these methods were not explicitly detailed in the transcript, they are implied to be standard techniques that leverage extracted features to identify informative samples (e e.g., diverse sampling, uncertainty sampling, core-set selection). The selection was performed across three different sampling rates: 10%, 30%, and 50% of the full dataset size. This allowed for an analysis of how performance scales with the size of the selected subset.

Performance Metrics

The evaluation focused on two complementary perspectives:

• Best Extractor Frequency: This metric captured how consistently each extractor type (TD, TIN, Single FM) yielded the best performance (highest accuracy) across all settings (combinations of dataset, selection method, and sampling rate). It measured the "number of wins" for each extractor type.
• Performance Dominance: This metric assessed whether a particular extractor achieved the best accuracy for a specific sampling rate on a given dataset, providing a more granular view of where each method excelled.

Initial Experimental Results (FM vs. Traditional IE)

The initial findings from this extensive comparison were nuanced:

• Overall: The surprising initial conclusion was that FMs do not always outperform traditional IEs. Traditional IEs, despite their limitations, performed better in several specific cases.
• CIFAR-10N (Noisy Coarse-Grained): The single FM was preferred in only 4 out of 12 settings (4 selection methods * 3 sampling rates). This highlighted the limited and unstable advantage of FMs on noisy, coarse-grained data.
• CIFAR-10 (Clean Coarse-Grained): The FM never achieved the best accuracy, demonstrating its weakness on clean, coarse-grained data.
• CIFAR-100 (Clean Fine-Grained): The FM showed dominance at 30% and 50% sampling rates but not at 10%. This indicates that for very small subsets of fine-grained data, FMs might still struggle, but their advantage becomes clear as the subset size increases.
• Pet and PetN (Clean and Noisy Fine-Grained): Critically, the FM consistently dominated across all three subset ratios (10%, 30%, 50%) on both the clean and noisy fine-grained Pet datasets. This provided strong empirical evidence for the FMs' superior performance in fine-grained scenarios.

These results empirically validated the two key conclusions: FMs significantly and consistently outperform traditional IEs for subset selection on fine-grained datasets (both clean and noisy), but show limited or unstable advantages on coarse-grained datasets, especially when noise is present. The observation that different FMs perform differently and that their classification superiority doesn't guarantee subset selection superiority further underscored the complexity.

RAM-APL Experimental Results

Motivated by these insights, the multi-FM RAM-APL framework was tested:

• Baselines: RAM-APL was compared against 12 strong baselines. While the specific baselines were not enumerated, they are implied to be other state-of-the-art or common subset selection methods, including those based on single FMs or traditional IEs.
• Performance: RAM-APL consistently outperformed all 12 baselines across three fine-grained datasets (likely CIFAR-100, Pet, and PetN, where FMs previously showed promise) and all tested sampling rates (10%, 30%, 50%). This establishes RAM-APL as a state-of-the-art method for fine-grained one-shot subset selection.
• Multi-FM Efficacy: The study also investigated the impact of using multiple FMs. The results clearly demonstrated that combining multiple FMs can yield better performance than any single model. This validated the core hypothesis behind RAM-APL's design.
• Optimal Combination: Further analysis revealed that the combination of DINOv2 and CLIP achieved the best trade-off between performance gains and computational efficiency. This pragmatic finding suggests a highly effective default configuration for RAM-APL in practice.

In summary, the experimental results provide compelling evidence for the nuanced role of FMs in subset selection, precisely delineating their strengths and weaknesses. More importantly, they demonstrate the significant performance uplift achieved by the proposed multi-FM RAM-APL framework, particularly for challenging fine-grained tasks.

Practical Implications

▶ Watch: RAM-APL's superior performance and insights on combining FMs (20:00)

The findings from this research carry substantial practical implications for various stakeholders in the machine learning ecosystem, from individual model builders to large infrastructure teams. The core message is clear: the intelligent use of Foundation Models for subset selection can significantly enhance efficiency and performance, but a nuanced, context-aware approach is essential.

For Practitioners and Model Builders

• Context-Aware FM Selection: Model builders should not blindly adopt Foundation Models for all subset selection tasks. The study clearly demonstrates that FMs excel on fine-grained datasets (e.g., medical imaging, detailed object recognition, specific animal breeds) but show limited or unstable advantages on coarse-grained datasets (e.g., general object classification like CIFAR-10), especially when data is noisy. Practitioners must first understand the granularity and quality of their data. For coarse-grained tasks, traditional IE-based methods or simpler sampling techniques might still be competitive or even superior.
• Leveraging Multi-FM Approaches for Fine-Grained Tasks: For applications involving fine-grained data, the RAM-APL framework offers a significant performance boost. Practitioners dealing with complex visual distinctions should consider implementing multi-FM strategies. The finding that DINOv2 and CLIP together offer an optimal performance-efficiency trade-off provides a concrete starting point, reducing the need for extensive experimentation with numerous FMs. This means model builders can achieve higher accuracy with smaller training sets, directly reducing training time and computational costs.
• Reduced Retraining Costs and Increased Flexibility: The zero retraining cost associated with FMs (compared to traditional IEs) translates into massive savings in time and computational resources. When new data arrives or datasets are updated, practitioners can simply re-run the one-shot subset selection process using the pre-trained FMs without needing to retrain an IE, dramatically accelerating iteration cycles and increasing workflow flexibility.
• Focus on Data Quality over Quantity: The research reinforces the notion that quality trumps sheer quantity. By using sophisticated subset selection techniques like RAM-APL, model builders can achieve comparable or superior performance with significantly smaller, more informative datasets. This allows for more focused data curation efforts, potentially leading to better generalization and reduced bias.

For Infrastructure Teams and Deployers

• Optimized Resource Utilization: Training deep learning models is computationally intensive. By training on intelligently selected subsets, infrastructure teams can observe substantial reductions in GPU/TPU hours, memory usage, and storage requirements. This directly translates to lower operational costs, reduced carbon footprint, and the ability to train more models or larger models within existing resource constraints.
• Scalable Data Pipelines: One-shot subset selection, especially with FMs, simplifies data pipelines. Once the subset is selected, the full dataset is no longer strictly needed for training. This can streamline data governance, reduce data transfer bottlenecks, and potentially enable training in environments with limited storage or bandwidth after the initial selection phase.
• Faster Experimentation and Deployment: Reduced training times mean faster iteration cycles for researchers and faster deployment times for production models. Infrastructure can be configured to support rapid subset generation and model training, accelerating the entire ML development lifecycle.

Tradeoffs and Limitations

• Initial Setup Complexity: While FMs simplify the IE aspect, implementing a multi-FM framework like RAM-APL introduces some initial complexity in terms of integrating multiple FM feature extractors and implementing the RAM and APL scoring mechanisms. However, once set up, the benefits outweigh this initial effort.
• Computational Cost of Multiple FM Inferences: While FMs are not retrained, running inference through multiple large FMs for all samples in the full dataset during the selection phase still incurs a one-time computational cost. This cost must be weighed against the subsequent savings in training time.
• Generality Across Modalities: The research focuses on image datasets. While the principles of FM-based subset selection might extend to other modalities (e.g., text, audio), further research would be needed to validate RAM-APL's effectiveness in those domains.
• Hyperparameter Tuning: The weighting between RAM and APL components in the final importance score might require tuning for optimal performance on specific datasets, adding a layer of hyperparameter search.
• Specific FM Choices: While DINOv2 and CLIP were identified as an optimal pair, the rapidly evolving landscape of FMs means that future, even more powerful combinations might emerge, requiring continuous re-evaluation.

In conclusion, this work provides a robust framework and empirical guidance for making data-efficient deep learning a reality. By offering a nuanced understanding of FM performance and a powerful multi-model solution, it empowers practitioners to build more performant models with fewer resources, ultimately accelerating progress in AI/ML deployment.

Key Takeaways

• Diminishing Returns of Data: Simply adding more data to deep learning models often leads to diminishing performance returns and significantly higher costs due to redundancy, noise, and imbalance. Efficient subset selection is crucial to address this.
• Foundation Models for One-Shot Subset Selection: Foundation Models (FMs) offer a scalable and cost-effective solution for one-shot subset selection by acting as dataset-independent information extractors, eliminating the need for costly IE retraining.
• Context is King for FM Performance: FMs demonstrate significant and consistent advantages for subset selection on fine-grained datasets (both clean and noisy), but show limited or unstable benefits on coarse-grained datasets, especially in the presence of noise.
• No Single FM is Universally Optimal: Different FMs possess varying strengths, and a model's superior performance in downstream classification does not guarantee its superiority for subset selection. This necessitates a strategic, possibly multi-model, approach.
• RAM-APL: A Multi-FM Breakthrough: The novel RAM-APL framework, which combines Rank Aggregation Mapping (RAM) for intra-class representativeness and Average Pseudo-Accuracy Loss (APL) for inter-class ambiguity, achieves state-of-the-art performance for fine-grained one-shot subset selection.
• Optimal Multi-FM Combination: For practical implementations of multi-FM strategies, combining DINOv2 and CLIP offers the best trade-off between enhanced performance and computational efficiency.

About the Speaker(s)

The primary author of this work is Zhijing Wan from Wuhan University. The presentation was given on her behalf, indicating her significant contribution to this research. The other listed speakers, Zhixiang Wang, Zheng Wang, Xin Xu, and Shin'ichi Satoh, are co-authors of the paper, reflecting a collaborative effort in developing these insights and the RAM-APL framework.