Evaluating and developing machine learning models: äN introduction

äN

Gulaschprogrammiernacht 24 (GPN24) · Day 1 · ZKM Medientheater

Overview

In a compelling presentation at GPN24, äN, a graduate student from KIT, delivered an insightful "introduction" to the evaluation and development of machine learning models that went far beyond basic concepts. Titled "Evaluating and developing machine learning models: äN introduction," the talk served as a practical guide for aspiring researchers and developers looking to navigate the often unstructured landscape of ML innovation. äN, bringing over four years of experience with PyTorch and having worked on more than 20 models across diverse domains including video prediction, time series forecasting, image classification, generative image modeling, and natural language processing, shared a wealth of practical advice and philosophical insights.

Watch on YouTube

Visual summary for Evaluating and developing machine learning models: äN introduction by äN
Visual summary for Evaluating and developing machine learning models: äN introduction by äN

Key moments

  1. 0:00 Introduction and personal motivation for ML
  2. 7:50 Minimum requirements for evaluating an ML model
  3. 13:20 Research vs. production: Prioritizing iteration over perfect code
  4. 18:50 What to record and why in ML experiments
  5. 25:10 Open challenge: Model expressivity and its limitations
  6. 35:00 Q&A: How to form the right research baseline
  7. 43:40 Q&A: Solving compute problems in private research

Evaluating and developing machine learning models: äN introduction

Speakers: äN, Graduate Student, KIT

Conference: GPN24

YouTube: https://www.youtube.com/watch?v=MH_CUrValC4

Overview

In a compelling presentation at GPN24, äN, a graduate student from KIT, delivered an insightful "introduction" to the evaluation and development of machine learning models that went far beyond basic concepts. Titled "Evaluating and developing machine learning models: äN introduction," the talk served as a practical guide for aspiring researchers and developers looking to navigate the often unstructured landscape of ML innovation. äN, bringing over four years of experience with PyTorch and having worked on more than 20 models across diverse domains including video prediction, time series forecasting, image classification, generative image modeling, and natural language processing, shared a wealth of practical advice and philosophical insights.

The core of the talk revolved around fostering independent research, promoting scientific rigor, and demystifying the development process. äN likened the field of machine learning to "cowboy math"—an untamed frontier where new discoveries abound, offering immense opportunities for creativity and significant contributions. This perspective underscores the speaker's belief that meaningful advancements are not solely confined to large institutions but are accessible to motivated individuals willing to delve deep into the scientific literature and embrace an iterative, experimental mindset. The presentation is particularly relevant for anyone seeking to move beyond superficial understanding to actively developing and evaluating novel ML solutions, emphasizing the critical importance of robust methodology, meticulous record-keeping, and a keen eye for unresolved challenges.

Background

▶ Watch: Introduction and personal motivation for ML (0:00)

äN's journey into machine learning was sparked by childhood fascinations with science fiction, citing Lieutenant Commander Data from Star Trek: The Next Generation and the futuristic societies envisioned by Isaac Asimov as early inspirations. This early exposure to artificial intelligence's potential, coupled with the inherent problems and complexities it presented, laid the groundwork for a career dedicated to understanding and shaping the field. Upon entering university, äN found machine learning to be a realm of "cowboy math"—a wild west of discovery where one could forge new paths and make original contributions, a stark contrast to more established, less flexible research domains.

The speaker highlighted a critical gap in the current learning landscape: while abundant online resources offer superficial introductions, they often fall short of providing the scientific understanding necessary for genuine innovation. True understanding, äN argued, comes from meticulously reading scientific papers, most of which are freely available on platforms like arXiv. This process, even if initially challenging, gradually builds the capacity to grasp complex concepts, make novel connections between ideas, and formulate new research questions. Beyond theoretical understanding, äN stressed the importance of examining the repositories associated with published papers. These often contain high-quality code and standardized data preprocessing methods, providing a crucial starting point for replicating and extending research.

However, not all papers or repositories are created equal. äN advised checking author credibility (e.g., via H-index and I-index), prioritizing peer-reviewed conference papers, and reading them meticulously to extract crucial keywords for future searches. A subtle but significant point was made regarding the nuanced definitions of similar terms, such as hypernetworks and dynamic neural networks, whose boundaries can shift between authors. This deep contextual understanding, äN noted, is something that current Large Language Models (LLMs) struggle to provide, underscoring the irreplaceable value of human scientific inquiry. This foundational approach—rooted in deep paper comprehension and adherence to research best practices—forms the bedrock for effective ML model development and evaluation.

Key Findings

▶ Watch: Research vs. production: Prioritizing iteration over perfect code (13:20)

The talk outlined several critical findings and best practices for anyone engaging in machine learning research and development. äN emphasized that the minimum requirements for evaluating a model include a data set, the model itself, a baseline (preferably implemented within the same framework for comparability), an optimizer, appropriate evaluation metrics, and robust recording mechanisms. The choice of metrics is crucial, as training metrics may not always align with evaluation goals.

A significant finding concerned the accessibility of ML coding. Python, being a high-level scripting language, combined with frameworks like PyTorch, lowers the barrier to entry. However, äN cautioned against blind trust, advising developers to write short scripts to understand ambiguous operator behaviors and test performance for speed optimization, as seemingly interchangeable operators can have vastly different runtimes and memory footprints. A notable anecdote involved free LLMs failing to produce good code for math-heavy tasks like high-dimensional tensor computations, reinforcing the need for manual implementation and verification.

A strong warning was issued against relying on generic online repositories: "Do not fall for these repos." Many, despite popularity, hide errors, fail to adhere to research standards, or use non-standard implementations. A prime example cited was the Fréchet Inception Distance (FID) score in generative image modeling, which is numerically sensitive to its implementation. Using a non-standard FID implementation renders results incomparable. The recommendation was clear: use repositories directly from published papers or, ideally, build your own, as much of ML code is boilerplate.

For testing, äN proposed a pragmatic approach balancing meticulousness with rapid development. Static tests are suitable for unchanging components like data sets and splits. For evolving model implementations, lightweight Python scripts (20-30 lines) are preferred to quickly verify mathematical correctness, allowing for easy iteration and sharing. This "research not production" philosophy permits code quality to be secondary to the ability to iterate and test ideas quickly, with rewriting code encouraged for gaining experience and improving extensibility.

Regarding computational complexity, äN found existing flop counters in Python to be unreliable, yielding different answers due to varying sets of operations they account for. Alternatives include theoretical flop counting (pen and paper), measuring runtime and memory utilization, or using system-level GPU monitoring. However, flops alone don't tell the full story, as highly parallelized code, despite higher flop counts, can be significantly faster on modern hardware.

Finally, effective research hinges on meticulous record-keeping and data analysis. Tools like Weights & Biases were recommended for automatically saving results, including full model, data, and training configurations, alongside code versions and changes. This prevents confusion when revisiting experiments. äN also advocated for developing custom metrics if common ones fail to capture meaningful improvements. Crucially, analyzing results qualitatively and quantitatively, looking for patterns, and constantly questioning experimental validity leads to new hypotheses and a continuous flow of ideas, emphasizing that both positive and negative results are vital for updating hypotheses and furthering research.

Technical Deep Dive

▶ Watch: What to record and why in ML experiments (18:50)

The talk delved into several fundamental open challenges in machine learning, highlighting areas ripe for independent research and potential breakthroughs. äN emphasized that many of these problems can be tackled with minimal computational resources, often on a CPU, making them accessible to individual researchers.

One significant challenge discussed was expressivity. äN presented examples of very small, CPU-runnable models (with hidden sizes of 200 or less) attempting to approximate simple target distributions, such as independent standard normal distributions or a circle. These experiments, which can be replicated at home, reveal fundamental limitations of common architectures. For instance, single-layer feed-forward neural network blocks (often used between attention blocks in modern Transformer models) struggle with certain patterns, producing very sharp or only bend lines, failing to approximate smooth or complex distributions effectively, especially without bias terms. The speaker also touched upon emerging architectures like those based on Kolmogorov-Arnold Networks (KANs), which introduce spline activation functions and show promising expressive capabilities, though current formulations might lack linear output projections leading to sharp edges.

A second critical problem is state tracking, particularly within Recurrent Neural Networks (RNNs). äN referenced a 2024 paper, "The illusion of state and state space models," which demonstrates that certain classes of RNNs cannot solve even simple state tracking problems. An illustrative example involves tracking the position of a "man in a field" given a long sequence of directions interspersed with noise. Even larger models struggle with this task, despite its simplicity, while very basic models can solve it. This presents an opportunity to combine the strengths of different model classes, potentially leading to significant advancements in the field. This problem, too, can be investigated on a CPU due to its low computational demands.

The context window length in time series prediction was identified as another perplexing issue. While one might expect model performance to improve with more historical data, increasing the context window size beyond a certain point often leads to performance degradation. The question arises: why can't models effectively leverage more information, even when the necessary patterns are present? This challenge is also highly amenable to independent research on a CPU, as time series datasets are typically small, often mere megabytes.

Lastly, the fundamental question of activation functions remains largely open. While numerous activation functions exist, many are outdated or only relevant in specific domains. The emergence of Comorb networks and their spline activation functions highlights ongoing innovation in this area. The core question is: what kind of non-linear function most benefits neural networks as their activation? äN encouraged researchers to experiment with existing functions, analyze their effects on various datasets (including small synthetic ones), and even invent new ones, emphasizing that a simple, novel activation function could lead to the "next big thing."

äN concluded this section by noting that many significant breakthroughs in machine learning can often be encapsulated in "five lines of code or less," a powerful motivator for independent exploration. The discussion then briefly touched upon Artificial General Intelligence (AGI), acknowledging that a perfect solution remains elusive due to immense computational complexity and our incomplete understanding of the human brain, making concrete predictions about its arrival highly speculative.

Demo / Proof of Concept

▶ Watch: Q&A: How to form the right research baseline (35:00)

While the talk did not feature a live, interactive demo in the traditional sense, äN presented several visual examples and conceptual experiments that served as illustrative proofs of concept. Specifically, the discussion around expressivity (0:25:00) showcased various small model architectures attempting to approximate simple 2D distributions (like a standard normal distribution or a circle). These visual representations, depicting the models' outputs in different colors and patterns, effectively demonstrated the inherent limitations of certain fundamental building blocks, such as single-layer feed-forward neural networks, in capturing specific mathematical behaviors. äN explicitly referred to these as "experiments you can do at home" on a CPU, emphasizing their accessibility for independent verification and exploration. Similarly, the conceptual problem of state tracking (0:27:00) and the context window length issue in time series prediction (0:29:00) were presented with clear diagrams and examples, serving as thought experiments that researchers could easily set up and test using readily available resources.

Defensive Implications

▶ Watch: Q&A: Solving compute problems in private research (43:40)

Although äN's talk primarily focused on the development and evaluation of machine learning models from a research perspective rather than direct cybersecurity, the insights offered have significant indirect defensive implications for building robust and trustworthy ML systems.

Firstly, the emphasis on meticulous model evaluation and the use of appropriate baselines is fundamental. Understanding how a model performs against established benchmarks, and critically, ensuring these comparisons are valid (e.g., using the same framework and adhering to research standards for metrics like FID score), helps prevent the deployment of underperforming or misleadingly evaluated models. In a defensive context, this means ensuring that ML-driven security tools (like anomaly detection or malware classification) are genuinely effective and not just appearing to be so.

Secondly, the strong caution against using unreliable open-source repositories directly impacts security. Fabricated details, hidden errors, or non-standard implementations in such repositories can introduce subtle vulnerabilities or performance issues that are hard to detect. For defenders, integrating components from untrusted or unverified sources into security products could lead to exploitable weaknesses or unreliable threat intelligence. äN's advice to use repositories from peer-reviewed papers or to build one's own code reinforces the principle of supply chain security for ML components.

The discussion on open challenges like expressivity and state tracking provides critical insights into the inherent limitations of current ML architectures. For defenders, understanding these limitations is crucial for anticipating potential failure modes or blind spots in ML-powered security systems. For example, if a model struggles with state tracking, it might be vulnerable to evasion techniques that exploit its inability to maintain context over time. Awareness of these weaknesses can inform the design of more resilient systems or compensatory defensive layers.

Finally, the call for meticulous recording of results and versioning of code changes is a cornerstone of auditable and explainable AI. In a security incident, understanding exactly which model configuration, data, and code version produced a particular outcome is paramount for forensic analysis, debugging, and demonstrating compliance. This practice enhances the transparency and accountability of ML systems, which are increasingly important for regulatory and trust purposes in security-critical applications. By adopting äN's recommended research practices, organizations can build more reliable, verifiable, and ultimately, more secure machine learning defenses.

Key Takeaways

  • Deep Scientific Understanding is Paramount: Move beyond shallow online resources by meticulously reading and deeply understanding peer-reviewed scientific papers. Actively seek out associated code repositories to ensure adherence to research standards and proper data preprocessing.
  • Exercise Caution with Open-Source Code: Avoid generic online repositories that may contain errors or non-standard implementations (e.g., sensitive metrics like FID). Prioritize code directly from published papers or, ideally, develop your own foundational implementations.
  • Embrace Iteration Over Premature Optimization in Research: In a research context, prioritize rapid iteration and testing of ideas over perfect code quality or premature optimization. Use lightweight scripts for mathematical verification and be willing to rewrite code to gain experience and improve extensibility.
  • Meticulous Recording and Analysis of Experiments: Systematically record all model, data, and training configurations, along with code versions and changes, using tools like Weights & Biases. Actively analyze results both quantitatively and qualitatively to identify patterns, generate new hypotheses, and continuously refine your research direction.
  • Don't Hesitate to Define Custom Metrics: If standard evaluation metrics fail to capture meaningful improvements in your model's performance for a specific task, develop and utilize your own metrics that better reflect the problem's nuances.
  • Focus on Fundamental Open Challenges: Significant breakthroughs can arise from tackling core problems in ML, such as enhancing model expressivity, improving state tracking capabilities, understanding context window limitations in time series, or discovering optimal activation functions. Many of these can be explored with accessible computational resources.

About the Speaker(s)

äN is a graduate student at the Karlsruhe Institute of Technology (KIT), currently pursuing a doctoral program in the field of machine learning. With over four years of extensive experience using the PyTorch framework, äN has developed and evaluated more than 20 machine learning models across a wide array of domains, including video prediction, time series forecasting, image classification, generative image modeling, and natural language processing. Driven by a childhood fascination with artificial intelligence inspired by science fiction figures like Lieutenant Commander Data and the works of Isaac Asimov, äN views machine learning as a creative and largely unexplored field, akin to "cowboy math," where independent research and novel contributions are highly accessible. äN is passionate about fostering independent knowledge and skill development outside of traditional institutional structures.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

A competent research-methodology talk from someone clearly in the trenches with PyTorch, but it's fundamentally a 'how to do ML research' primer rather than novel security or ML content. Useful for grad students and hobbyists entering the field; nothing here that experienced practitioners haven't learned the hard way already.

Heather Calloway (CISO) — WEAK

This is an academic ML pedagogy talk aimed at graduate students learning to do research. It has no security content, no governance angle, and no relevance to enterprise security programs. Not my lane, and not relevant to my audience.

→ Top-rated talks at Gulaschprogrammiernacht 24 (GPN24)

All talks from Gulaschprogrammiernacht 24 (GPN24)