Position: Certified Robustness Does Not (Yet) Imply Model Security

Position: Certified Robustness Does Not (Yet) Imply Model Security

Speakers: Andrew C. Cullen, University of Melbourne; Paul MONTAGUE, University of Melbourne; Sarah Erfani, University of Melbourne; Benjamin Rubinstein, University of Melbourne

Conference: ICML 2025

YouTube: https://slideslive.com/39043886

Overview

In this thought-provoking position paper presented at ICML 2025, Dr. Andrew Cullen, alongside collaborators Paul Montague, Sarah Erfani, and Benjamin Rubinstein from the University of Melbourne and DST Group in Australia, challenges the prevailing perception of certified robustness in machine learning. The talk, titled "Certified Robustness Does Not (Yet) Imply Model Security," argues that while certified defenses offer theoretical guarantees against adversarial attacks, their current framing and application fall short of providing genuine security for real-world deployed systems. The core message is that the community's focus on purely technical metrics, often divorced from practical threat models and human factors, creates a dangerous misalignment between perceived and actual security.

The presentation critically examines the fundamental assumptions underlying certified robustness, particularly highlighting a "certification contradiction" where these defenses cannot distinguish between clean and adversarial inputs. This inherent limitation, coupled with an overemphasis on abstract mathematical norms (like L2 norm bounded certifications) that may not reflect real-world attack vectors, can lead to a false sense of security—what the authors term "security theater." The speakers advocate for a paradigm shift, urging the adversarial machine learning community to adopt a user-first perspective, learn from other security-focused domains like differential privacy, and address the "soft human issues" that are integral to building truly secure and reliable AI systems. This talk serves as a vital call to action for researchers to reconsider how certified robustness is conceptualized, developed, and communicated to ensure it genuinely contributes to robust and secure ML deployments.

Background

▶ Watch: Introduction to adversarial ML and certified defenses (0:00)

The landscape of machine learning security has been profoundly shaped by the emergence of adversarial machine learning (AML). As ML models become increasingly ubiquitous in critical applications, from autonomous vehicles to medical diagnostics, the threat of adversarial attacks—subtly perturbed inputs designed to cause misclassification—poses a significant risk to their reliability and trustworthiness. Early responses to these threats often involved developing adversarial defenses that were reactive and attack-specific. However, as Dr. Cullen points out, these defenses proved to be inherently brittle: a knowledgeable attacker could simply treat the defense as part of the model itself, effectively circumventing it by attacking the combined system. This led to a continuous arms race where new defenses were quickly broken by novel attacks, creating a cycle of reactive patching rather than proactive security.

In response to this challenge, the concept of certified defenses emerged as a more robust, "attack-agnostic" approach. The goal of certified robustness is not to defend against a specific attack, but rather to provide a mathematical guarantee (a "certificate") that a model's prediction will remain stable within a certain region around a given input. This region is typically defined by a distance metric, such as the L2 norm, and the size of this region is referred to as the certified radius. A larger certified radius implies greater robustness. Techniques like randomized smoothing became popular for achieving certified robustness due to their architecture independence and applicability across various systems. Randomized smoothing works by adding random noise (e.g., Gaussian noise) to an input and then classifying based on the majority vote of predictions over many noisy versions of the input. The statistical properties of this process allow for the derivation of a provable lower bound on the radius within which the classifier's prediction is guaranteed to remain unchanged.

However, the talk argues that this promise of certified robustness, while mathematically sound, suffers from a critical conceptual flaw: its framing. The certification literature typically presents the certified radius as the "distance to the nearest possible adversarial example." Dr. Cullen argues that this is only true if the input sample is known to be clean. If the sample itself is already an adversarial example, the certification mechanism is merely measuring how far that adversarial example is from the decision boundary, not its distance from a clean, correctly classified input. This fundamental inability of a certified defense to distinguish between a clean sample and an attack sample is what the speakers term the "certification contradiction." This contradiction highlights a crucial alignment issue between how these techniques are presented and how they are perceived, particularly by those seeking to deploy secure ML systems. The speaker further notes that certifications are often tied to specific, sometimes narrow, threat models (e.g., specific perturbation norms) that may not fully capture the complexity of real-world attacks. This gap between theoretical guarantees and practical security implications forms the core motivation for the paper's critical examination of the field.

Key Findings

▶ Watch: Misaligned perception of certifications creates security risks (3:00)

The central finding of this position paper is that certified robustness, as currently framed and implemented, does not yet imply genuine model security. This assertion stems from several critical observations and proposed questions for the research community:

1. The Certification Contradiction: A fundamental flaw exists in the interpretation of certified robustness. While framed as measuring the distance to the nearest adversarial example, this is only valid for clean samples. For an input that is already adversarial, the certification merely measures its distance from the decision boundary, failing to detect or mitigate the attack. This inherent inability to distinguish between clean and adversarial inputs undermines its utility as a security guarantee.
2. Perception vs. Reality Mismatch: The strong language of "guarantee" associated with certified robustness can lead to a dangerous misconception among users and deployers. Naive interpretation of these guarantees can foster a false sense of absolute security, potentially leading to security theater and complacency in system design. This alignment issue between how researchers present certifications and how users perceive them is a significant source of vulnerability.
3. Narrow Threat Models: Current certifications are often tightly coupled to specific, abstract threat models, predominantly L2 norm bounded certifications. The research community has not adequately investigated how these theoretical norms translate to real-world attack scenarios and human perception, raising questions about the practical relevance of achieving high certified radii under these specific constraints.
4. Flawed Measurement & Metrics: The prevailing practice of measuring progress by simply increasing the average certified radius (e.g., from 0.7 to 0.71) reflects a naive view of risk. This approach treats all samples as having equal risk, whereas in real-world systems, different inputs inherently carry different levels of risk and importance. A more nuanced, risk-differentiated approach to measurement is needed.
5. Information Leakage to Attackers: Paradoxically, certifications can potentially provide more useful information to an attacker than to a defender. By indicating regions where an adversarial example cannot exist (within the certified radius), certifications might inadvertently help attackers refine their search space for effective perturbations outside this certified zone.
6. Call for a User-First Perspective: The paper strongly advocates for a shift from purely technical problem-solving to a user-first perspective in developing secure ML systems. This involves considering how humans will develop, implement, test, and use these systems, moving beyond abstract technical metrics to focus on auditable, trustworthy, and usable solutions.
7. Lessons from Differential Privacy: The OpenDP initiative in the differential privacy community is highlighted as a successful model. OpenDP prioritized making differential privacy practical, auditable, and plug-and-play, even if it meant not always using the absolute state-of-the-art approaches. Their focus on trustworthiness and clear communication of assumptions offers a blueprint for the adversarial machine learning community.
8. Limited Scope: Model-Level Only: A critical limitation identified is that current certified robustness mechanisms operate almost exclusively at the model level. They do not account for security vulnerabilities in the broader data pipeline, system processes, or underlying hardware, leaving significant attack surfaces unaddressed.

Ultimately, the paper proposes a set of crucial questions for the community: How do users perceive certifications? How can ML learn from other communities that have successfully built secure systems? How can certifications be improved in their construction, presentation, testing, and comparison? And most importantly, how can they genuinely enhance real-world security?

Technical Deep Dive

▶ Watch: Key questions for the certified robustness community (3:50)

The technical core of certified robustness often revolves around providing mathematical guarantees against adversarial perturbations within a defined geometric space. A prominent technique enabling this is randomized smoothing. This method involves wrapping a base classifier, $f$, with a randomized procedure. For a given input $x$, instead of directly classifying $x$, randomized smoothing adds Gaussian noise (or other types of noise) to $x$ multiple times, obtaining a set of noisy samples $x' = x + \delta$, where $\delta \sim \mathcal{N}(0, \sigma^2 I)$. The smoothed classifier, $g(x)$, then outputs the class that is most frequently predicted by the base classifier $f$ across these noisy samples. Mathematically, $g(x) = \arg\max_{c \in \mathcal{Y}} P(f(x+\delta) = c)$, where $\mathcal{Y}$ is the set of classes.

The statistical properties of this majority vote, particularly when using Gaussian noise, allow for the computation of a provable lower bound on the certified radius $R$. For an input $x$ and a class $c_A$ predicted by the smoothed classifier $g(x)$, the certification guarantees that for any adversarial perturbation $\tilde{x}$ such that $||\tilde{x} - x||_2 < R$, the smoothed classifier $g(\tilde{x})$ will still predict $c_A$. The radius $R$ is typically derived using concentration inequalities (e.g., Chernoff bounds) or specific properties of Gaussian distributions, where the magnitude of $\sigma$ (the standard deviation of the noise) plays a crucial role: larger $\sigma$ generally leads to larger certified radii but can also reduce clean accuracy. While randomized smoothing is highly appealing due to its architecture independence (it can be applied to virtually any base classifier, including deep neural networks) and computational efficiency (relative to other certification methods), its guarantees are inherently tied to the chosen noise distribution and the L2 norm for measuring perturbation distance.

The talk, however, scrutinizes the interpretation and applicability of these technical guarantees. The "certification contradiction" highlights that the meaning of the certified radius $R$ fundamentally changes based on the nature of the input. If $x$ is a clean sample, $R$ indicates the maximum L2 distance an attacker can perturb $x$ before $g(x)$ might change its prediction. But if an attacker provides an input $x_{adv}$ that is already an adversarial example (e.g., crafted with an L2 perturbation slightly larger than $R$ from a clean image), and the smoothed classifier $g(x_{adv})$ still outputs a class $c_B$, then a certification run on $x_{adv}$ might still yield a radius $R'$. This $R'$ now signifies the distance from $x_{adv}$ to the decision boundary for class $c_B$, not a safe radius around a clean input. This distinction is critical because it means the defense does not inherently detect or flag an adversarial input; it merely reports a local robustness property, regardless of the input's origin.

Furthermore, the reliance on L2 norm bounded certifications is a major point of contention. While mathematically convenient and tractable for research, the L2 norm (Euclidean distance) often does not align with human perception of imperceptibility or with the types of perturbations that are semantically meaningful in real-world attacks. An attack with a small L2 norm might still be visually noticeable, or an attack with a slightly larger L2 norm but affecting only a few pixels (e.g., L0 norm) might be more potent or practical. Conversely, an L2 perturbation that is perceptually significant might not be captured by a certification focused on semantic integrity. The speaker cites Gilmer's work ("Motivating the Rules of the Game of Adversarial Machine Learning") which attempts to bridge this gap, but emphasizes that these connections are not yet well-understood or systematically integrated into certification research.

The critique extends to the very metrics used for progress. The common practice of reporting an improvement in average certified radius (e.g., from 0.7 to 0.71) is seen as a "naive view of risk." This metric implicitly assumes that all data points are equally important and carry the same level of risk. In real-world systems, however, certain inputs (e.g., a stop sign in an autonomous driving context, or a medical image diagnosis) are far more critical than others. A small improvement in the average radius across a dataset might mask significant vulnerabilities in critical edge cases, or it might be achieved by sacrificing robustness on less important samples. A truly secure system would require differentiated risk assessment and metrics that reflect the varying criticality of different inputs.

Finally, the talk draws a parallel to the differential privacy (DP) community and the OpenDP initiative. DP, like certified robustness, originated as a highly abstract, mathematical concept. OpenDP's success lies in its transition from theoretical guarantees to practical, auditable, and usable code. They prioritize trustworthy code, plug-and-play functionality, and clear communication of assumptions, often opting for established, well-understood methods over the cutting edge if it enhances audibility and reliability. This approach, which emphasizes engineering for security and usability over raw performance metrics, is presented as a crucial blueprint for the adversarial machine learning community to adopt, moving beyond isolated model-level guarantees to a more holistic, system-wide security posture.

Experimental Setup & Results

▶ Watch: Why 'average certified radius' is a naive risk metric (6:00)

This talk presents a position paper rather than empirical research; therefore, it does not include a traditional experimental setup, specific datasets, baselines, hardware, or headline numbers from new experiments. The primary objective of the presentation is to critically evaluate the current state of certified robustness research, highlight its conceptual limitations, and propose a new direction for the community.

Instead of presenting novel results, the talk analyzes existing practices and literature within the field of certified robustness. It references common metrics used in research, such as the average certified radius, and discusses the implications of focusing solely on these metrics without considering real-world context. The speaker's arguments are built upon observations of how certifications are typically constructed, presented, and perceived within the academic community and by potential deployers. The paper's contribution lies in its analytical framework and the critical questions it poses, aiming to guide future research towards more practically relevant and secure outcomes, rather than reporting on specific model performance or attack success rates.

Practical Implications

▶ Watch: Adopting a user-first perspective for secure systems (OpenDP) (6:40)

The insights presented in this talk carry significant practical implications for anyone involved in building, deploying, or relying upon machine learning systems, from individual practitioners to large infrastructure teams. The core message that certified robustness does not yet imply model security demands a re-evaluation of current security strategies and expectations in the face of adversarial threats.

For model builders and researchers, the immediate implication is a need to broaden the scope of their work beyond merely achieving higher average certified radii or improving performance under specific L2 norm bounded certifications. The talk suggests that this narrow focus can be misleading, as these metrics often fail to translate meaningfully to real-world threat models. Instead, researchers should prioritize understanding how certifications are perceived by end-users and how they align with practical security requirements. This might involve developing new metrics that account for varying levels of risk across different data samples or exploring certification techniques that are more robust to a wider array of perturbation types (e.g., semantic attacks, L0/Linf norms) that are more relevant to real-world adversaries.

For infrastructure teams and deployers, the most critical takeaway is to avoid a false sense of security. Relying solely on a model's certified robustness as a guarantee of security can lead to security theater, where the appearance of security masks underlying vulnerabilities. This complacency can be particularly dangerous in high-stakes applications. Instead, deployers must adopt a holistic, full-stack security approach. As highlighted in the Q&A, current certifications are almost exclusively model-level only, neglecting the broader data pipeline, pre-processing steps, post-processing, system integrations, and even hardware vulnerabilities. True security requires addressing these layers comprehensively, recognizing that a certified model is just one component within a larger, potentially vulnerable system.

The talk also underscores the importance of transparency and clear communication of assumptions. Like the OpenDP initiative for differential privacy, the adversarial ML community needs to make the limitations and specific conditions of certified guarantees explicitly clear to users. This includes detailing the specific threat models covered, the type and magnitude of perturbations considered, and the inherent "certification contradiction." This level of clarity can prevent misinterpretation and help practitioners make informed decisions about where and how to deploy certified models, understanding their specific strengths and weaknesses.

Furthermore, the call for a user-first perspective suggests a shift in design philosophy. Instead of abstract mathematical guarantees, the emphasis should be on building auditable, trustworthy, and plug-and-play systems. This means designing ML security components that are easy to integrate, whose behavior is predictable, and whose underlying assumptions are well-documented and verifiable. This approach, while potentially requiring a trade-off with "state-of-the-art" performance in some benchmarks, can lead to more robust and genuinely secure deployments in practice.

Finally, the talk implicitly encourages a more critical and interdisciplinary approach to ML security. Learning from established security engineering practices and other communities like differential privacy can provide valuable blueprints for transitioning from research code to production-ready secure systems. This includes considering the "soft human issues"—how human developers, implementers, and users interact with and interpret these security mechanisms—as these factors are often the ultimate determinants of a system's real-world security posture.

Key Takeaways

• Certified robustness, as currently framed, does not imply model security. The "certification contradiction" highlights that certified defenses cannot distinguish between clean and adversarial inputs, fundamentally limiting their security guarantees.
• The perception of absolute security from certifications is dangerous. Over-reliance on "guarantees" can lead to a false sense of security, creating "security theater" and fostering complacency in system design.
• Current metrics and threat models are often misaligned with real-world needs. Focusing solely on increasing "average certified radius" under abstract L2 norm bounded certifications ignores practical attack vectors, human perception, and the differentiated risk of various data samples.
• A "user-first" perspective is crucial for building genuinely secure ML systems. The community should learn from initiatives like OpenDP in differential privacy, prioritizing auditable, trustworthy, and usable solutions with clear communication of assumptions over purely technical performance metrics.
• Current certifications are limited to the model level. They do not address full-stack security vulnerabilities in the broader data pipeline, system processes, or hardware, leaving significant attack surfaces unaddressed.
• Addressing "soft human issues" is essential. Researchers must consider how humans will develop, implement, test, and use these systems, as these factors are critical for translating theoretical robustness into practical security.

About the Speaker(s)

Dr. Andrew Cullen is affiliated with the University of Melbourne. He presented this work in collaboration with Paul Montague, Sarah Erfani, and Benjamin Rubinstein, who are also associated with the University of Melbourne and the DST Group in Australia. The talk reflects their collective research interests in the practical implications and limitations of adversarial machine learning and certified robustness for real-world system security.