Rectifying Privacy and Efficacy Measurements in Machine Unlearning: A New Inference Attack Perspective
Nima Naderloui
34th USENIX Security Symposium (USENIX Security '25) · Day 3 · ML and AI Privacy 2
This talk, presented by Nima Naderloui, addresses critical flaws in the current evaluation frameworks for **machine unlearning** algorithms. While the field has seen an "explosion" of inexact unlearning methods, often showing incremental improvements, Naderloui argues that this apparent success stems not from the methods themselves, but from limitations in how they are assessed. The research highlights that existing evaluation metrics, primarily relying on average-case **Membership Inference Attacks (MIAs)**, are inherently weak and fail to capture the true privacy leakage and unlearning efficacy.
AI review
Solid, technically rigorous work that exposes a genuine methodological failure in how the field validates machine unlearning — the core insight that average-case MIAs are structurally incapable of catching per-sample leakage on vulnerable data is correct, important, and not obvious. The 69% MIA success rate on Tiny ImageNet against algorithms previously benchmarked below 10% is the kind of result that should make people uncomfortable, which is exactly what good measurement research does.