SoK: Prudent Evaluation Practices for Fuzzing
Moritz Schloegel, Nils Bars, Nico Schiller, Lukas Bernhard, Tobias Scharnowski, Addison Crump
IEEE Symposium on Security and Privacy 2024 · Day 2 · Continental Ballroom 4
This talk, presented by Moritz Schloegel, delves into the critical issue of evaluation practices within the rapidly expanding field of fuzzing research. Titled "SoK: Prudent Evaluation Practices for Fuzzing," the work received a distinguished paper award at IEEE S&P, highlighting its significance and timely contribution to the security community. The core objective of the research is to scrutinize the reproducibility and validity of fuzzing evaluations, addressing concerns that the sheer volume of new fuzzing techniques might be leading to a "replication crisis" similar to those observed in other scientific disciplines.
AI review
This distinguished paper meticulously dissects the alarming "replication crisis" in fuzzing research, offering a brutal but essential critique of prevalent evaluation practices. It provides updated, rigorous guidelines for the field, exposing critical flaws in methodology from target selection to statistical analysis and CVE reporting. This is foundational work that sets a new bar for scientific integrity in security research.