SoK: Towards a Unified Approach to Applied Replicability for Computer Security

Daniel Olszewski, Tyler Tucker, Kevin R. B. Butler, Patrick Traynor

34th USENIX Security Symposium · Day 1

In an era where scientific rigor and empirical validation are paramount, the computer security community, like many other scientific disciplines, faces increasing calls for improved research validity. This paper, "SoK: Towards a Unified Approach to Applied Replicability for Computer Security," by Daniel Olszewski, Tyler Tucker, Kevin R. B. Butler, and Patrick Traynor from the University of Florida, addresses a critical gap in this discourse. It systematically reviews over three decades of research on reproducibility, replicability, and validity, highlighting the inconsistencies and practical limitations of existing definitions within the context of computer security research. The authors argue that while reproducibility—achieving the same results with the same code and data—is important, it is often insufficient and sometimes unattainable, especially given the unique challenges of security studies.

AI review

Finally, someone wrote down what we all complain about at the bar after AEC meetings. The Tree of Validity framework is genuinely useful—not because it's revolutionary, but because it forces precision where the community has been sloppy for decades. Minor quibble: the case studies do more work than the formalism.