SoK: State of the Krawlers – Evaluating the Effectiveness of Crawling Algorithms for Web Security Measurements
Aleksei Stafeev, Giancarlo Pellegrino
33rd USENIX Security Symposium · Day 1 · USENIX Security '24
This talk, titled "SoK: State of the Krawlers – Evaluating the Effectiveness of Crawling Algorithms for Web Security Measurements," delves into the critical role of web crawlers in empirical security research and the often-overlooked inconsistencies in their application and evaluation. Presented by Andrea on behalf of authors Aleksei Stafeev and Giancarlo Pellegrino, the research addresses the fundamental challenge of effectively traversing the complex landscape of the modern web to gather data for large-scale security measurements. The presentation highlights that despite their ubiquity, the actual effectiveness of different crawling algorithms in varied web security contexts remains largely unclear.
AI review
This SoK cuts through the usual academic hand-waving, providing a rigorous, data-driven assessment of web crawling algorithms for security measurements. Their comprehensive empirical evaluation, backed by the Arkarium benchmark, exposes the deficiencies of common practices and offers critical, actionable guidance for anyone building or using web security tools at scale. Essential work for anyone serious about empirical web security research.