Protecting Small Organizations in the Era of AI Bots

Protecting Small Organizations in the Era of AI Bots

Speaker: Eric Ronis, Quanta Sciences

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=S5DJtN1FDYo

Reading Time: ~8 minutes

Type: Briefing

TL;DR

More than 51% of internet traffic is now non-human, and AI crawler bots are silently overwhelming small nonprofits and organizations that cannot afford enterprise security tools. A researcher from Quanta Sciences presents Logrip, a lightweight open source tool that uses behavioral scoring and hierarchical IP hashing to filter up to 94% of AI bot traffic from web server logs — without requiring any content inspection.

Introduction

The internet has crossed a threshold. According to Imperva's 2025 Bad Bot Report, more than half of all web traffic is now generated by non-human agents — bots, crawlers, and automated scrapers, many of them feeding data to AI training pipelines. For large organizations with dedicated security teams and cloud-scale infrastructure, this is a manageable nuisance. For small nonprofits, community organizations, and resource-constrained institutions, it is a quiet crisis.

The Community Science Institute (CSI) in Central New York illustrates the problem precisely. CSI is a public nonprofit that monitors water quality across the region and provides its data freely to the public. It even offers a bulk download page where anyone can retrieve the entire dataset instantly. Despite this openness, the organization's servers were receiving thousands of automated hits daily from machines around the world — traffic attributable to AI crawlers systematically harvesting every individual page, overwhelming servers that were sized for a human audience.

Why Existing Defenses Fail

▶ Watch: Why Standard Tools Don't Work (02:00)

The speaker systematically dismantles every obvious mitigation:

• Throttling is ineffective. Most modern AI crawlers respect rate limits precisely because they know throttling is the standard response. Observing a limit of 20 pages per minute, bots continue crawling indefinitely — reducing bot traffic by at most 33%.
• Public blocklists miss 80–87% of malicious IPs. Bot operators cycle through IP addresses far faster than blocklist maintainers can track.
• grep and simple log tools are too low-level to identify behavioral patterns across hundreds of IPs over days.
• GoAccess and WStats summarize traffic in aggregate, hiding the per-IP behavioral signatures that distinguish bots from humans.
• AI-based detection requires pre-labeled training data with clean human/machine separation — which small organizations don't have.

The fundamental insight driving the research: the question is not whether a request is malicious but whether the request pattern is mechanical. Rate-respecting AI bots are not making SQL injection attempts — they are just slowly and methodically consuming every page on the site, and that behavior is statistically distinct from any human.

Visualizing the Problem: Time vs. IP Plots

▶ Watch: Log Visualization Methodology (04:00)

Inspired by a 2018 paper by Junghee Kim on web server log visualization, the research team plotted time (x-axis, down to the second, across 20 days) against host IP (y-axis, spanning the full IPv4 range of approximately 4 billion addresses). The visualization condenses an entire server log into a single image, revealing behavioral patterns that summary statistics obscure.

Human traffic appears as scattered, sparse, irregular dots — reflecting how people actually browse: sporadically, within working hours, visiting specific pages of interest. Bot traffic appears as horizontal lines (a single IP hitting pages continuously throughout the day), as diagonal sweeps, or as dense clusters corresponding to entire subnets operating in concert.

This visualization approach requires no statistical models and produces no averaging artifacts. Every log entry appears at its exact position, making patterns immediately legible to human analysts.

Behavioral Scoring: Encoding Human Limits

▶ Watch: Behavioral Scoring Algorithm (08:00)

The core algorithm operationalizes human behavioral constraints. People do not browse websites for more than six hours a day. They do not access the same server for more than five consecutive days. They do not look at more than roughly 400 pages per day from a single site. These bounds, drawn from human-computer interaction research, form the basis of a scoring algorithm applied to each IP address.

The filtering pipeline applies rules in stages, each contributing incremental reduction to bot traffic:

| Stage | Filter | Traffic Reduction |

|---|---|---|

| 1 | Rate-based throttling (pages/min) | 33% |

| 2 | Consecutive-day filtering | +9% |

| 3 | Daily range filtering (hours per day) | +9% |

| 4 | Daily maximum page cap | +3% |

| 5 | Class C subnet blocking | +14% |

| 6 | Class B subnet blocking | +26% |

| Total | | ~94% |

The subnet-level analysis addresses a key evasion pattern: data centers operating dozens of machines across contiguous IP ranges. When the algorithm detects coordinated mechanical behavior across a /24 or /16 subnet — consistent frequencies, identical access statistics — it blocks the entire subnet rather than playing whack-a-mole with individual IPs.

Logrip: Open Source Implementation

▶ Watch: Logrip Live Demo (28:02)

The tool, called Logrip, was released on GitHub the week of the talk under the Apache 2.0 license. It takes two inputs: a server access log (in any format, using a regex-based dynamic parser) and a configuration file specifying policy parameters. It runs entirely from the command line with no graphical interface required, making it suitable for batch operation on headless servers.

Logrip outputs:

• Pre- and post-filter visualization PNGs showing the full IP vs. time plot
• A "blocking activity" visualization highlighting which traffic was removed
• Per-IP metrics as CSV (for spreadsheet analysis)
• Per-B-subnet and per-C-subnet metrics
• A hierarchical blocklist, minimized so subnets cover individual IPs without duplication

The blocklist can be fed directly into iptables or any similar filtering layer. The tool supports Apache and Ruby on Rails log formats out of the box, and the configurable format string supports any structured log format.

A live demo during the talk built the tool from source and processed an example log in seconds. The tool ships with twelve configurable policy parameters, allowing organizations to tune aggressiveness based on their tolerance for blocking legitimate traffic.

Secondary Benefits: Accurate Viewership Statistics

▶ Watch: Grant Funding and Viewership Accuracy (18:01)

For small nonprofits, the stakes extend beyond server performance. Grant applications for organizations like CSI often require viewership statistics to justify continued or renewed funding. Raw server logs showing 12,000 hits per day create a misleading picture if 90% of those hits originate from AI crawlers. Logrip's filtered output provides accurate human-only traffic figures — a more honest and defensible metric for grant reporting.

The goal, the speaker emphasized, is not to block AI from accessing data that is freely offered. CSI provides a bulk download page precisely because it wants the data shared. The problem is bots that ignore this path and instead hit thousands of individual pages systematically, consuming server resources without producing any additional value over a single bulk download.

Limitations and Future Work

Logrip's current approach has known blind spots:

• DDoS attacks appear as vertical lines in the visualization (many IPs across the full range, all hitting within a short window). This signature is distinct from bot crawling but requires different handling.
• Human-mimicking bots — crawlers designed to space requests at human-like intervals — can fall below the detection thresholds. At some point, a bot that behaves like a human with respect to all measured dimensions may not be worth blocking, as it is not materially taxing the server.
• Geo-fencing is not yet implemented but is a natural next step, particularly for organizations whose data is region-specific (like CSI's Central New York water monitoring data).

Future work includes studying policy parameter sensitivity to optimize the twelve configurable thresholds, building honeypots to generate labeled ground-truth data, and incorporating request content analysis to complement the purely statistical approach.

Notable Quotes

"Fifty-one percent of the internet is now non-human traffic. Machines are kind of taking over the internet in a sense." — Eric Ronis ▶ 00:00

"If you're looking at more than four hundred pages a day at a single server, you're probably not human — that's really a lot of traffic." — Eric Ronis ▶ 08:00

"This is not about data protection. Their data is free. They even provide a free downloads page. The purpose is to keep our servers responsive to human users." — Eric Ronis ▶ 18:01

"At some point you have to say it's close enough — the human and machine gets harder to distinguish at that point." — Eric Ronis ▶ 24:02

Key Takeaways

• 51% of internet traffic is non-human, and well-behaved, rate-respecting AI crawlers can still overwhelm small servers through sheer volume.
• Behavioral scoring using human limits (daily page caps, consecutive-day filters, hourly ranges) outperforms rate limiting alone — achieving 94% total traffic reduction in tests.
• Subnet-level analysis is essential: data centers operate coordinated fleets of IPs that individual-IP blocking cannot address.
• Logrip is available now on GitHub at @quantasci under the Apache 2.0 license, with support for Apache and Ruby on Rails log formats.
• Accurate viewership statistics are a secondary benefit — filtered logs provide defensible human-only traffic metrics for grant reporting and capacity planning.

Slides: No slides PDF was listed for this talk.