Trending:
AI & Machine Learning

AI bot traffic hits 1 in 31 web visits as RAG scraping surges 33%

Retrieval bots from ChatGPT and Gemini now drive more traffic than model training scrapers, with human visits down 5% in Q4 2025. OpenAI's ChatGPT-User scrapes five times more per page than any competitor. The shift: users bypass traditional search, letting bots pull information instead.

AI bots accounted for one in every 31 website visits by Q4 2025, up from one in 200 visits just nine months earlier, according to TollBit's latest traffic analysis. The surge is driven by retrieval augmented generation (RAG) bots, not training scrapers.

RAG traffic jumped 33% between Q2 and Q4 2025, while training scrapes dropped 15% in the same period. OpenAI's ChatGPT-User bot leads the pack, scraping five times more per page than Meta's closest competitor. AI search indexers, which feed RAG systems, saw 59% traffic growth.

The pattern is clear: users are abandoning direct web browsing. Marketing firm Eight Oh Two found 37% of active AI users now start searches in ChatGPT or Gemini rather than Google. Pew Research reports 62% of US adults use AI tools at least weekly. Human web traffic declined 5% in Q4 alone.

Imperva's February 2025 data reinforces the trend. Bots hit 51% of global web traffic in 2024, with AI-driven bad bots reaching 37%, up from 32% in 2023. That marks the first time automated traffic has surpassed human visits in a decade.

B2B sites, national news, and lifestyle content see the heaviest scraping. Tech and consumer electronics saw the steepest increase: 107% since Q2. TollBit attributes this to users turning to AI for information retrieval rather than traditional search.

The infrastructure impact is real. Publishers report bot traffic consuming up to 70% of server resources on dynamic pages, forcing migrations from shared hosting to VPS. Cloudflare and TollBit now offer bot management tools specifically for AI crawlers.

TollBit warns its numbers are conservative. Many AI scrapers now mimic human behavior convincingly enough to evade detection. The company's testing found several web scrapers "indistinguishable from human visitors."

Microsoft data shows AI referrals from Copilot and Gemini grew 155% in eight months. Traditional search isn't dead, Googlebot traffic rose 96% year-over-year, but the balance is shifting. TollBit's CEO predicts majority-bot traffic is inevitable, framing it as a management challenge rather than an existential threat.

What this means in practice: enterprise site owners need to plan for bot-majority traffic in 2026. That means capacity planning, authentication protocols, and decisions about which bots to allow. The question isn't whether bots will dominate your logs. It's which ones you'll let in.