• by Laurie Sullivan , Staff Writer, July 6, 2026

It may seem surreal because of the internet's age, but for every hour spent online searching for information, only 15 minutes is spent on the open web.

Traditional search behavior continues to collapse as users shift to artificial intelligence (AI)-driven discovery and consumption.

The AI prompt provides a near instantaneous, consolidated answer.

Cloudflare, which provides a range of internet services, has shared key data insights to illustrate how quickly the Internet business model continues to change, and what this new content market means for publishers and website owners.

The data, which shows a major imbalance between AI companies and website publishing, can be seen through Cloudflare's dashboard, Attribution Business Insights.

The dashboard estimates how often artificial intelligence (AI) systems visit a web page compared with how many readers they send back to a site.

The number can approach 50,000 crawls for one referral, according to the company's data.

Cloudflare can cite content ranging from 118 crawls per referral to nearly 50,000, mainly because a crawler can grab the same publisher's pages tens of thousands of times before one reader arrives from the product those pages helped to build.

AI bots have crawl-to-referral ratios of 118:1 to nearly 50,000:1.

The data also describes a breakdown in the traditional "open web" business model caused by AI data scraping. When a search engine like Google crawls a website, it downloads a page to index it. In exchange, it sends human readers to that website via search results. Traditional search bots may crawl five pages for every one user it sends back.

AI bots do not exist to send users to websites, but rather to download and absorb data to train their models and send information back to consumers, especially in the case of a shopping agent. 

Cloudflare in a blog post called AI "a platform shift happening at more than 2x the speed that smartphones were adopted. In just 3.5 years, over 30% of humanity — 2.5 billion active users — has adopted regular use of generative AI. The adoption curve isn't merely steep: it's going vertical."

There are billions of agent requests across the Cloudflare network each day. And there has been an increase in daily agent requests — 1,700% — in one year, from June 2025 to May 2026.

Internet crawlers have their own new purpose, with 52% of requests now for AI training as of June 2026, up from 22% in spring 2025.

Pure search crawling now represents a small and declining share of overall activity, although it remains critical for publisher visibility.

Some of the most heavily crawled categories including IT and services as well as computer software now see human traffic decline as much as 40% in less than one year.

Google remains the dominant gateway to online discovery, accounting for approximately 88% of referral traffic, according to Cloudflare data. Increasingly, however, Google is helping users consume content directly within Google-owned AI experiences.

Google sees 87.8% of the searches and sends nearly all the referrals, followed by Bing with 3.4%, TikTok with 3.2%, Yandex with 2.1%, Baidu with 1.6% and DuckDuckGo with 1.4%.

Cloudflare stated that most AI companies separate discovery crawlers from training crawlers -- making it easier for publishers to choose which type of content access to allow -- but Google does not.

Today, Google has access to about twice as more information compared to leading AI companies because it leverages a mixed-use bot that makes it difficult for customers to participate in Google's search ecosystem without also participating in Google's AI ecosystem, according to the report.

How does Cloudflare have so much insight into this topic? More than 20% of the web sits behind its network. Of the world's most visited websites, 36% rely on the Cloudflare network, and more than 40% of the Fortune 500 are Cloudflare customers.

Nearly 80% of AI companies use Cloudflare, alongside thousands of developers and emerging AI companies.