• by Ray Schultz , Columnist, March 3, 2025

Moreover, ToolBit determined that AI chatbots drive 95.7% less referral traffic to publishers than traditional Google search. 

Publishers should remain alert: “When sites block Perplexity, we see that they continue to send referrals which means they appear to be continuing to scrape sites under the radar,” the study argues.

It adds, “Blocking AI bots via robots.txt remains an insufficient mechanism to prevent unwanted scraping.” 

Overall, the average scraping rate per website was roughly 2 million, and the scraping rate per page was 7.199. 

What does this mean for publishers? 

“Tollbit’s data confirms what publishers have known for years – generative AI chat bots are not providing anywhere near the amount of traffic as traditional search,” says Danielle Coffey, president and CEO of the News/Media Alliance.  

Coffey continues, “By illegally scraping our content, repackaging it, and giving it to consumers without adequately directing them to our sites, AI companies are using our own content to undermine our businesses. Without web traffic, news and media organizations lose subscription and advertising revenue, and cannot continue to fund the quality work that both AI companies and consumers rely on. 

What are users scraping? Here are the scraping levels per page by content category in Q4: 

• Deals & shipping — 16.28 
• National news — 10.57 
• Consumer technology — 6.50 
• Lifestyle — 6.02 
• Special interest — 5.97 
• Local news — 5.22%
• Health & wellness — 3.28
• Entertainment & pop culture — 1.58 
• Parenting — 1.20
• B2B/professional — 0.66 

The user agent per page scraping levels for Q4:

• ChatGPT-User — 64.63 
• FacebookBot — 16.79 
• PerplexityBot — 16.75 
• Meta-externalagent — 6.86 
• Timpibot — 5.51 
• OAI-SearchBot — 5.36 
• Bytespider — 4.43 
• DuckAssistBot — 4.24
• GPTBot — 4.24
• Meta-externalfether — 4.00 
• Cohere-ai — 2.80

Finally, here is the AI bot share of total AI traffic:

• ChatGPT-User — 15.60% 
• Bytespider — 12.44% 
• Meta-ExternalAgent — 11.34% 
• OAI-SearchBot — 10.81% 
• GPTBot — 10.32%
• DuckAssistBot — 9.37%
• ClaudeBot — 8.62%
• PerplexityBot — 7.79%
• CCbot  — 4.82% 
• Timpibot — 4.39% 
• AmazonBot — 3.83%
• Omgili — 0.60% 

TollBit’s analytics platform went live in April 2024.