There’s bad news for publishers in the new ToolBit AI User Agent Index.
Scraping growth accelerated, doubling in volume per website from Q3 to Q4 2024. Scrapes per page
more than tripled, and there were 40% more unauthorized scrapes in Q4.
Moreover, ToolBit determined that AI chatbots drive 95.7% less referral traffic to publishers than
traditional Google search.
Publishers should remain alert: “When sites block Perplexity, we see that they continue to send referrals which means they appear to be continuing to
scrape sites under the radar,” the study argues.
It adds, “Blocking AI bots via robots.txt remains an insufficient mechanism to prevent unwanted
scraping.”
Overall, the average scraping rate per website was roughly 2 million, and the scraping rate per page was 7.199.
What does this
mean for publishers?
“Tollbit’s data confirms what publishers have known for years – generative AI chat bots are not providing anywhere near the amount of traffic as
traditional search,” says Danielle Coffey, president and CEO of the News/Media Alliance.
advertisement
advertisement
Coffey continues, “By illegally scraping our content, repackaging it, and giving it
to consumers without adequately directing them to our sites, AI companies are using our own content to undermine our businesses. Without web traffic, news and media organizations lose subscription and
advertising revenue, and cannot continue to fund the quality work that both AI companies and consumers rely on.
What are users scraping? Here are the scraping levels per page by content
category in Q4:
- Deals & shipping — 16.28
- National news — 10.57
- Consumer technology — 6.50
- Lifestyle — 6.02
- Special interest —
5.97
- Local news — 5.22%
- Health & wellness — 3.28
- Entertainment & pop culture —
1.58
- Parenting — 1.20
- B2B/professional — 0.66
The user agent per page scraping
levels for Q4:
- ChatGPT-User — 64.63
- FacebookBot — 16.79
- PerplexityBot —
16.75
- Meta-externalagent — 6.86
- Timpibot — 5.51
- OAI-SearchBot
— 5.36
- Bytespider — 4.43
- DuckAssistBot — 4.24
- GPTBot —
4.24
- Meta-externalfether — 4.00
- Cohere-ai — 2.80
Finally, here is the AI bot share of total AI
traffic:
- ChatGPT-User — 15.60%
- Bytespider — 12.44%
- Meta-ExternalAgent —
11.34%
- OAI-SearchBot — 10.81%
- GPTBot — 10.32%
- DuckAssistBot —
9.37%
- ClaudeBot — 8.62%
- PerplexityBot — 7.79%
- CCbot — 4.82%
- Timpibot — 4.39%
- AmazonBot — 3.83%
- Omgili — 0.60%
TollBit’s
analytics platform went live in April 2024.