• by Laurie Sullivan , 3 hours ago

Websites have seen an increase of more than 60% in visits from artificial intelligence (AI) harvesting bots during the past 12 months, according to data released Thursday.

Publishers are concerned about unwanted website scraping, which means the site's content is taken and repurposed without permission and adds to all types of costs. 

Nina Gould, chief innovation officer at Forbes, told MediaPost last month that many bots will take the content, adding that it’s like "playing whack-a-mole" to get them to stop.

“AI bot traffic is undermining the viability of the open web publishers used to reach audiences with their advertising," James Rosewell, CEO of 51Degrees, told MediaPost. "AI bots impose costs on publishers twice. Once in the bandwidth their bots use when harvesting, and then again when they steal traffic from sites with AI results."

Based on analysis from 51Degrees of more than 3 billion website visits from the 12 months leading up to May 2026, the percentage of AI harvesters detected on website traffic increased by 62.5% -- up from 8% to 13% of all web sessions, the U.K.-based technology company reported. 

51Degrees, which provides real-time digital data services, identified the most prolific bot as AmazonBot, Amazon’s web crawler. 

AmazonBot, which is used to index content and gather data to improve Alexa, Rufus, and product search results, represented 4.1% of all web visits.

Meta-ExternalAgent represented 2.9%, while Microsoft BingBot represented 1.35%, followed by Microsoft BingBot at 1.35% Anthropic at 0.19%, OpenAI at 0.09%, Perplexity at 0.04% and Google Gemini at 0.0003% of web sessions.

Google also uses its search bot for AI harvesting -- a number that is not included in this data, the company said.

The assumption is that lower numbers represent more mature technology, and that they have already taken most of the information on the web they need to date, according to 51Degrees.

The statistics represent web sessions from bots that identify themselves as AI bots including ClaudeBot, PerplexityBot and others.

Although some bots masquerade as human users, 51Degrees could not capture those. None are valid as advertising traffic, but 51Degrees does not claim that site owners are passing these on as ad impressions -- only that these are the volumes of visits they are receiving.

Publishers face a number of significant challenges. While some bots reach sites and deliver more traffic from search and other means, publishers often block crawling tags that bots use. Malicious bots work undercover and falsify data that makes them look like a human user. 

Technology is being created to identify specific bots as Google, Cloudflare, and other technology companies rewrite the way the internet handles automated traffic.

Cloudflare data released earlier this year indicates that agentic AI bots generate roughly 57.4% of global web requests.

To address this surge, Google and Cloudflare developed technologies including Private Access Control Tokens to differentiate human users from malicious scripts and authorized AI agents.