• by Ray Schultz , September 19, 2023

Almost one-third of the world's top 50 news sites have blocked AI crawlers from accessing their content, AltIndex reports, citing data from a Kirwan Digital Marketing Agency survey.

Included are CNN, The New York Times, The Daily Mail, Reuters, and Bloomberg, all of which have blocked at least one AI crawler. 

But the totals vary by country. In the U.S., 24% of the 50 largest news sites have blocked one crawler. In contrast, only three of 21 leading sites did the same in the UK.  

OpenAI’s GPTBot has been blocked the most.

The most targeted AI technology is GPTBot, which has been blocked by CNN, Edition, The New York Times, The Washington Post, The Times of India, News Now (UK), Indian Express, Reuters, Business Insider and several other publishing entities.  

But most of those same publications allowed ChatGPT-User; The Washington Post being the one exception that blocked it. 

As the report states, publishers see no benefit to providing crawlers with their data.

According to AltIndex, things escalated last month when OpenAI launched its GPTBot crawler.

OpenAI found GPTBot being blocked despite its promise that paywalled content would be excluded from websites.

GPTBot has been blocked 22% of the time across the top 50 news sites.