ChatGPT is a failure as a search engine, at least when it comes to accurately attributing news stories to sources.  

Publishers “face the risk of their content being misattributed or misrepresented regardless of whether they allow OpenAI’s crawlers,” according to a Tow Center analysis, as reported by the Columbia Journalism Review. 

The Tow Center selected twenty publishers at random, including some who have deals with OpenAI, others that are suing OpenAI and unaffiliated publishers that have either allowed or blocked ChatGPT’s search crawler. 

The researchers pulled 200 quotes from the 20 publications and asked ChatGPT to identify the sources of each.  

Of that 200, ChatGPT provided “partially or entirely incorrect responses on a hundred and fifty-three occasions, though it only acknowledged an inability to accurately respond to a query seven times. Only in those seven outputs did the chatbot use qualifying words and phrases like ‘appears,’ ‘it’s possible,’ or ‘might,’ or statements like ‘I couldn’t locate the exact article.’”

“What we found was not promising for news publishers,” the Tow Center states. “Though OpenAI emphasizes its ability to provide users ‘timely answers with links to relevant web sources,’ the company makes no explicit commitment to ensuring the accuracy of those citations. This is a notable omission for publishers who expect their content to be referenced and represented faithfully.”

The Tow Center adds that pasting an exact quote into a traditional search engine like Google or Bing returns either “a visual indication that the search engine has located the source—bolded text that matches your search—or a message that informs you there are no results. However, ChatGPT rarely declined to answer our queries and instead resorted to making false assertions when it could not identify the correct source.”  

“While this issue is likely not unique to queries about publisher content, it does have implications for things publishers care about, such as trustworthiness, brand safety, and recognition for their work,” the study states.