• by Joe Mandese , Yesterday

The only thing worse than AIs hallucinating false facts about human behaviors is humans hallucinating false facts about AI behaviors. Today's post is about a prime example of that: a new global research study from Snowflake purporting to measure the ROI of companies overall -- and in specific industrial sectors -- from adopting AI.

The findings are based on self-reported responses from executive at 2,050 businesses and technology leaders across 10 countries, all of whom influence their organization's current and future AI purchases.

I got personally pitched on the results for the "ad/media" category, which the company described as combining the responses of advertisers, agencies and media suppliers, because it generated the best AI ROI of any industry: $1.69 vs. $1.49 for the average of all industries measured, per $1 invested.

That the advertising/media industry had the best return didn't surprise me. What surprised me was how the Snowflake study benchmarked it: based on the responses of a self-reported survey.

I asked a spokesperson if they could share the data on the actual responses to the question -- or even how the question was worded -- but the Snowflake team declined, saying only that the respondents who indicated they had formally measured the dollar ROI were asked to provide a specific figure (see the ones reported above).

When I asked how those ROIs compared with other capital investments made by all industries and the advertising/media industry, the Snowflake team said they didn't actually measure that.

So I asked some LLMs if they knew and you can see the results in the chart above, but the bottom line is that the Snowflake study doesn't make a very compelling case for the return on AI investment. Just ask some AIs.