• by Ray Schultz , Columnist, Yesterday

Marketers who are beside themselves over the advance of AI ought to sober up. At this point, AI can barely handle frontline tasks, judging by a study from Layer3Labs.  

For everyday chores like billing disputes, support tickets and resume screening, leading models produced finished work only 34% of the time. For the remaining two thirds, humans had to clean things up. 

This can lead to what Layer3Labs calls client-facing blunders. Particularly vulnerable are high-volume workflows like outbound email, customer support and social media posts. These are publicly visible, and damage to the brand is instantaneous when the model provides an incorrect price or sends robotic-sounding copy.  

Don’t be reassured by the fact that an AI model has passed the bar exam.  

"Passing a standardized exam doesn't reconcile a vendor statement, resolve an angry customer refund, or catch hallucinations in a commercial lease," says Jonathan Teplitsky, founder of Layer3Labs. "The frontier labs are trapped in an arms race over academic tests the average company never uses.” 

Teplitsky adds, “Owners don't care if a model can write a sonnet—they care whether it takes work off their team's plate without causing more headaches." 

As it turns out, they can’t always do that, either.  

Layer3Labs’ Real Work Index evaluated 18 top AI models in marketing, legal, recruiting and operations, running each of 64 tasks five times for a total of 5,750 tests. 

It found that the risks vary with the assignment. For instance, 14% of model runs produced polished and authoritative drafts for projects like commercial lease abstracts, tax filings and compliance analyses. But these texts concealed serious errors like fabricated legal clauses, altered numbers or invented regulations. 

And the #1 AI model was outperformed by cheaper alternatives on over 40% of standard business jobs, Layer3Labs continues. 

Then there is the economic factor when using flagship frontier models. They cost 7x more per finished task  ($0.21 vs. $0.03) and are scarcely worth it in terms of measurable gain. Let’s say you’re doing 10,000 tasks per month -- that will cost $2,100, versus $300 for the cheaper alternative.

We can’t judge if this study is entirely accurate. Certainly, it reinforces our prejudices, but one year from now, the results might be different. 

For now, it seems AI is a long way from being able to destroy humanity.