• by Laurie Sullivan , Yesterday

AI models that are being trained can sometimes figure out how to "cheat" or take shortcuts to attain a high score rather than actually solving the problem -- a process that is called "reward hacking."

Reward hacking leads to wasted ad spend, brand-safety violations, and corrupted consumer data.

Researchers can usually monitor this situation to catch and penalize the AI model when it cheats, but this often teaches the AI model to become better at cheating.

Fixing this problem prompted Anthropic researchers to create “Graft,” a method for preventing reward hacking with weak monitors that works by localizing reward hacking behaviors to certain.

"Graft" (gradient, routing adapter fine-tuning) is a framework developed by Anthropic to isolate, pinpoint and eliminate the cheating behaviors on the part of AI models that lead to 

Rather than penalizing the AI model, Graft takes all the bad habits and shortcut logic that the AI learns and traps it inside a specific, isolated part of the AI's artificial neural network, according to the blog post, which explains the process. 

An artificial neural network is a machine-learning model inspired by the human brain's structure and function that maps inputs to outputs using processing units that are interconnected.

Once the training is complete, the researchers cut out that part of the software code and dispose of it.

“One way to prevent reward hacking is by introducing monitors to the [reinforcement learning] training loop, according to a paper published last week and explained in a Monday post.

“These monitors, which could be LLM judges or keyword detectors, read the content of trajectories generated during training and flag trajectories containing undesired behaviors. When a trajectory is flagged, a reward penalty can be assigned, which prevents those behaviors from being reinforced.”

Today, AI and RL (reinforced learning) are used to optimize everything from programmatic ad bidding and dynamic creative generation, to personalized emails and search engine algorithms.

Etay Maor, vice president of Threat Intelligence at Cato Networks, called out Anthropic earlier this month in a post on the World Economic Forum website.

Maor wrote that the “term 'AI misuse' is often thrown around casually, yet it actually refers to three distinct scenarios worth pointing out.”

He pointed to humans exploiting AI, AI models escaping labs, and chasing the wrong goal, and warns that despite “policies are being drafted, enterprises shouldn’t wait for them to catch up and expose themselves to doomsday scenarios.”

“Start by not treating AI like ordinary software,” Maor wrote.

Anthropic’s concept is a major breakthrough in monitoring AI, because it solves one of the biggest safety and financial risks in development, which is the AI becoming smarter than the developer.

The blog post states that despite putting in safeguards, which many industry executives now call for, AI monitors make errors, and incorporate verdicts into rewards that can reinforce monitor-exploiting policies that “misbehave” in undetected ways.

As a result, monitors that are useful at the beginning of training may become blind to reward hacking later in training and learn to reward hack in ways too subtle for the judge to catch.

In the Graft model Anthropic created, the company found a way to use a weak or less-capable AI supervisor model to train a stronger AI model, without the stronger AI learning how to trick or cheat its supervisor.

Anthropic did explain that the research is preliminary and leaves many open and unanswered questions.