• by Ray Schultz , Columnist, March 4, 2024

Marketers who were around back in 1988 may recall Morris, the first computer worm. 

Well, meet Morris II, “the first worm designed to target GenAI ecosystems through the use of adversarial self-replicating prompts,” according to ComPromptMized: Unleashing zero-click worms that target GenAI-powered applications, their paper explaining this malign tool. 

The authors show how attackers “can launch cyber-attacks against GenAI ecosystems by creating dedicated adversarial inputs, that we name adversarial self-replicating prompts, using jailbreaking and adversarial machine learning techniques.”

What’s the harm? It can launch spam campaigns and steal personal data, the Independent writes. 

This is dangerous information if it gets out onto the street. And in a sense, it already has with the publication of this paper. 

The authors applied Morris II against GenAI-powered email assistants in two use cases (spamming and exfiltrating personal data), under two settings (black-box and white-box accesses), using two types of input data (text and images)."

The paper reveals “two new classes of attacks against GenAI-powered applications: the first class of attacks steers the flow of a GenAI- powered application toward a desired target, and the second class poisons the database of the RAG of GenAI-powered applications in inference time.”

Both of these attacks are "applied in zero-click and exploit the automatic inference conducted by GenAI models on input data that is triggered by the GenAI-powered application," it continues. 

What can you do to prevent it? For one thing, "GenAI models could be secured by rephrasing the entire output in order to ensure that the output does not consist of pieces that are similar to the input and do not yield the same inference," the paper says.

Then there are countermeasures against propagation. These techniques can be “used to detect worms by analyzing the interactions of agents with (1) other agents in the GenAI-powered ecosystem (i.e., by monitoring the interactions in the GenAI ecosystem), and (2) 3rd party services, such as SMTP servers, and messaging application services (by monitoring the interactions of the agents in the GenAI ecosystem).”

All well and good. But the authors end on a grim note: “While we hope this paper’s findings will prevent the appearance of GenAI worms in the wild, we believe that GenAI worms will appear in the next few years in real products and will trigger significant and undesired outcomes,” the paper concludes.

The paper was authored by Stav Cohen or Cornell Tech and Technion, of the Institute of Technology, Haifa, Israel; Ron Bitton of Intuit, of Petach-Tikva, Israel and Ben Nassi of Cornell Tech.