אירועים והרצאות בפקולטה למדעי המחשב ע"ש הנרי ומרילין טאוב

In the past year, numerous companies have incorporated Generative AI (GenAI) capabilities into new and existing applications, forming interconnected Generative AI (GenAI) ecosystems consisting of applications powered by GenAI services.

While ongoing research highlighted risks associated with the GenAI layer of agents (e.g., dialog poisoning, membership inference, prompt leaking, jailbreaking), a critical question emerges: Can attackers develop malware to exploit the GenAI component of an application and launch cyber-attacks on the GenAI-powered application or the entire GenAI ecosystem?

In this talk, we discuss the new evolving attack landscape of input prompts being used to conduct malicious activities in the context of an application.

We show how attackers can craft adversarial self-replicating prompts, which when sent to GenAI-powered applications can form as (1) malware that launches a DoS attack against a GenAI-powered application by causing it to enter an infinite loop, (2) a worm that extracts a user's sensitive information from GenAI-powered email assistants and compromises new GenAI-powered applications, and (3) APT (advanced persistent threat) that uses the advanced AI capabilities of the GenAI model to identify the assets in the context, determine the possible malicious activities to conduct, execute one of them, and cover its tracks.

The findings in our study are demonstrated against three different applications powered by three different GenAI models (Gemini Pro, ChatGPT 4.0, and LLaVA). At the end of the talk, we discuss the upcoming changes in application security that are about to appear in the next few years due to the increased integration of GenAI capabilities into existing and new applications.