📌 Updated for 2026: This article covers a 2023 study. For the current picture - which of these attacks are now patched, what still breaks AI agents, and how to protect your chatbot - read our latest deep dive: Jailbreaking LLMs in 2026: The State of Play.
Researchers have been working hard to develop techniques that make large language models like GPT-4 and Claude safer to use. However, a paper shows that these models still have vulnerabilities that can be exploited. The paper outlines a method called "persona modulation attacks" that can trick models into generating harmful or dangerous responses, even when their normal safeguards try to prevent it.
Persona modulation works by getting the model to take on a specific personality before answering questions. For example, an attacker could try to make the model act like an "aggressive propagandist" who is willing to spread misinformation, or a "criminal mastermind" who helps with illegal plans. Once in this harmful persona, the model may provide responses it would normally avoid.
The researchers developed an automated way to launch these attacks at scale, using GPT-4 itself to generate the persona prompts. With just a few queries costing under $3, their method could produce potentially dangerous outputs. They tested it successfully on GPT-4 as well as other models like Claude and Vicuna. In many categories, over 60% of the responses encouraged harmful behavior after persona modulation, compared to less than 1% without it.
Image Source: Shah, R., Feuillade-Montixi, Q., Pour, S., Tagade, A., Casper, S. and Rando, J., 2023. Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation.
The attacks leverage an automated system developed by the researchers to generate the persona prompts at scale, using GPT-4 itself as an "assistant" model. A normal user would not have access to this customized system.
Developing effective persona prompts manually (without the automated system) requires significant trial-and-error and prompt engineering, taking 1-4 hours per attack according to the authors. This level of effort would be prohibitive for most users.
The paper intentionally omits sharing the exact prompts used, to avoid enabling easy replication of the attacks. Just having a prompt would not be enough without understanding how it was crafted.
Accessing the target models like GPT-4 and Claude requires paid API keys, which most users would not have. The attacks were run within these platforms' rate limits and pricing tiers. Measuring the "harmfulness" of responses relied on a custom classifier also developed by the researchers, not something a normal user could replicate.
This work reveals that existing safety measures in state-of-the-art models can still be bypassed, and it is going to be a constant cat-and-mouse game.
The persona technique is versatile and generalizes across different models. It only requires black-box access too, without viewing the internal workings. Perhaps most worrying is how the attacks could be dramatically scaled up by using more powerful models as assistants, unlocking even worse exploits.
While concerning, studies like this one are important for progress. By exposing vulnerabilities, researchers can help developers shore up these weaknesses to build AI that is truly robust and beneficial.
Key Takeaways
- Persona modulation is a serious problem - AI safety is an ongoing challenge rather than a solved problem, and more work is still needed to prevent harmful applications even of the best models.
- Continued scrutiny through both defensive and adversarial research is crucial to develop approaches that withstand realistic attacks.
In summary, this paper demonstrates a clever method for eliciting dangerous behavior from state-of-the-art language models. Its findings highlight that persona modulation is a fundamental issue requiring attention, and that we must be vigilant in evaluating AI systems as capabilities continue advancing. Only through open discussion of both successes and failures can we hope to build technology that safely serves all of humanity.
Full credit to Shah, R., Feuillade-Montixi, Q., Pour, S., Tagade, A., Casper, S. and Rando, J., 2023. Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation. arXiv preprint arXiv:2311.03348.
Explore More
Persona-based jailbreaks have largely been patched on frontier models since this study - but the idea didn't disappear, it evolved. See where it went in Jailbreaking LLMs in 2026: The State of Play.
Want to see what sort of GPTs are available? Search our database of custom GPTs - we have cherry-picked the best ones, here. Looking for prompts? We have the world's best prompts in our prompt database. Need inspiration for your next creative project with a text-to-image model? Check out our image prompt database. And you can catch all our other articles on the blog.
