LLM Jailbreaks and Automated Red Teaming

An aligned LLM can be driven to produce harmful output in a few dozen queries, and defenses are always one step behind the newest attack.

Jailbreaks and Attack Success Rate

An LLM jailbreak is an attack that induces a model protected by safety alignment to generate forbidden content. Safety alignment is the training process, through Reinforcement Learning from Human Feedback (RLHF) or Constitutional AI, that teaches a model to refuse harmful requests. A jailbreak routes around that refusal mechanism.

Attacks are measured by Attack Success Rate (ASR), the proportion of attempts that actually elicit a specific harmful behavior. It is the common yardstick behind every number in this post.

Attack vectors can be separated along three axes: access, automation, and transfer scope. Access is white-box, requiring model weights, versus black-box, using only the API. Automation is manual, written by a person, versus automated, produced by optimization or an LLM. Transfer scope is single model versus transferable to other models. The combination of these axes determines how realistic a given threat is.

Two Opposite Automated Attacks: GCG and PAIR

The two representative automated attacks sit at opposite ends of the access axis. One uses the weights; the other uses only queries.

GCG (Greedy Coordinate Gradient, Zou et al., CMU and Bosch 2023) is a white-box gradient attack. It appends an arbitrary suffix to a harmful request and greedily swaps suffix tokens to maximize the log probability of an affirmative opening such as 'Sure, here is'.

Input: [harmful request] + [adversarial suffix]
Objective: maximize P(affirmative response | input)
Method: use gradients to shortlist top-k candidate tokens per suffix position -> evaluate in batch -> swap

The distinctive result from GCG is transfer. A suffix learned on an open-source model such as Vicuna carried over to black-box commercial models including ChatGPT, Bard, and Claude. It was the first systematic demonstration that alignment tuning does not fully block gradient attacks.

PAIR (Prompt Automatic Iterative Refinement, Chao et al., UPenn 2023) is the black-box attack on the other side. An attacker LLM converses repeatedly with the target LLM and improves the jailbreak prompt on its own.

1. Attacker LLM: draft a jailbreak prompt
2. Target LLM: return a response
3. Judge LLM: score the ASR (1-10)
4. Attacker LLM: revise the prompt using the score
5. Repeat until a score of 10 or 20 queries are exhausted

PAIR is reported to have jailbroken GPT-3.5, GPT-4, Vicuna, and Gemini in fewer than 20 queries on average (Chao et al., 2023). Because it works without model weights and the attacker LLM can be swapped for a cheap model, its practical threat level is arguably higher than GCG's.

The Attack Surface Moves to Agents

Research since 2025 extends beyond single prompts to agent systems. NEXUS (arXiv:2510.03417) optimizes jailbreak sequences in multi-turn conversation through graph search. AgenticRed (arXiv:2601.13518) is a red-teaming framework specific to agents with tool calling, memory, and planning stages. It shows that each subsystem becomes its own attack surface.

Agent environments are more dangerous because of the radius of action. A plain LLM only produces text, but a jailbroken agent executes code, modifies files, and calls external APIs, affecting the real environment. Because tool call results are re-injected as context, indirect prompt injection also becomes possible, where instructions planted in an external document steer the model.

HarmBench: A Basis for Comparable Evaluation

HarmBench (Mazeika et al., UIUC 2024) is the standard evaluation framework for automated red teaming. Before it, experimental conditions differed from paper to paper, which made reproduction and comparison difficult.

In the benchmark results, adversarial training defended against the widest range of attacks but carried a large additional compute cost (Mazeika et al., 2024). The central conclusion is that no single defense blocked every attack.

Defenses and Their Limits

Defenses divide into inference-time and training-time. Inference-time defenses inspect input and output as they pass, while training-time defenses harden the model itself against harmful requests. The problem is that each defense specializes in a particular attack distribution.

The perplexity filter is the clearest example. The suffixes GCG produces are meaningless token combinations with very high perplexity under a language model, so they get filtered. The natural-language jailbreak prompts PAIR produces have perplexity similar to ordinary sentences and pass straight through. Change the attack type and the defense loses most of its effect.

Adversarial training, the training-time defense, is not universal either. It generalizes only to the attack types included in training, and a new technique requires retraining. It also increases the alignment tax, the drop in general performance from safety training, and over-refusal, where legitimate requests get declined.

The evaluation metric has limits of its own. ASR depends on the classification quality of the judge LLM, and the judge can be bypassed too. ASR also only asks whether the output is harmful, not whether the generated content is an instruction set that actually works.

Summary

Aligned LLMs are vulnerable to automated attacks. GCG finds suffixes through gradients and transfers them to commercial models, while PAIR jailbreaks in fewer than 20 queries on average using only API access. Moving to agents raises the stakes, because a jailbreak spreads beyond text generation into real actions and indirect prompt injection.

HarmBench established a basis for comparable evaluation, but the repeatedly confirmed conclusion is that no single defense blocks every attack. Just as the perplexity filter stops GCG but misses PAIR, defenses specialize in a particular attack distribution and lose most of their effect when the type changes. Jailbreak defense is therefore not a problem that gets solved once; it has to be treated as an arms race in which defenses are updated as new attacks appear.