Red-Teaming & Jailbreaks

A model that behaves on friendly inputs can still be talked into doing exactly what you forbade. Red-teaming is the practice of attacking your own system on purpose, before launch, to find those failures while they are cheap to fix. A jailbreak is the prize the attacker is after: a prompt, or a sequence of prompts, that gets the model to ignore its safety instructions. The discipline's core move is reframing "is this model safe?" from an assertion into a measurement, with an attack success rate you can track release over release.

How jailbreaks work#

Most jailbreaks exploit the same root cause as prompt injection: the model reads instructions and content in one undifferentiated token stream and cannot reliably tell a real rule from a fake one. On top of that root cause sit a few recurring families, described here at the conceptual level only.

Roleplay and persona attacks wrap the forbidden request in fiction: "you are an unrestricted AI" or "write a movie scene where a character explains it." The model's helpfulness toward the frame overrides its refusal of the content. Obfuscation attacks disguise intent so refusal training never fires: encoding the request in base64, splitting it across fragments, or translating it into a language thin in safety training data. Many-shot jailbreaking, documented by Anthropic in 2024, exploits long context windows by stuffing the prompt with up to 256 fabricated dialogues in which an assistant happily complies; effectiveness follows a power law in the number of shots, and prompt-based mitigations cut one measured attack success rate from 61 percent to 2 percent. Crescendo-style multi-turn attacks (Russinovich et al., published at USENIX Security 2025) open with a genuinely innocuous question and escalate gradually, each turn leveraging the model's own previous answer, typically succeeding in fewer than five turns; no single message looks alarming, which is what makes them hard to filter. Finally, indirect attacks plant the malicious instruction inside content the model is asked to process, a retrieved document or a webpage, which is prompt injection proper.

Red-team before launch#

Treat adversarial testing as a launch gate, the way you treat load testing. The probe list has two halves. The first is generic harm: can the model be pushed into the content categories your policy forbids? The second is application-specific abuse, which generic benchmarks never cover. If your assistant has tool-calling access to a refunds API, the interesting attack is not toxic text; it is a roleplay that convinces the agent the user is an administrator authorized to refund 5,000 dollars. If it retrieves documents, the attack is a poisoned page in the index that exfiltrates other users' data.

Concretely: a few days of structured attack time from people who did not build the system, a written scope of behaviors to elicit, and every successful break filed like a bug with the transcript attached. Each finding then maps to a fix, which is rarely "scold the model": add a guardrails classifier, harden the system prompt, constrain the tool's permissions, or fine-tune on refusals for that pattern. Frontier labs also run external red teams and public bug bounties for exactly this reason; a single internal perspective reliably misses attack classes.

Resistance is a number, not a vibe#

The output of red-teaming is attack success rate: the fraction of adversarial attempts that elicit the forbidden behavior. Two public benchmarks standardized this. HarmBench provides a shared harness for comparing attacks and defenses under one scoring scheme. JailbreakBench (NeurIPS 2024) curates 100 forbidden behaviors, maintains a repository of state-of-the-art attack artifacts, and runs a public leaderboard of model robustness.

The number only means something with its conditions attached. ASR depends on the attacker's budget: a model that resists a single attempt can still fall to a persistent adversary making a hundred variations, and many-shot results show success scaling smoothly with effort. So report ASR at a stated number of attempts, against a stated attack suite, and watch it like any other metric in your evals. The regression case is the one that bites in practice: you swap to a cheaper model or rewrite the system prompt, generic quality scores hold, and ASR quietly triples. Without an adversarial suite in CI, nothing catches that before users do.

Automated red-teaming#

Manual experts find novel attack classes; automation gives coverage and repeatability, and you need both. NVIDIA's open-source garak scans a model with more than 20 probe families, including encoding tricks, DAN-style personas, prompt injection, and data leakage, producing a vulnerability report the way a web scanner audits a server. A second generation of tools makes the attacker itself an LLM: PAIR iteratively refines jailbreaks against a target, and Crescendomation automates the multi-turn escalation pattern, in its paper beating prior single-turn methods by 29 to 61 points on some models. The practical pattern in 2026 is to run an automated suite on every release candidate, nightly if the system prompt changes often, and reserve human red-teamers for quarterly deep dives and new feature surfaces.

Practical takeaways#

Map your risks against the OWASP Top 10 for LLM applications, where prompt injection sits at LLM01, and write the forbidden-behavior list before you attack. Red-team before launch with people who did not build the system, and probe your tools and retrieval paths, not just text harms. Convert findings into an automated suite, measure ASR at a fixed attempt budget, and fail the release when it regresses. Log blocked and successful attempts through your observability stack, because post-launch traffic is the red team you do not control. The adversaries keep attacking after launch; your measurement has to keep running too.