LLM Red Teaming & Jailbreaks

LLM red teaming is attacking your own model-backed system on purpose, before users or attackers do, to find the inputs that make it break its own rules. A jailbreak is one of those inputs: a prompt, or a run of prompts, that gets the model to ignore its safety instructions. The result of a red team is a number, the attack success rate, meaning the share of attempts that produced the forbidden behaviour, measured against a stated set of attacks at a stated number of tries.

Most published red teaming is about text, whether a model can be talked into writing something it should refuse. For a system that can act, that aims the effort at the wrong layer. If the assistant can call a refunds API, the attack worth finding is the roleplay that convinces it the user is an administrator, and the question worth answering is whether the refund can go through at all.

How jailbreaks work#

Most jailbreaks share a root cause with prompt injection. The model reads instructions and content as one stream of tokens and cannot reliably tell a real rule from a planted one. The attacks differ in how they plant it. Test each family separately, because a model hardened against one is not hardened against the others.

Persona attacks wrap the request in fiction, a scene in a film or an "unrestricted" assistant, so the model's willingness to play along outweighs its refusal. Obfuscation hides the intent from refusal training by encoding the request, splitting it into fragments, or translating it into a language with little safety data. Indirect attacks put the instruction inside something the model is asked to read, a retrieved document, a web page or a tool result.

Two attacks show why a single test prompt proves little. Anthropic's many-shot work filled long prompts with fake dialogues in which an assistant complies, tested up to 256 of them, and found success rose with the count along a power law. A step that classified and rewrote the prompt before the model saw it cut one measured attack success rate from 61 percent to 2 percent, which puts the defence in front of the model rather than in its instructions. Crescendo, from a USENIX Security 2025 paper, opens with a harmless question and escalates a little each turn, building on the model's own previous answers. Its automated version broke most tasks in fewer than five turns on average. No single message looks alarming, so a filter that reads one message at a time passes every one of them.

Rank findings by what the system can do#

Bigcircle ranks a red-team finding by the action it reaches, not by how bad the text is, because the tool layer is the part we design to hold when the model does not. On the insurance voice agent we built for an enterprise insurer, compliance sits in the tool contract rather than the prompt. "Will the model always make the disclosure?" cannot be answered. "Can this operation execute without the disclosure?" can, and the answer is no, so a request that fails the check escalates instead of running. On the DevOps agent that runs on live AWS, there is no general shell and no path by which a plan step becomes an arbitrary API call.

That changes what a red team is looking for. A jailbreak that convinces the model and then hits a tool that refuses the call is a text finding. A jailbreak that reaches a tool call with the wrong arguments, or pulls another user's records out of retrieval, blocks the launch. So the list of forbidden behaviours comes in two halves, written before anyone attacks: the generic harm categories from your content policy, and the abuse that only your application allows, which no public benchmark covers.

Each finding then goes to the layer that fixes it. That may be a classifier in front of the model, a narrower tool permission, a schema check on tool arguments, or a refusal fine-tune for that pattern. A stronger system prompt is the weakest of these, because the attack has already beaten the system prompt. Use people who did not build the system for the manual work, because builders test the paths they designed, and file every break like a bug with the transcript attached.

An attack success rate needs its conditions#

The number depends on the attacker's budget. A model that resists one attempt can still fall to a persistent attacker making a hundred variations, and the many-shot results show success climbing smoothly with effort. Report the rate at a fixed number of attempts against a named set of attacks, or two numbers from two weeks cannot be compared. HarmBench and JailbreakBench exist to standardise that comparison. JailbreakBench, from the NeurIPS 2024 datasets track, fixes 100 forbidden behaviours and keeps a public repository of attack prompts.

Bigcircle recommends failing a release when the rate rises at the same attempt budget, because the release that regresses is usually the one nobody tested for it. A model swap or a system-prompt rewrite can hold every generic quality score in your evals while the attack rate goes up, and only an adversarial suite that runs before release sees the change.

Scanners for coverage, people for new attack classes#

Automation gives repeatability and people find what is new, and a release process needs both. NVIDIA's open-source garak, as of October 2026, lists 21 probe families, among them encoding tricks, DAN-style personas, prompt injection and replay of training data, and reports the results the way a web scanner audits a server. A second generation of tools makes the attacker a model too. PAIR refines a jailbreak against the target and often succeeds in fewer than twenty queries, and the automated Crescendo reported 29 to 61 percent higher success than earlier methods against GPT-4.

Run a scanner on every release candidate and every system-prompt change. Treat a clean scan as a floor, because a scanner knows only the attacks someone has already written a probe for. Bring people in when a new tool, data source or user role ships, because that is when a new attack class becomes possible and no scanner has a probe for it yet.

Keep logging attacks after launch#

Red teaming before launch tests the attacks you thought of. Production traffic contains the rest. Log blocked and successful attempts through your observability stack with the same transcript detail as a pre-launch finding, and feed the successful ones back into the automated suite so the next release is tested against them. For mapping risks, OWASP published the 2026 edition of its LLM Top 10 on 3 August 2026. Prompt injection was LLM01 in the 2025 edition. Test your guardrails against the logged attacks as well as the planned ones.