ConceptFine-Tuning & Adaptation
Constitutional AI & RLAIF
At a glance
Align a model against a written set of principles by having it critique and revise its own outputs, then train on the result.
- Who this is for
- Engineers and technical readers learning the terms used in AI systems.
- Topics
- Fine-Tuning & Adaptation
- Concept
In short
Constitutional AI is Anthropic's method for training a model against a written list of principles: a model critiques and revises answers against the text, then labels which of two answers follows it better, and reinforcement learning on those labels is called RLAIF. Bigcircle recommends treating a constitution, or a system prompt written in the same spirit, as the layer that makes good behaviour likely, and enforcing anything a contract or regulator requires in code, because no written text makes one behaviour certain.
Key facts
Definition
Constitutional AI, introduced by Anthropic in December 2022, replaces human preference judgements with a written constitution that a model uses to critique, revise and rank answers, and the resulting data trains the assistant.
How Bigcircle does it
- Bigcircle put required disclosures in the tool contract of its insurance voice agent, because whether a model will always make a disclosure cannot be answered and whether an operation can run without it can.
- Bigcircle recommends keeping evals, red teaming and monitoring beside any constitution-trained model, because written principles are incomplete, they conflict, and two models can read the same sentence differently.
Comparison
RLHF learns from thousands of human judgements whose standard nobody can read, while Constitutional AI writes the standard down and lets a model apply it, so the target can be reviewed and changed.
Constitutional AI is a method for training a model against a written list of principles, the constitution, instead of relying only on human judgements of which answer is better. Bai and colleagues at Anthropic introduced it in December 2022. A model uses the principles to critique answers, revise them and choose between them, and the resulting data trains the assistant. The difference from standard RLHF is that the target is written down: thousands of human preference labels add up to a standard nobody can read, while a constitution can be read, argued over and changed.
Phase 1: critique and revise#
The first phase starts from an assistant's answer to a prompt, often a harmful one. A model is asked to critique the answer against one principle and then to write a revision that fixes what the critique found. The revised answers become supervised fine-tuning data. The principle is plain text, so a reviewer can read it, argue about it, change it and test the change.
Phase 2: RLAIF#
The second phase replaces human preference labels with model-written ones. A model sees two candidate answers and the constitution and picks the one that follows it better. Those choices train a preference model, and the assistant is then trained with reinforcement learning against that preference model. The paper named this reinforcement learning from AI feedback, RLAIF. In the 2022 paper the constitution covered harmlessness only, and the labels for helpfulness still came from people.
Whether AI labels can stand in for human ones has been tested outside Anthropic. Lee and colleagues at Google reported in 2023 that RLAIF performed comparably to RLHF on summarization and on helpful and harmless dialogue. Later work trains preferences directly with DPO rather than a separate preference model, and the label source stays the same idea.
The 2026 constitution#
On 22 January 2026 Anthropic published a new constitution for Claude, replacing the version it published in May 2023. The earlier one was, in Anthropic's words, "a list of standalone principles". The new one explains its reasoning, because Anthropic wants the model to understand why it should act a certain way. It sets a priority order for conflicts: broadly safe first, then broadly ethical, then compliant with Anthropic's guidelines, then genuinely helpful. A small set of hard constraints covers behaviour the model should never engage in. The text is released under CC0, and Anthropic says Claude itself uses it to construct many kinds of synthetic training data, including rankings of possible responses.
For a team building on Claude, the priority order says which way a conflict resolves. The constitution treats following an operator's instructions as part of being helpful, the last of the four, so a system prompt that pulls against safety or ethics as the model reads them loses. Design the product so that no feature depends on winning that conflict, and test the edge cases where a legitimate request looks like one the constitution steers away from.
A constitution can also train a guard rather than the model. Anthropic's Constitutional Classifiers, described in a January 2025 paper, use a written constitution to generate data for classifiers that screen inputs and outputs for jailbreak attempts. Collective Constitutional AI, in October 2023, asked about 1,000 Americans to propose and vote on principles and trained a model on the result.
What a constitution does not settle#
Written principles are incomplete, and they conflict, which is the case the 2026 priority order is written for. Two models can read the same sentence differently, so the same constitution does not produce the same behaviour in every model trained on it. A constitution shapes what the model tends to do. It does not make any one behaviour certain, and it sits beside evals, red teaming and monitoring rather than replacing them.
Rules that must hold go in code#
A product team usually meets this as a choice between writing a rule into the model's instructions and enforcing it in the system around the model. On the insurance voice agent, Bigcircle put required disclosures in the tool contract rather than in the prompt. Whether the model will always make the disclosure cannot be answered. Whether the operation can execute without it can be answered, and the answer is no. Treat a constitution, or a system prompt written in the same spirit, as the layer that makes the right behaviour likely. Anything a regulator or a customer contract requires belongs in a check that runs whether or not the model followed the text.