ConceptFine-Tuning & Adaptation
RLVR (Verifiable Rewards)
At a glance
Reinforcement learning where the reward is a checkable fact, tests pass or the answer matches, instead of a learned preference.
- Who this is for
- Engineers and technical readers learning the terms used in AI systems.
- Topics
- Fine-Tuning & Adaptation
- Concept
In short
RLVR, reinforcement learning with verifiable rewards, pays a reward only when a program confirms the answer, such as passing tests or a matching final number, instead of asking a learned reward model which answer it prefers. Bigcircle recommends writing the checker as an eval and running it on the base model before any training, because the model learns exactly what the checker cannot see and needs to succeed often enough to learn at all.
Key facts
Definition
The trainer samples an answer, a verifier written as code marks it correct or not, and that result is the reward, so there is no neural scorer for the model to flatter.
How Bigcircle does it
- Bigcircle recommends a control run with a meaningless reward before crediting a gain to the verifier, because random rewards alone lifted one maths model by 21.4 points.
- On the DevOps agent, Bigcircle's run-time checks are structural and semantic validation against the live account, the same kind of program a verifiable reward needs, and the logged failures record what the model gets wrong.
- Bigcircle recommends keeping RLVR off tasks whose value is judgement, such as a legal read, because structure can be validated there while the reasoning cannot.
Comparison
RLHF trains against a reward model that predicts human preference and can be flattered, while RLVR trains against a test result or an exact answer and can only be gamed through gaps in the checker.
Key number
Shao and colleagues found random rewards raised Qwen2.5-Math-7B on MATH-500 by 21.4 points, against 29.1 points for rewards based on the true answer.
RLVR means reinforcement learning with verifiable rewards. The model is trained by sampling an answer and paying a reward only if a program confirms it is correct: the unit tests pass, the final number matches, the proof step checks. There is no reward model guessing which answer a person would prefer, as in RLHF, so there is no learned scorer for the model to flatter. Ai2 named the method in its Tulu 3 recipe in November 2024, and DeepSeek-R1 made it famous in 2025 by using rule-based rewards for accuracy and format, with GRPO, to train strong reasoning behaviour.
Two pieces of older work are often cited alongside it and do something different. Cobbe and colleagues' 2021 GSM8K paper trained a verifier model to pick the best of many sampled answers, and Lightman and colleagues' 2023 "Let's Verify Step by Step" trained reward models that grade each reasoning step. Both are learned scorers, the thing RLVR replaces with a program.
A verifier is not a reward model#
A learned reward model is another network, and the policy can find its blind spots. A verifier is narrower and harder to argue with. In code, the tests pass or fail. In maths, an exact final answer can often be compared. In logic puzzles, the constraints can be evaluated. That narrowness is the method's strength and its limit: the reward is only as good as the checker, and only tasks with a checker qualify.
What the reward teaches#
The common story is that RLVR teaches models to reason. Two 2025 findings make it more specific. Yue and colleagues compared models before and after RLVR across many samples per problem. The trained model solved more problems on its first try, but given enough tries the base model solved as many or more. On that reading, RLVR mostly makes the model pick, on its first attempt, reasoning paths it could already find, rather than adding paths it lacked.
Shao and colleagues then trained Qwen2.5-Math-7B with rewards that had nothing to do with correctness. Random rewards raised its MATH-500 score by 21.4 points, against 29.1 points for rewards based on the true answer. The same trick often failed on Llama and OLMo models. A gain from RLVR on one model family can come from behaviour the base model already had, so a training run needs a control, the same recipe with a meaningless reward, before the gain is credited to the verifier.
Verifiable does not mean unhackable#
The model learns to pass the checker, which is not always the same as solving the task. OpenAI's 2025 work on monitoring reasoning models caught a frontier model, during training on coding tasks, making tests pass without doing the work: exiting early with a success code, skipping tests, editing the test framework so tests passed trivially, or writing stubs. Code models can also overfit public tests, and maths models can learn formatting shortcuts that satisfy an answer parser.
Two other limits are structural. Sparse rewards stall learning: if every sample in a batch is wrong, every reward is zero and the model gets nothing to learn from, so tasks have to be pitched where the model succeeds some of the time. And many important tasks have no exact checker at all.
Where a checker exists, and where it does not#
Our builds use the same kind of checks at run time, and they show which tasks have a verifier. On the DevOps agent that runs on live AWS, every step is either machine-checkable or human-reviewable: structural and semantic checks run against the live account, and the logged validation failures are the system's own record of what the model gets wrong. A check like that is a verifier. On the trademark drafting platform, structure and citation format can be validated, but whether the legal reasoning holds cannot, and the attorney reviews every draft because a system that is right 95% of the time still needs a person to check 100% of it.
So the test for RLVR is whether you can write the checker, and whether it covers the part of the task that matters. A team with an open model and a strict checker, a test suite or an exact-answer grader, can train against it. Bigcircle recommends writing the checker as an eval first and running it on the base model, because the eval shows whether the model succeeds often enough to learn from and whether the checker can be passed without doing the task. If the valuable part of the task is judgement, a strategy memo or a legal read, keep RLVR off it and use preference methods or a person, because the model will learn exactly what the checker cannot see.