GRPO

GRPO stands for Group Relative Policy Optimization. It keeps the basic reinforcement-learning loop from RLHF: sample outputs, score them, and update the policy toward better outputs. The difference is that it does not train a separate critic or value model. It estimates advantage by comparing samples within the same prompt group.

The group-relative idea#

For one prompt, sample several candidate answers. Score each answer with a reward function. Then normalize those rewards within the group: above-average samples get positive advantage, below-average samples get negative advantage. The model is nudged toward the winners.

What it removes from PPO#

PPO typically uses a value model to estimate expected reward. That model costs memory and compute, which matters when the policy is already large. GRPO removes that critic. In verifiable domains like math or code, the reward can come from exact checks, tests, or answer matching, so the group comparison is enough to produce a useful learning signal.

Why it mattered for reasoning#

DeepSeek-R1 made GRPO famous because it showed large-scale reasoning improvements from RL on verifiable rewards. The model samples long chains, gets rewards for correct final answers, and learns behaviors that improve success: checking work, exploring alternatives, and using more test-time computation.

Pathologies#

GRPO can reward length if longer answers correlate with passing checks. It can also become unstable when all samples in a group are wrong or when reward normalization creates misleading advantages. DAPO and Dr. GRPO style variants adjust clipping, length normalization, and reward handling to reduce these issues.

Practical takeaways#

GRPO is useful when you can score outputs cheaply and objectively. It is not a general replacement for preference training. Pair it with RLVR domains, monitor length and reward hacking, and treat the verifier as part of the model's training surface.