Curiosity · Concept
Reinforcement Learning from AI Feedback (RLAIF)
Reinforcement Learning from AI Feedback (RLAIF), introduced by Google Research in 2023, replaces the expensive human-labelling step of RLHF with an AI 'labeller'. A capable model is prompted with a constitution or rubric and asked to pick the better of two candidate responses; those AI preferences train a reward model or drive direct RL. Bai et al.'s Constitutional AI used RLAIF to produce harmless-yet-helpful models; Lee et al. showed RLAIF can match human-feedback RLHF on summarisation and dialogue at a fraction of the cost.
Quick reference
- Proficiency
- Intermediate
- Also known as
- RLAIF, AI-feedback RL
- Prerequisites
- RLHF, Constitutional AI
Frequently asked questions
What is RLAIF?
Reinforcement Learning from AI Feedback is a post-training method where an AI labeller produces preference judgments between pairs of model completions, and those preferences are used to train a reward model or guide RL directly.
How does RLAIF compare to RLHF?
RLHF uses humans to rank outputs; RLAIF uses a capable AI model prompted with a rubric or constitution. Google's 2023 study showed RLAIF can match RLHF on summarisation and dialogue while being cheaper and more scalable.
Isn't RLAIF just distilling one model into another?
There's overlap, but RLAIF uses the labeller model only to rank pairs — the trained model is still doing the reasoning. This transfers judgement, not full knowledge. It also lets you align a model stronger than the labeller on narrow tasks.
Where does Constitutional AI fit in?
Constitutional AI (Anthropic, 2022) is an early RLAIF-style method: the model critiques and revises its own responses against a written constitution, and those self-generated preferences train a harmless reward model.
Sources
- Lee et al. — RLAIF vs RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback — accessed 2026-04-20
- Bai et al. — Constitutional AI: Harmlessness from AI Feedback — accessed 2026-04-20