Reinforcement Learning from Human Feedback (RLHF) Demystified

When GPT-3.5 was released, the improvement over raw GPT-3 was striking - not in raw capability, but in usability. The model followed instructions. It was helpful rather than pedantic. It acknowledged limitations. That transformation came from RLHF: Reinforcement Learning from Human Feedback. Understanding it changes how you build with these models.

The Problem RLHF Solves

A language model trained purely to predict next tokens on internet text learns to mimic all of internet text - including unhelpful, toxic, and false content. Ask it to help you and it might respond with a forum post or a fictional story, because those are patterns it learned from.

RLHF bridges the gap by using human preferences as the training signal: given an instruction, produce a response a human would rate as helpful, harmless, and honest.

The Three-Stage Process

Stage 1: Supervised Fine-Tuning (SFT)

Start with a pretrained base LLM. Collect (instruction, ideal response) pairs - human demonstrators write high-quality responses to a variety of prompts. Fine-tune the base model on this dataset. This produces an SFT model that follows instructions but is imperfect.

Stage 2: Reward Model Training

Collect pairwise preference data: show human labelers two responses to the same prompt, ask which is better. Train a reward model (RM) that takes (prompt, response) as input and outputs a scalar reward score. Training objective: score preferred responses higher than rejected responses.

import torch

def reward_model_loss(reward_chosen: torch.Tensor, reward_rejected: torch.Tensor) -> torch.Tensor:
    # Bradley-Terry pairwise ranking loss
    return -torch.log(torch.sigmoid(reward_chosen - reward_rejected)).mean()

Stage 3: RL Fine-Tuning with PPO

Use the reward model as the reward function for RL. For each training step: sample a prompt, generate a response, score it with the reward model, update the policy with PPO. A KL divergence penalty prevents reward hacking - the policy exploiting the reward model by generating responses that score highly but look nothing like useful text.

What RLHF Does to Model Behavior

Why models hedge and qualify: Human raters preferred responses that acknowledged uncertainty. The reward model learned this preference. RLHF-trained models hedge more - sometimes appropriately, sometimes excessively.

Why models follow multi-part instructions reliably: Raters penalized missing instruction parts. The policy learned to be more thorough.

Why models have specific refusal patterns: Raters rated harmful responses as bad. The specific phrasing of refusals is learned behavior from preference data.

Direct Preference Optimization (DPO): A Simpler Path

PPO works well, but its three-stage process, especially the RL fine-tuning, can be complex. You need to manage a separate reward model and an active policy model. This adds compute overhead and training instability. Direct Preference Optimization (DPO) offers a different approach. Instead of training a separate reward model and then using RL, DPO directly optimizes the language model's policy using the human preference data. It re-frames the problem. You still collect the same pairwise preference data: which response is better. But DPO derives a loss function that directly adjusts the policy to assign higher probabilities to preferred responses and lower probabilities to rejected ones. This removes the need for PPO, the KL divergence penalty, and

The Logistics of Human Feedback

Building the datasets for RLHF is a massive undertaking. For the Supervised Fine-Tuning stage, human writers craft thousands of ideal responses. They follow detailed guidelines. This ensures the model learns good initial behaviors. Then for the Reward Model, you need human labelers to compare outputs. Imagine showing a labeler two different ways an AI answered a prompt. They pick the better one. This seemingly simple task scales quickly. A project might require tens of thousands of these pairwise comparisons. Companies like Scale AI or Appen often provide these labeling services. You're not just looking for "good" responses, but also for specific failure modes. You want to train the reward model to spot subtle errors or unhelpful outputs. This data collection is a continuous, expensive process. It demands careful quality control to ensure labelers remain consistent in their preferences.

Beyond 'Helpful': Tailoring Specific Behaviors

While "helpful, harmless, honest" covers the basics, RLHF lets you fine-tune for much more nuanced behaviors. For example, when I worked on code generation models, we didn't just want "correct" code. We wanted idiomatic code. We wanted efficient code. We wanted code that included comments where appropriate. You can collect preference data specifically for these attributes. Labelers might prefer a Python snippet using a list comprehension over a traditional loop, even if both work. Or they might prefer a more concise explanation in a summarization task. This specific feedback refines the reward model. Then the RL policy learns to generate outputs that align with those detailed preferences. This allows you to sculpt the model's output for very specific use cases, like generating valid JSON or adhering to a particular brand's tone of voice.

The Continuous Cycle of Alignment

RLHF isn't a one-time training event. It's an ongoing process. After an RLHF-trained model ships, you start seeing how real users interact with it. Sometimes, new failure modes appear. Users might find ways to "jailbreak" the model or elicit unhelpful responses. This is where "red teaming" comes in. Teams actively try to break the model, find its weaknesses, and document them. These new problematic prompts and responses become fresh data. This new data feeds back into the system. You might collect more SFT examples for edge cases. You'll definitely collect more preference data to refine the reward model. This iterative loop helps your models stay aligned as their capabilities grow and as user expectations evolve. It's a constant effort to chase the moving target of "good" behavior.

RLHF's Limitations

Sycophancy. Human raters tend to prefer responses that agree with their prior beliefs and are flattering. RLHF models learn to tell you what you want to hear. Research from Anthropic (2023) showed that RLHF models would change their stated position on factual questions when users pushed back, even when the model's original answer was correct.

Reward hacking. The reward model is an imperfect proxy. Advanced policies find ways to score high on the RM that don't satisfy human preferences. The KL penalty mitigates but doesn't eliminate this.

Labeler disagreement. Humans disagree on what constitutes a good response. Averaging across labelers loses the signal of genuine disagreement.

DPO: The Simpler Alternative

Direct Preference Optimization (DPO, Rafailov et al., 2023) makes a key insight: the optimal RLHF policy has a closed-form solution. Instead of training a separate reward model and running RL, you can directly optimize the policy on preference data.

def dpo_loss(
    policy_logprob_chosen,
    policy_logprob_rejected,
    ref_logprob_chosen,
    ref_logprob_rejected,
    beta: float = 0.1
):
    chosen_ratio = policy_logprob_chosen - ref_logprob_chosen
    rejected_ratio = policy_logprob_rejected - ref_logprob_rejected
    return -torch.log(torch.sigmoid(beta * (chosen_ratio - rejected_ratio))).mean()

DPO advantages: no separate reward model, no RL instability, 2-5x less compute, easier to implement. DPO is now the dominant approach for fine-tuning open-source models with preference data.

from trl import DPOTrainer, DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")

training_args = DPOConfig(
    beta=0.1,
    output_dir="./dpo-output",
    num_train_epochs=1,
    per_device_train_batch_size=4,
    learning_rate=5e-7
)
trainer = DPOTrainer(
    model=model, args=training_args,
    train_dataset=preference_dataset,
    tokenizer=tokenizer
)
trainer.train()

What This Means for Product Teams

  1. Refusal patterns are learned, not hardcoded. Frame your legitimate use case to distinguish itself clearly from what the model was trained to treat as harmful.
  2. Sycophancy is a real failure mode. Explicitly prompt models to challenge your assumptions: "Play devil's advocate on this proposal."
  3. Preference data is valuable for fine-tuning. DPO on 1,000 high-quality preference pairs often beats SFT on 10,000 demonstrations.
  4. Constitutional AI is the frontier. Anthropic's approach uses AI feedback with explicit principles - scaling better than human labeling.

RLHF is the invisible force shaping every interaction you have with a modern LLM. Understanding it makes you better at prompt engineering, more realistic about model limitations, and better equipped to make fine-tuning decisions when off-the-shelf behavior doesn't meet your needs.

Further Reading