How do RLHF, DPO, and KTO compare in terms of scalability and performance across standard reasoning benchmarks for frontier LLMs?
Online reinforcement learning (RL) methods, such as RLHF using Proximal Policy Optimization (PPO) or Group Relative Policy Optimization (GRPO), are the current gold standard for maximizing performance on complex reasoning benchmarks like AIME24 and MATH-500 Verified Answer #1, Verified Answer #2. These methods utilize an active exploration mechanism, allowing the model to generate novel token trajectories and discover logical shortcuts or self-correction capabilities during training Verified Answer #1, Verified Answer #3. While PPO is highly effective for reasoning, it is the least scalable method because it is compute-bound, requiring up to four models (policy, reference, reward, and critic) to be maintained in memory simultaneously Verified Answer #2, Verified Answer #4. GRPO improves this scalability by removing the critic model and bootstrapping advantages from multiple rollouts, which reduces memory overhead by roughly 50% Verified Answer #4.
Direct Preference Optimization (DPO) and Kahneman-Tversky Optimization (KTO) offer superior training scalability compared to traditional RLHF because they bypass the need for a separate reward model and online reinforcement learning loops Verified Answer #4. DPO uses an offline contrastive classification loss over paired preference data, while KTO maximizes utility using binary feedback Verified Answer #4. However, these methods are structurally limited in reasoning tasks because they are discriminative and rely on static datasets Verified Answer #1. They excel at aligning models with human preferences but struggle to discover correct multi-step reasoning chains that are not already present in the training data Verified Answer #1.