LLM alignment methods

Post-training alignment for frontier Large Language Models (LLMs) has transitioned into a specialized pipeline architecture where methods are selected based on specific objectives, such as human preference or logical discovery Verified Answer #1. The primary approaches include Reinforcement Learning from Human Feedback (RLHF) variants like Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), as well as Direct Preference Optimization (DPO) and Kahneman-Tversky Optimization (KTO) Verified Answer #1Verified Answer #2.

Online Reinforcement Learning Methods

Online RL methods, such as PPO and GRPO, are considered the gold standard for maximizing performance on complex reasoning benchmarks like AIME24 and MATH-500 Verified Answer #1. These methods utilize active exploration, generating new response trajectories during training that allow the model to discover novel logical shortcuts and self-correct through iterative rollouts Verified Answer #1.

Proximal Policy Optimization (PPO)

• PPO requires training a standalone reward model on human preference data before the reinforcement learning phase Verified Answer #2.
• During training, the LLM generates novel responses (rollouts) which are scored by the reward model to update the policy Verified Answer #2.
• PPO is resource-intensive and requires maintaining multiple models in memory, including the actor, critic, reward, and reference models Verified Answer #2.
• Despite its complexity, PPO consistently achieves a higher performance ceiling on mathematical benchmarks, outperforming DPO by an average of up to 2.5% Verified Answer #2.

Reinforcement Learning with Verifiable Rewards (RLVR)

Frontier reasoning breakthroughs are increasingly utilizing RLVR, where rewards are based on ground truth outcomes like code execution success or correct mathematical answers Verified Answer #1. This approach eliminates the instability of learned reward models and reduces the risk of reward hacking Verified Answer #1.

Offline Optimization Methods

DPO and KTO are classification-based methods that optimize a model to increase the probability of preferred paths found within a static dataset Verified Answer #1.

• Alignment Utility: These methods are highly effective at learning human preferences and instruction following Verified Answer #1Verified Answer #2.
• Reasoning Limitations: Because they rely on static data, they are structurally limited in discovering multi-step reasoning chains not explicitly present in the training set Verified Answer #1.
• Operational Efficiency: DPO and KTO offer greater operational simplicity compared to online RL methods Verified Answer #1.