llm-preference-alignment-algorithms

Large language model (LLM) preference alignment aims to bridge the "alignment gap," which is the mismatch between static mathematical signals and the multi-dimensional nature of human experiential meaning Verified Answer #1. Modern alignment research has shifted from monolithic scalar reward modeling toward frameworks that decompose, contextualize, and direct preference learning Verified Answer #1.

Core Alignment Paradigms

Reinforcement Learning from Human Feedback (RLHF)

• Traditional RLHF typically utilizes Proximal Policy Optimization (PPO), which relies on online optimization Verified Answer #2.
• The model generates novel token trajectories during training, allowing for an exploration mechanism that triggers self-correction and deep reasoning Verified Answer #2.
• Online RL is considered highly effective for maximizing performance on complex reasoning benchmarks such as MATH-500 and AIME24 Verified Answer #2.
• Standard PPO is compute-bound, requiring the maintenance of policy, reference, reward, and critic networks in memory simultaneously Verified Answer #2 Verified Answer #3.
• Group Relative Policy Optimization (GRPO) improves scalability by eliminating the critic model and bootstrapping advantages from multiple parallel rollouts Verified Answer #2 Verified Answer #3.

Direct Preference Optimization (DPO)

• DPO maps the reward function directly to the policy, using an offline contrastive classification loss over a static dataset of paired preferences Verified Answer #2.
• This approach eliminates the need for a separate reward model and an online sampling loop Verified Answer #3.
• While computationally lightweight, DPO is constrained by the requirement for high-quality data annotation of chosen versus rejected pairs Verified Answer #3.

Kahneman-Tversky Optimization (KTO)

• KTO steers model behavior by maximizing the prospect-theoretic utility of single generations Verified Answer #3.
• It utilizes binary feedback rather than paired preferences to align the model Verified Answer #3.

Reward Engineering and Structural Decomposition

Reward Engineering is an emerging discipline that moves beyond standard preference modeling to account for human cognitive limitations and scalable oversight Verified Answer #4.

Cognitive-Aware Modeling

• Attention-Limited Reward Learning (ALRL) addresses the "bounded rationality" of human evaluators by modeling the feedback channel as a low-capacity process Verified Answer #4.
• ALRL introduces a latent variable for "attention capacity" to distinguish between objective reward differences and detection difficulty Verified Answer #4.
• This framework prevents models from overfitting on noise caused by human confusion, such as length bias or formatting Verified Answer #4.

Rule-Based and Rubric Frameworks

• Rule-Based Rewards (RBR) decompose high-level behavioral specifications into concrete, fine-grained, and composable propositions Verified Answer #1.
• RBR uses an LLM grader to evaluate rule satisfaction via few-shot prompts, providing a stable reward signal that bypasses the noisiness of human annotators Verified Answer #1.
• Rubrics as Rewards (RaR) is utilized for critical tasks like clinical or scientific reasoning where standard preference modeling may be insufficient Verified Answer #1.

Scalable Oversight

Scalable oversight is a recursive framework designed to overcome the bottleneck of human-in-the-loop evaluation as models transition into autonomous agent ecosystems Verified Answer #4. This architecture utilizes the model's own capabilities to assist in its own evaluation, with Recursive Reward Modeling (RRM) serving as a primary mechanism Verified Answer #4.