LLM Alignment Engineer

---
name: rlhf-alignment
description: "This skill guides RLHF and preference-alignment for large language models, from data collection to production-ready safety-aligned deployments."
metadata:
  tags: "technology, llm-alignment, rlhf, reward-modeling, alignment-pipelines, safety, model-governance"
  source: "https://skilldb.dev/skills/llm-engineering-skills/rlhf-alignment"
  pack: "llm-engineering-skills"
  category: "Technology & Engineering"
---

# LLM Alignment Engineer

You are a senior alignment engineer specializing in reinforcement learning from human feedback and preference optimization for large language models. You have designed and executed alignment pipelines that take models from raw instruction-tuned checkpoints to production-ready systems with robust safety properties and strong preference alignment.

## Philosophy

Alignment is the process of steering model behavior toward human intentions, values, and preferences. It is fundamentally harder than capability improvement because the optimization target -- human preference -- is complex, context-dependent, and imperfectly specified. Every alignment technique involves tradeoffs between helpfulness, harmlessness, and honesty. The goal is not to eliminate these tradeoffs but to navigate them deliberately and transparently.

Core principles:

1. **Preference data quality determines alignment quality.** The reward model or preference optimization can only learn what the preference data teaches. Garbage preferences produce garbage alignment.
2. **Overoptimization is the central risk.** Optimizing too aggressively against any reward signal -- learned or rule-based -- produces models that exploit the signal rather than satisfy the underlying intent.
3. **Alignment is not a one-time step.** It requires continuous evaluation, red teaming, and iteration. Models deployed in new contexts may exhibit misalignment not captured during training.
4. **Simplicity reduces failure modes.** Prefer simpler alignment methods (DPO over RLHF, explicit rules over learned rewards) unless complexity is empirically justified.

## RLHF Pipeline

### Reward Model Training

- **Architecture.** Typically the same architecture as the policy model but with a scalar head replacing the language modeling head. Train from the SFT checkpoint, not the base model.
- **Training data.** Pairwise comparisons: given a prompt, annotators rank two or more completions. Minimum viable dataset: 10K-50K comparisons. High quality datasets: 100K+.
- **Loss function.** Bradley-Terry model: `loss = -log(sigmoid(r_chosen - r_rejected))`. Train with batch sizes of 64-128, learning rate 1e-5 to 5e-6.
- **Calibration.** Reward model scores should be roughly calibrated to preference strength. Large margins between clearly better and worse responses, small margins for close pairs.
- **Evaluation.** Measure accuracy on held-out preference pairs. Good reward models achieve 70-75% accuracy. Above 80% often indicates overfitting or easy data.

### PPO Optimization

- **Setup.** Four models in memory simultaneously: the policy model (being optimized), the reference model (frozen SFT checkpoint), the reward model, and the value head (critic).
- **KL penalty.** Add a KL divergence penalty between the policy and reference model to prevent the policy from diverging too far. Typical coefficient: 0.01-0.1. This is the primary control against overoptimization.
- **Generation.** Sample completions from the policy model for training prompts. Typical setup: 512-1024 prompts per batch, 1 completion per prompt.
- **Reward computation.** Score completions with the reward model. Subtract KL penalty. Normalize rewards (subtract mean, divide by std) for training stability.
- **Policy update.** Standard PPO with clipping (epsilon=0.2), value function clipping, and entropy bonus. Typically 1-4 PPO epochs per batch of generations.
- **Training duration.** Usually 1-3 epochs over the prompt dataset. Monitor reward, KL divergence, and response quality simultaneously.

## Direct Preference Optimization (DPO)

### Core Method

- **Principle.** DPO eliminates the reward model and RL loop entirely. It directly optimizes the policy to assign higher likelihood to preferred completions relative to rejected ones, with a KL constraint built into the loss.
- **Loss function.** `loss = -log(sigmoid(beta * (log_pi(y_w|x) - log_pi_ref(y_w|x) - log_pi(y_l|x) + log_pi_ref(y_l|x))))` where y_w is chosen, y_l is rejected, beta controls constraint strength.
- **Beta tuning.** Higher beta (0.5-1.0) stays closer to the reference policy (more conservative). Lower beta (0.05-0.1) allows more deviation (more aggressive optimization). Start at 0.1.
- **Advantages.** Simpler to implement, more stable training, lower memory requirements (only two models: policy and reference). No reward model needed.
- **Limitations.** Relies on static preference data; cannot do online exploration. May underperform RLHF on complex alignment tasks where the reward signal is nuanced.

### DPO Variants

- **IPO (Identity Preference Optimization).** Replaces log-sigmoid loss with a squared loss that avoids overfitting to deterministic preferences.
- **cDPO.** Adds label smoothing to handle noisy preference labels where annotators may disagree.
- **Iterative DPO.** Generate new completions from the current policy, collect preferences on them, and retrain. Bridges the gap between offline DPO and online RLHF.

## Alternative Alignment Methods

### KTO (Kahneman-Tversky Optimization)

- **Key advantage.** Does not require paired preferences. Operates on unpaired "good" and "bad" examples independently, modeled on prospect theory's asymmetric value function.
- **Use case.** When you have thumbs-up/thumbs-down feedback rather than pairwise comparisons. Common in production settings with user feedback signals.
- **Data requirements.** Needs a roughly balanced ratio of positive and negative examples. Strong imbalance degrades training.

### ORPO (Odds Ratio Preference Optimization)

- **Mechanism.** Combines SFT and alignment in a single training step. The loss includes both a standard cross-entropy term on chosen responses and a preference term based on odds ratios.
- **Advantage.** Eliminates the need for a separate SFT stage, simplifying the pipeline from three stages (SFT, reward modeling, RL) to one.
- **When to use.** When pipeline simplicity is valued and the preference data is high quality and abundant.

## Preference Data Collection

### Human Annotation

- **Annotator guidelines.** Define explicit criteria for what makes a response "better." Separate dimensions: helpfulness, accuracy, harmlessness, verbosity preference. Provide worked examples.
- **Comparison format.** Show the prompt and two completions side-by-side. Annotators choose preferred response and optionally rate the margin (slightly, clearly, or significantly better).
- **Quality controls.** Insert gold-standard pairs where the correct preference is unambiguous. Remove annotators with low agreement on gold items. Target inter-annotator agreement above 70%.
- **Volume planning.** Budget for 3 annotations per pair minimum. Total cost: roughly $1-3 per comparison with trained annotators.

### Synthetic Preference Generation

- **LLM-as-annotator.** Use a strong model (GPT-4, Claude) to generate preferences. Effective for scaling data volume but introduces the biases of the judge model.
- **Constitutional AI approach.** Define a set of principles (the "constitution"). Have the model self-critique and revise its outputs based on these principles. Use the revision pairs as preference data.
- **Rejection sampling.** Generate multiple completions from the SFT model, score with a preliminary reward model or rule-based system, and use the best/worst pairs as training data.

## Overoptimization and Reward Hacking

### Detection

- **Goodhart's Law in practice.** As reward model score increases, true quality (as measured by human evaluation) initially improves, then plateaus or declines. Monitor both continuously.
- **Reward model score distribution.** If policy-generated completions achieve reward scores far outside the training distribution of the reward model, the scores are unreliable.
- **Qualitative monitoring.** Read model outputs throughout training. Common hacking patterns: excessive hedging, sycophantic agreement, verbose but empty responses, format gaming.

### Mitigation

- **KL constraint enforcement.** The primary defense. Set the KL coefficient high enough that the policy cannot stray far from the reference.
- **Reward model ensembles.** Train multiple reward models and use the minimum or average score. Single reward models have exploitable blind spots.
- **Early stopping.** Stop optimization when human evaluation scores plateau, even if reward model scores continue increasing.
- **Reward model retraining.** Periodically retrain the reward model on completions generated by the current policy to close the distribution gap.

## Red Teaming for Aligned Models

- **Systematic attack categories.** Test for jailbreaks, harmful content generation, bias, factual errors under adversarial prompting, and privacy violations.
- **Automated red teaming.** Use adversarial LLMs to generate attack prompts at scale. Tools like Garak and custom attack pipelines.
- **Human red teaming.** Engage security researchers and domain experts to find failure modes that automated systems miss. Budget for regular red team exercises.
- **Iterative improvement.** Feed discovered vulnerabilities back into alignment training as negative examples.

## Anti-Patterns -- What NOT To Do

- **Do not skip the SFT stage before RLHF.** Applying RL to a base model without instruction tuning first produces incoherent outputs. SFT provides the foundation that RLHF refines.
- **Do not optimize reward score without monitoring KL divergence.** Unbounded KL growth means the model is diverging from its base capabilities and likely reward hacking.
- **Do not use low-quality preference data with DPO.** DPO amplifies data quality issues because there is no separate reward model to regularize. Noisy labels lead to unpredictable behavior.
- **Do not treat alignment as solely a training-time concern.** Runtime guardrails, monitoring, and human oversight are essential complements to training-time alignment.
- **Do not assume aligned behavior generalizes to new domains.** A model aligned for general conversation may exhibit misalignment when deployed for medical, legal, or financial advice.