Fine-Tuning & Model Adaptation

I. The Alignment Tax & HHH

Pre-training on the Pile or Common Crawl optimizes for the objective $ \\max_\\theta P(x_{next} | x_{context}) $. This produces a "Base Model" capable of high-perplexity completion but prone to toxicity, hallucination, and disobedience. Alignment is the process of steering this probability distribution towards the HHH criteria: Helpful, Honest, and Harmless (Askell et al., 2021).

Accomplishing this without destroying the model's reasoning capabilities (catastrophic forgetting) is the central challenge. The Alignment Tax refers to the empirical observation that aggressive safety filtering often degrades performance on neutral benchmarks (e.g., lower code generation accuracy).

Supervised Fine-Tuning (SFT) The first step is Behavior Cloning. We fine-tune the base model on a high-quality dataset of (Instruction, Response) pairs. $$ \mathcal{L}_{SFT} = -\sum \log P_\theta(y | x) $$ This aligns the format but fails to capture the nuance of human preference, especially when multiple valid answers exist.

The Imitation Gap

SFT models often hallucinate because they are forced to mimic expert demonstrations even when they lack the underlying knowledge.

II. Low-Rank Adaptation (LoRA)

Full fine-tuning of a 70B parameter model requires updating all weights, which is computationally prohibitive. LoRA (Hu et al., 2021) freezes the pre-trained weights $ W_0 $ and injects trainable rank decomposition matrices:

$$ h = W_0 x + \frac{\alpha}{r} BAx $$ where $ W_0 \in \mathbb{R}^{d \times k} $, $ B \in \mathbb{R}^{d \times r} $, $ A \in \mathbb{R}^{r \times k} $, and $ \alpha $ is a scaling constant.

For a rank $ r=8 $, the number of trainable parameters is reduced by >99.9%. Crucially, because $ BA $ is linear, it can be merged back into $ W_0 $ during inference, introducing zero latency overhead.

QLoRA (Dettmers et al., 2023) extends this by quantizing $ W_0 $ to 4-bit NormalFloat (NF4) and fine-tuning the adapters, allowing a 65B model to be fine-tuned on a single 48GB GPU.

Rank Decomposition Visualizer

Simulate the parameter savings. The dense matrix $ W $ is approximated by the product of two low-rank matrices.

Rank ($ r $)

r = 4

Matrix A ($ d \times r $)

Matrix B ($ r \times d $)

Saving: 99.8% parameters

III. Proximal Policy Optimization (PPO)

The industry standard for alignment (used in GPT-4, Claude) involves a 3-step pipeline. After SFT, we train a Reward Model (RM) on pairwise comparisons ($ y_w \\succ y_l $) using the Bradley-Terry loss.

Then, the SFT model acts as a policy $ \\pi_\\theta $ in an RL environment. The objective function for PPO (Schulman et al., 2017) is:

$$ \text{maximize } \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi} [r(x,y) - \beta \log \frac{\pi(y|x)}{\pi_{ref}(y|x)}] $$

The KL Penalty term $ -\\beta \\log(\\pi/\\pi_{ref}) $ is critical. It forces the new policy not to diverge too far from the SFT model (the "reference"), preventing "Reward Hacking"—where the model exploits the reward model's biases to generate gibberish that scores high.

PPO Clipping PPO introduces a "clipped surrogate objective" that limits the size of the policy update step. If the probability ratio $ r_t(\\theta) $ moves outside the range $ [1-\\epsilon, 1+\\epsilon] $, the gradient is clipped. This ensures training stability, which is notoriously difficult in RL.

IV. Direct Preference Optimization (DPO)

RLHF is complex: it requires maintaining four models in memory (Actor, Critic, Reward Model, Reference Model). DPO (Rafailov et al., 2023) simplifies this by deriving a closed-form solution for the optimal policy under the RLHF objective.

DPO reparameterizes the reward function in terms of the optimal policy. The loss function becomes a simple classification task on preference pairs:

$$ \mathcal{L}_{DPO} = - \mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)} \right) \right] $$

DPO essentially increases the likelihood of the preferred response $ y_w $ relative to the reference model, while decreasing the likelihood of the rejected response $ y_l $. It is numerically stable, requires no explicit reward model training, and runs on standard SFT hardware.

Rejection Sampling (Best-of-N)

Generate $ N $ completions for a prompt, score them all with a Reward Model, and return the highest scoring one. This is an inference-time alignment technique that trades compute for quality, often used to bootstrap datasets for DPO.

V. Catastrophic Forgetting & EWC

Neural networks suffer from Catastrophic Forgetting: when fine-tuned on Task B, they overwrite the weights optimized for Task A. This is the "Stability-Plasticity Dilemma".

Elastic Weight Consolidation (EWC) Kirkpatrick et al. (2017) introduced EWC, which calculates the Fisher Information Matrix regarding Task A. It identifies which weights are critical for the old task and strictly penalizes changing them, while allowing unimportant weights to adapt freely to Task B.

$$ \mathcal{L} = \mathcal{L}_B(\theta) + \sum_i \frac{\lambda}{2} F_i (\theta_i - \theta_{A,i})^2 $$

Replay Buffers

A simpler alternative: store a small subset of Task A data and mix it into the training batch for Task B.

VI. Advanced PEFT: LoHa, LoKr & IA3

While LoRA uses matrix multiplication ($W_{new} = W + BA$), newer methods use stronger algebraic structures (LyCORIS library).

LoHa (Hadamard)

Uses the Hadamard product ($A \\odot B$) to achieve a higher rank adaptation with fewer parameters than LoRA.

LoKr (Kronecker)

Uses the Kronecker product ($A \\otimes B$). Ideal for very large weight matrices where we need extreme compression.

IA3

Instead of adding weights, IA3 multiplies activations by learned vectors. extremely parameter efficient for ICL tasks.

VII. Constitutional AI & Safety

Scaling human feedback is expensive. Constitutional AI (Bai et al., 2022) proposes RLAIF (RL from AI Feedback). Instead of humans, a "Helpful & Harmless" model critiques its own outputs based on a set of principles (a Constitution).

Critique-and-Revise: The model generates a response, critiques it for harmfulness, and revises it.
SFT Phase: Fine-tune on the revised (safe) responses.
RL Phase: Train a Reward Model on AI-generated preferences (AI choosing the safer of two options).

Adversarial Robustness Despite alignment, models remain vulnerable to "Jailbreaks" (e.g., DAN, GCG). These attacks exploit high-dimension vectors that push the model into a state where safety filters are bypassed. Robustness is an active area of research involving techniques like "Red Teaming" and "Circuit Breaking".

Primary Sources & Further Reading

Alignment Foundations

Ouyang et al. (InstructGPT, 2022). Training language models to follow instructions with human feedback.
Rafailov et al. (DPO, 2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model.
Christiano et al. (2017). Deep Reinforcement Learning from Human Preferences.

PEFT & Safety

Hu et al. (LoRA, 2021). LoRA: Low-Rank Adaptation of Large Language Models.
Bai et al. (Anthropic, 2022). Constitutional AI: Harmlessness from AI Feedback.
Dettmers et al. (QLoRA, 2023). QLoRA: Efficient Finetuning of Quantized LLMs.