Model Optimization & Efficiency

I. The Memory Wall & VRAM Bottlenecks

In the current era of LLMs, the primary constraint is not compute power (FLOPS), but Memory Bandwidth. Training and serving models with billions of parameters requires moving massive amounts of data from High Bandwidth Memory (HBM) to the GPU cores.

We define the "Memory Wall" as the point where the speed of data transfer becomes the limiting factor in AI performance. To overcome this, we must employ compression techniques that reduce the model's footprint without severely degrading its reasoning capabilities.

Training is a search problem in a non-convex space. We move from vanilla Stochastic Gradient Descent (SGD) to adaptive methods that accounting for the **curvature** and **momentum** of the loss surface.

II. Adaptive Optimization: Beyond SGD

Training Large Language Models requires navigating a highly non-convex loss landscape. Vanilla **Stochastic Gradient Descent (SGD)** struggles with saddle points and ravines. Modern training relies on **Adaptive Moment Estimation (Adam)**.

AdamW (Loshchilov & Hutter, 2017) decouples weight decay from the gradient update. It maintains per-parameter learning rates based on the first moment (mean) \( m_t \) and second moment (variance) \( v_t \) of the gradients.

III. PEFT: Low-Rank Adaptation (LoRA)

Full-parameter fine-tuning of LLMs is prohibitively expensive. LoRA (Hu et al., 2021) solves this by freezing the original weights and training two small, low-rank matrices that represent the delta of the weights.

Mathematically, if the original weight matrix is \( W \), we learn its update as \( \\Delta W = A \times B \), where A and B have a very small rank \( r \). This reduces the number of trainable parameters by over 10,000x, enabling fine-tuning on a single GPU.

IV. Learning Rate & Scheduler Dynamics

The **Learning Rate (\( \\eta \))** is the most critical hyperparameter. Schedulers like 'Cosine Annealing' or 'OneCycleLR' fluctuate the rate to escape local minima and ensure convergence to flat, generalizable regions.

V. Quantization: INT8, FP4 & NF4

Affine Quantization maps the floating point range \([min, max]\) to the integer range \([0, 2^b-1]\) using a scale factor \( S \) and zero-point \( Z \).

Normal Float 4 (NF4): Introduced in QLoRA (Dettmers et al., 2023), NF4 is an information-theoretically optimal data type for weights that are normally distributed. It defines the quantization levels based on the quantiles of a standard normal distribution \( \\mathcal{N}(0, 1) \).

VI. Teacher-Student Knowledge Distillation

A compact "Student" model can achieve higher performance by learning from the **soft probabilities** of a massive "Teacher" model rather than the raw hard labels.

The "Temperature" parameter \( \\tau \) softens the probability distribution, forcing the student to learn the structual relationships between incorrect classes (e.g., "Dog" is more similar to "Cat" than "Car"), known as Dark Knowledge.