Back to Curriculum
Advanced•MLOps
Quantization & LoRA
Low-Rank Adaptation (ΔW = B·A), rank selection, FP8/INT4 uniform quantization, and scale-zero point math.
Interactive Playground
Initializing Interactive Playground...
Research-Level Deep Dive & Equations
Full fine-tuning of 70B+ LLMs requires updating billions of weights and storing 16-bit parameters, 16-bit gradients, and 32-bit AdamW optimizer states (, total ). **Low-Rank Adaptation (LoRA)** (Hu et al., 2021) freezes pre-trained weight matrix and injects trainable low-rank rank matrices.
•Low-Rank Decomposition Equation:
where is initialized with Gaussian distribution , is initialized to exact zeros (), is the rank hyperparameter, and is a constant scaling hyperparameter.
•Initial Forward Pass Guarantee: Because at step 0, . The model starts training with exact pre-trained outputs!
•Parameter Savings Ratio: For , full tuning requires parameters. A LoRA adapter with rank requires only parameters—a **99.6% parameter reduction**!
Key Equations
PyTorch Custom LoRALinear Layer & Zero-Latency Weight Mergingpython
import torch
import torch.nn as nn
import math
class LoRALinear(nn.Module):
def __init__(self, in_features: int, out_features: int, rank: int = 8, alpha: float = 16.0):
super().__init__()
# 1. Freeze base pretrained linear layer
self.linear = nn.Linear(in_features, out_features, bias=False)
self.linear.weight.requires_grad = False
self.rank = rank
self.scaling = alpha / rank
# 2. Low-rank matrices A and B
self.lora_A = nn.Parameter(torch.zeros(rank, in_features))
self.lora_B = nn.Parameter(torch.zeros(out_features, rank))
# 3. Initialization: A ~ Gaussian, B = 0
nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5))
nn.init.zeros_(self.lora_B)
def forward(self, x: torch.Tensor) -> torch.Tensor:
base_out = self.linear(x)
lora_out = (x @ self.lora_A.T @ self.lora_B.T) * self.scaling
return base_out + lora_out
def merge_weights(self):
"""Merges B*A into base weight matrix W_0 for zero-latency inference."""
self.linear.weight.data += (self.lora_B @ self.lora_A) * self.scaling
print("LoRA weights merged successfully into base linear layer!")Test Your Knowledge
Check whether you have mastered this concept with a quick quiz.
Was this lesson helpful?
Your feedback helps us continuously improve the curriculum and interactive visualizations.