Quantization & LoRA

Low-Rank Adaptation (ΔW = B·A), rank selection, FP8/INT4 uniform quantization, and scale-zero point math.

For a small example, map four floating values to a tiny integer range. Choose scale and zero point, quantize and dequantize, then calculate the introduced error.

In practice

A model needs domain style on limited hardware. Train LoRA adapters on curated data, compare with prompting, evaluate regressions, and test quantized inference quality and memory.

Initializing Interactive Playground...

Experiment notes

Record your experiment

Predict how one control changes the Quantization & LoRA result.

PredictionWhat changed, and why?

Implementation

Repair a failing Quantization & LoRA starter

The starter deliberately follows a tempting but flawed assumption: Assuming a bit width predicts real speed

Run the starter once and inspect the first wrong value, failed check, or incomplete branch.
Repair the smallest relevant part of the code. Do not rewrite the whole solution.
Check the solution, then explain why this correction works: Benchmark supported kernels and memory traffic on target hardware.

Check: The browser checks pass and you can name the failure mechanism before opening the explanation.

Failing starter lab

Apply a LoRA update

Compute W + scale * B @ A for a low-rank adaptation.

Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.

Console

Run the code to inspect its output. Use Check solution only when you want to run the exercise checks.

Code runs only in a time-limited browser worker. It is not sent to the IntuitiveAI server. Do not paste secrets.

Expected result

[[1.5 0. ] [-0.25 1. ]]
ALL TESTS PASSED

Chapter 1 of 6 · Core

Low-Rank Adaptation (LoRA) Rank Decomposition Theory

Full fine-tuning of 70B+ LLMs requires updating billions of weights and storing 16-bit parameters, 16-bit gradients, and 32-bit AdamW optimizer states (, total ). Low-Rank Adaptation (LoRA) (Hu et al., 2021) freezes pre-trained weight matrix and injects trainable low-rank rank matrices.

Low-Rank Decomposition Equation:

where is initialized with Gaussian distribution , is initialized to exact zeros (), is the rank hyperparameter, and is a constant scaling hyperparameter.

Initial Forward Pass Guarantee: Because at step 0, . The model starts training with exact pre-trained outputs!

Parameter Savings Ratio: For , full tuning requires parameters. A LoRA adapter with rank requires only parameters—a 99.6% parameter reduction!

Key equations

PyTorch Custom LoRALinear Layer & Zero-Latency Weight Mergingpython

import torch
import torch.nn as nn
import math

class LoRALinear(nn.Module):
    def __init__(self, in_features: int, out_features: int, rank: int = 8, alpha: float = 16.0):
        super().__init__()
        # 1. Freeze base pretrained linear layer
        self.linear = nn.Linear(in_features, out_features, bias=False)
        self.linear.weight.requires_grad = False
        
        self.rank = rank
        self.scaling = alpha / rank
        
        # 2. Low-rank matrices A and B
        self.lora_A = nn.Parameter(torch.zeros(rank, in_features))
        self.lora_B = nn.Parameter(torch.zeros(out_features, rank))
        
        # 3. Initialization: A ~ Gaussian, B = 0
        nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5))
        nn.init.zeros_(self.lora_B)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        base_out = self.linear(x)
        lora_out = (x @ self.lora_A.T @ self.lora_B.T) * self.scaling
        return base_out + lora_out

    def merge_weights(self):
        """Merges B*A into base weight matrix W_0 for zero-latency inference."""
        self.linear.weight.data += (self.lora_B @ self.lora_A) * self.scaling
        print("LoRA weights merged successfully into base linear layer!")

NextUniform Quantization Mechanics (INT8, INT4 & Scale-Zero Factor Math)

References

Primary sources and free technical reading for Quantization & LoRA.

LoRA

Introduces low-rank adaptation for parameter-efficient fine-tuning.

PaperFree

QLoRA

Combines quantized base models with low-rank adapters for memory-efficient fine-tuning.

PaperFree

Back to Topic Library

Feedback & sharing

Quantization & LoRA