Quantization & LoRA

Two different ways to save memory

LoRA freezes a base weight matrix and trains a low-rank update. Quantization stores an approximation of weights using fewer bits. They solve different problems: adapter capacity versus representation cost. QLoRA combines a frozen quantized base with trainable adapters; four-bit storage does not mean every computation uses four bits.

Start with matrix shapes and linear layers. In this lesson, use a tiny matrix to understand the forward pass before considering a GPU-sized model.

Initializing Interactive Playground...

Follow every matrix

Use a column input x=[2,1], W=[[1,0],[0,1]], A=[[1,−1]], B=[[0.5],[−0.5]], rank=1 and alpha=2. A has shape 1×2; B has shape 2×1. First Ax=1. Then BAx=[0.5,−0.5]. Scale by alpha/r=2 and add Wx: y=[3,0].

Merging gives Wmerged=[[2,−1],[−1,2]], whose product with x is also [3,0]. If you keep adding the adapter after merging, you get [4,−1]: the update was counted twice. The lab checks both paths and that the original W stays unchanged.

This tiny rank-one adapter has four parameters, as many as W itself. Savings appear when r(d_in+d_out) is much smaller than d_in×d_out. For a 4096×4096 projection and rank 8, the adapter has 65,536 entries—not a 99.6% reduction in total training memory.

Measure the approximation

For signed four-bit symmetric quantization, this lab chooses codes −7…7. With max absolute weight 1, scale=1/7. A weight 0.3 rounds to code 2 and returns as 2/7≈0.285714. Adding an outlier 10 changes the shared scale to 10/7, rounding 0.3 to zero. Per-block scaling can reduce this damage but needs more metadata.

Compare base, adapter, quantized base and quantized-plus-adapter outputs on the same held-out input. Measure output error as well as weight error. This is uniform quantization, not NF4, and Python floats do not demonstrate actual packed storage or GPU speed.

Experiment notes

Record your experiment

Keep four-bit precision and turn on the outlier. Predict the reconstructed value of 0.3. Then change adapter rank and explain why the quantization error does not change.

PredictionWhat changed, and why?

Implementation

Make merging an equivalence test

A layer appears correct on a square input but double-counts the adapter after merging.

Repair the browser forward pass, including the rectangular case.
Implement forward, merge and quantize functions in starter.py; run all lab tests.
Compare the four output paths and quantify rounding on held-out inputs.

Check: Show merged/unmerged parity, an unchanged base, and an honest output-error report—not an inferred GPU speedup.

Failing starter lab

Implement the adapter forward pass

These CPU labs isolate mechanisms. Passing the drill is not completion of the project, LLM training, or independent evidence of mastery.

Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.

Dataset: Authored rectangular matrices v1

Console

Run the code to inspect its output. Use Check solution only when you want to run the exercise checks.

Code runs only in a time-limited browser worker. It is not sent to the IntuitiveAI server. Do not paste secrets.

Expected result

ALL TESTS PASSED

Prerequisites: Matrix multiplication and shapes · Linear layers · Mean squared error

Build the complete CPU experiment

Download all six files into one folder. Python 3.10+; standard library only. No API key, GPU, account, model download or paid service. The browser exercise above is one component of this project.

Run the reference tests, then the reference experiment. Inspect the configuration, fixture hash, numerical results and limitations.
Implement the functions in starter.py, then run the learner tests and experiment. The unfinished starter intentionally fails; it never falls back to the answer.
Change one setting using the experiments below. Save a report with a new filename and add your prediction, result, explanation and limitation.

python3 test_labs.py --reference
python3 run_lab.py lora --implementation reference
python3 test_labs.py --topic lora
python3 run_lab.py lora --output my-lora-report.json

Experiment plan & debugging checks

Set B to zero, then restore the worked-example B.

Predict: Zero B preserves the base output; nonzero B gives [3,0].

Inspect: Compare merged and unmerged outputs to 1e−10; inspect original W.

Run --bits 4 and --bits 8 on identical fixtures.

Predict: Finer resolution changes rounding error, not adapter parameter count.

Inspect: Weight MSE and output MSE for base, adapted, quantized, and quantized+adapted paths.

Replace a weight by an outlier, without changing the other weights.

Predict: A shared scale can erase small weights.

Inspect: Record the scale and each reconstructed value; propose a per-block comparison.

Next: port these invariants to a real model only after you can reproduce the small case. Freeze the checkpoint, tokenizer, split, objective, masks and evaluation before comparing runs. GPU training and independent learner validation remain separate work.

Where this fits in the GPT build path →

Chapter 1 of 6 · Core

Low-Rank Adaptation (LoRA) Rank Decomposition Theory

Full fine-tuning of 70B+ LLMs requires updating billions of weights and storing 16-bit parameters and gradients, an FP32 master copy, and two FP32 AdamW moment states in one common mixed-precision setup (, total ). Low-Rank Adaptation (LoRA) (Hu et al., 2021) freezes pre-trained weight matrix and injects trainable low-rank rank matrices.

Low-Rank Decomposition Equation:

where is initialized randomly (the code below uses Kaiming uniform), is initialized to exact zeros (), is the rank hyperparameter, and is a constant scaling hyperparameter.

Initial Forward Pass Guarantee: Because at step 0, . The model starts training with exact pre-trained outputs!

Parameter Savings Ratio: For , full tuning requires parameters. A LoRA adapter with rank requires only parameters—a 99.6% reduction in trainable entries for this projection, not total VRAM. Activations, frozen weights and other runtime state remain.

Key equations

PyTorch adapter around an existing linear layerpython

import copy
import math
import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, pretrained: nn.Linear, rank=8, alpha=16.0):
        super().__init__()
        if rank < 1:
            raise ValueError("rank must be positive")
        self.linear = copy.deepcopy(pretrained)
        self.linear.requires_grad_(False)
        self.scaling = alpha / rank
        self.lora_A = nn.Parameter(pretrained.weight.new_empty(rank, pretrained.in_features))
        self.lora_B = nn.Parameter(pretrained.weight.new_zeros(pretrained.out_features, rank))
        nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5))

    def forward(self, x):
        return self.linear(x) + (x @ self.lora_A.T @ self.lora_B.T) * self.scaling

    @torch.no_grad()
    def merged_copy(self):
        # Return a plain layer: its forward must NOT add the adapter again.
        merged = copy.deepcopy(self.linear)
        merged.weight.add_((self.lora_B @ self.lora_A) * self.scaling)
        return merged

# Compare layer(x) with layer.merged_copy()(x) using dtype-aware tolerances.
# This illustration assumes a floating-point nn.Linear, not a quantized kernel.

NextDeclare the quantization convention

References

Primary sources and free technical reading for Quantization & LoRA.

LoRA

Introduces low-rank adaptation for parameter-efficient fine-tuning.

PaperFree

QLoRA

Combines quantized base models with low-rank adapters for memory-efficient fine-tuning.

PaperFree

Back to Topic Library

Feedback & sharing

Quantization & LoRA

Two different ways to save memory

Follow every matrix

Measure the approximation

Record your experiment

Implementation

Implement the adapter forward pass

Build the complete CPU experiment

Set B to zero, then restore the worked-example B.

Run --bits 4 and --bits 8 on identical fixtures.

Replace a weight by an outlier, without changing the other weights.

Knowledge check

8 questions

Prove it on a new case

Low-Rank Adaptation (LoRA) Rank Decomposition Theory

Further reading

LoRA

QLoRA

References

LoRA

QLoRA

Helpful?