LLM decision guide
Quantization vs Distillation
Quantization uses lower-precision numbers; distillation trains a smaller student to reproduce useful teacher behavior.
Choose Quantization when
- The architecture must stay the same
- Fast deployment savings are needed
Choose Distillation when
- A smaller architecture is required
- Training data and evaluation capacity are available
Decision table
| Dimension | Quantization | Distillation |
|---|---|---|
| Mechanism | Lower numeric precision | New student training |
| Primary risk | Numerical quality loss | Behavior coverage loss |
| Evidence | Parity by task and hardware | Teacher-student and independent evals |
Example
A realistic choice
Quantize an existing model for memory savings; distill when the target device needs a much smaller architecture.
Hands-on
Test the decision
Design a quality-latency-memory scorecard that can reject either compression path.