Quantization Research¶

What you'll learn¶

• What quantization is and why it helps
• Quantize/dequantize math (tested)
• Post-training vs quantization-aware
• The accuracy tradeoff
• Tooling

Quantization represents a model's numbers with fewer bits — typically converting 32-bit floats to 8-bit integers. This shrinks the model ~4×, speeds up inference, and cuts power use, at a small accuracy cost. It's essential for deploying big models to phones, edge devices, and cost-sensitive servers. The quantize/dequantize math here is run-verified.

Why quantization¶

A core question explored in Quantization: Why quantization.

A trained neural network is millions/billions of float32 weights. That's a lot of memory and compute:

• Size — float32 → int8 is a 4× reduction. A 400 MB model becomes ~100 MB.
• Speed — integer math is faster, and less data movement means less time (memory bandwidth often dominates).
• Power — critical for battery devices and data-center cost.

The catch: fewer bits means less precision, so you trade a little accuracy for big efficiency gains.

The quantize/dequantize math (tested)¶

The quantize/dequantize math — a key concept in Quantization.

Quantization maps a float range onto a small integer range using a scale and a zero-point. Runnable:

def quantize(values, bits=8):
    lo, hi = min(values), max(values)
    levels = 2 ** bits - 1                       # 255 for int8
    scale = (hi - lo) / levels
    q = [round((v - lo) / scale) for v in values]
    return q, scale, lo

def dequantize(q, scale, lo):
    return [x * scale + lo for x in q]

original = [0.0, 0.25, 0.5, 0.75, 1.0]
q, scale, lo = quantize(original)
recovered = dequantize(q, scale, lo)

print("int8 values:", q)
print("recovered:  ", [round(x, 4) for x in recovered])

Output:

int8 values: [0, 64, 128, 191, 255]
recovered:   [0.0, 0.251, 0.502, 0.749, 1.0]

The floats are mapped to integers 0-255 (fitting in a byte), and dequantizing recovers values very close to the originals — the tiny differences (0.25 → 0.251) are quantization error. That error is the price of using 8 bits instead of 32; for most neural networks it barely affects predictions, because networks are robust to small perturbations.

Post-training vs quantization-aware¶

Quantize after training (fast, some accuracy loss) vs during training (best accuracy).

Two ways to quantize a model:

• Post-Training Quantization (PTQ) — quantize an already-trained model. Fast and easy (no retraining), but can lose more accuracy. Often you feed a small "calibration" dataset to pick good scales.
• Quantization-Aware Training (QAT) — simulate quantization during training so the model learns to be robust to it. More work (retraining), but recovers most of the lost accuracy.

Rule of thumb: try PTQ first (cheap); if accuracy drops too much, use QAT.

The accuracy tradeoff¶

The accuracy tradeoff — a key concept in Quantization.

Precision options form a spectrum: float32 (full) → float16/bfloat16 (half, common on GPUs) → int8 (common for inference) → int4 and below (aggressive, for huge LLMs, more accuracy risk).

Tooling¶

PyTorch/ONNX quantization utilities that do the heavy lifting.

Real quantization uses framework tools (documented; not installed here):

# PyTorch post-training dynamic quantization (documented API)
import torch
quantized = torch.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8
)

Practice exercises¶

1. Extend quantize to use a symmetric scheme (zero-point at 0) and compare error.
2. Quantize to 4 bits (bits=4) and measure how much the max error grows versus 8 bits.
3. Explain why neural networks tolerate quantization error better than, say, a checksum would.
4. Describe when you'd choose QAT over PTQ.
5. Explain how 4-bit quantization lets a large LLM fit on a smaller GPU, with rough size math.