Skip to content

Chapter 2.2 — Quantization and model compression: from 4-bit to real deployment

🎯 Objective

Cover quantization at a level sufficient to decide between lab/local and enterprise production. The focus is not on training quantization, but on understanding what changes in memory, throughput, quality, kernel and hardware when a model leaves FP16 and drops to INT8/INT4.

🧠 Core concept

Quantization trades numerical precision for reduced size and operation cost. In LLMs, the three dominant operations are matmul, attention and normalization. Quantizing reduces the size of the weights and, eventually, the activations; the real gain depends on the kernel supporting the operation in the new format.

🧮 Formats and where they appear

Format Bits Typical use Required hardware
FP32 32 Standard training, baselines, comparison General CPU/GPU
TF32 19 (accumulation) Training with Tensor Cores (NVIDIA Ampere+) A100, H100
FP16 16 Mixed-precision training, GPU inference V100+
BF16 16 Stable training (range ~ FP32), inference A100/H100, TPU
FP8 (E4M3/E5M2) 8 Modern inference/training on new GPUs H100/H200, MI300, Trainium
INT8 8 Quantized inference (W8A8 or W8A16) Broad GPU/CPU/edge
INT4 / NF4 4 Quantized inference and QLoRA Recent GPU, llama.cpp, MLC, GGUF
INT3 / INT2 3, 2 Research, "aggressive" GGUF formats Variable; loss common

🧠 Quantized inference vs fine-tuning

These are two different problems:

  • Quantization for inference — reducing cost/memory when serving. The GPTQ, AWQ, SmoothQuant families, GGUF/AWQ/EXL2 formats, kernels from TensorRT-LLM, vLLM, llama.cpp.
  • Quantization during fine-tuning — keeping the base model in low precision and training adapters (LoRA) at higher precision. The canonical example is QLoRA with NF4 + double quantization + paged optimizers.

The confusion between the two usually shows up in "is QLoRA better than GPTQ?" discussions — they are techniques for different problems (training vs serving).

🧪 Main methods

Post-Training Quantization (PTQ)

  • GPTQ — layer-by-layer quantization using a second-order approximation of the reconstruction error; typically INT4 or INT3 on weights only. A good quality/cost balance for serving. (Frantar et al., 2022).
  • AWQ (Activation-aware Weight Quantization) — preserves "salient" channels (high activation magnitude) and quantizes the rest aggressively. It tends to keep quality better than round-to-nearest and is friendly to optimized kernels. (Lin et al., 2023).
  • SmoothQuant — redistributes quantization difficulty between weights and activations via rescaling; enables W8A8.
  • GGUF (llama.cpp) — a file format + quantization variants (Q4_K_M, Q5_K_M, Q8_0, IQ-quants, etc.) popular for local execution on mixed CPU/GPU. Good for lab and local dev.

Quantization-Aware Training (QAT)

Trains (or continues training) already accounting for quantization noise. Higher engineering cost, typically better quality than PTQ in aggressive formats.

bitsandbytes

A widely used Python library for loading models in 8-bit (LLM.int8) and 4-bit (NF4) with Hugging Face Transformers. Focused on research, prototyping and fine-tuning. It is not the typical path for low-latency serving in large-scale production — for that, prefer kernels from vLLM, TensorRT-LLM, or native implementations of the serving framework.

QLoRA

Combines (a) a base model frozen in NF4 (4-bit "NormalFloat", a distribution informationally optimized for near-normal weights); (b) double quantization of the quantization constants themselves; (c) paged optimizers to handle memory spikes; (d) LoRA adapters in BF16. Result: fine-tuning large models on single GPUs with quality close to full FT in many scenarios.

⚖️ Trade-offs

Axis Reality
VRAM Almost always drops with quantization. INT4 reduces weights by ~4x vs FP16.
Throughput Not always increases. It depends on having an efficient quantized kernel for the hardware. On modern hardware with already well-optimized FP16, INT4 without an adequate kernel can be slower.
Latency (TTFT) May improve due to less weight traffic over memory bandwidth, the main bottleneck of decode.
Quality Large models (>30B) tolerate INT4 reasonably. Small models (≤7B) suffer more, especially in multi-step reasoning and tool calling.
Compatibility FP8/INT4 require specific hardware and kernels. Moving between stacks (Transformers, vLLM, TGI, TensorRT-LLM, llama.cpp) frequently breaks.
Determinism Quantization increases numerical variation; reproducibility becomes more fragile.
Tool calling / structured output Aggressively quantized models may degrade more in following schemas and step-by-step reasoning than in free-form responses.

🏗️ How this shows up in production

  • Lab / local dev — INT4 via GGUF + llama.cpp or via bitsandbytes works, with models running on laptops. Accepts loss; the goal is experimentation.
  • Enterprise self-hosting — usually FP16/BF16 or FP8 on recent GPUs; AWQ/GPTQ INT4 when the density gain offsets the measurable loss. Always with a comparative eval set across versions.
  • API providers — abstracts the reader. The client only sees model + price; quantization is the provider's responsibility.
  • Edge / on-device — INT4/INT3, MLC, llama.cpp, mlx, GGUF formats; quality is explicitly negotiated.

🚨 Failure modes

  • Adopting aggressive quantization because of "VRAM savings" without measuring real throughput on the target hardware.
  • Evaluating quantization only with perplexity on general benchmarks; ignoring task-specific degradation in tool calling, JSON output and RAG.
  • Switching frameworks (e.g., bitsandbytes -> vLLM) without re-evaluating quality.
  • Quantizing a small model that was already at the limit.
  • Mixing QLoRA adapters trained on one NF4 version with a base quantized in another library.

🛡️ Controls and mitigations

  • Keep a fixed eval set (golden + adversarial + tool calling + structured output) to compare FP16 vs INT8 vs INT4.
  • Measure, on the target hardware: TTFT, TPOT, token/s throughput, p95, cost per success.
  • Version model_id + quant_scheme + framework_version in tracing.
  • Pin kernel/library versions via lock files.
  • Adopt canary with shadow inference before promoting a quantized model.

📈 Metrics

  • Weight memory: params × bits_per_param / 8.
  • Perplexity (when applicable) on a domain corpus.
  • Task success rate per category.
  • Hallucination rate vs FP16 baseline.
  • Tool call accuracy and schema validity rate (especially for 4-bit).
  • Tokens/s (decode) and TTFT (prefill).
  • Cost per success, not just cost per call.

🧪 Distillation (short review)

A "student" model learns to imitate a "teacher" (labels, logits, behavior on real traces). Unlike quantization, distillation changes the architecture/size, not just the numerical precision.

Useful patterns:

  • Distill a large model into a smaller one for a specific task.
  • Distill behavior (format, style) and leave factual knowledge in RAG.
  • Combine distillation + quantization to reduce aggregate cost.

🧪 Fine-tuning (short review)

  • Full fine-tuning — updates all weights. Expensive, requires a significant dataset, an eval pipeline, governance.
  • LoRA — low-rank adapters over selected layers. Keeps the base frozen.
  • QLoRA — LoRA over a 4-bit base (NF4); enables fine-tuning large models on modest hardware.
  • Adapter / Prefix / IA³ / DoRA — variations with different parameter and quality trade-offs.
  • Instruction tuning / SFT — supervised training on instruction-response pairs.
  • Preference tuning (DPO, KTO, ORPO, etc.) — aligning behavior to preferences; more delicate and out of scope for this book.

⚠️ When NOT to use fine-tuning

  • To add mutable factual knowledge — prefer RAG.
  • To impose a business rule — prefer policy-as-code.
  • For a single format requirement — prefer structured outputs.
  • To "teach tool use" without a structured dataset and eval — almost always not worth it; adjust the description, examples and harness.

📌 Checklist

  • [ ] Is a comparative eval set across formats (FP16/INT8/INT4) ready?
  • [ ] Were tool calling and structured output metrics measured, not just perplexity?
  • [ ] Was real throughput measured on the target hardware, not just theoretical VRAM?
  • [ ] Does the serving framework support the format stably?
  • [ ] Is there a rollback plan to the FP16 version?
  • [ ] Are fine-tuning adapters versioned together with the base?
  • EX-LLM-07 — FP16 vs INT8 vs INT4 (GPTQ/AWQ) comparison with a tool calling and JSON output eval set.
  • EX-LLM-08 — minimal QLoRA with Hugging Face + bitsandbytes on a small model for style, keeping knowledge in RAG.

📚 References