Glossary
quantization-aware training
Training or fine-tuning a model with simulated quantization in the loop, so its weights stay accurate at low precision like int4 or FP8, unlike post-hoc post-training quantization.
Quantization-aware training (QAT) folds simulated quantization into the forward pass while the model is still training or being fine-tuned. The weights are stored in full precision, but a “fake-quant” step rounds them to the target low-precision grid (int4, FP8, 2-bit) before each matmul, so the gradients learn weights that survive that rounding. The contrast is with post-trainingtrainingEverything that happens after pretraining ends: supervised fine-tuning, preference optimization, red-teaming, distillation, and safety work that turns a base into a shippable assistant. Open full entry quantizationweightsStoring or computing model weights in lower-precision number formats (FP8, INT8, INT4) to reduce memory and bandwidth, accepting small quality loss. Open full entry (GPTQ, AWQ, GGUFweightsA binary container format for quantized model weights used by llama.cpp and its ecosystem; the dominant on-device LLM file format since 2023. Open full entry ), which quantizes a finished model with no further training. QAT costs a training run but recovers most of the quality that post-hoc methods leave on the table.
The signal that matters is who ships the quantized weights. When a lab runs QAT and releases the result, the official checkpoint is the good low-precision one, not a community approximation. Google released int4 QAT checkpoints for Gemma 3 that drop the 27B model from 54 GB to about 14 GB so it fits a single 24 GB GPU; Meta shipped 4-bit QAT-plus-LoRA builds of Llama 3.2 1B and 3B for on-devicesovereignty-decentralizationRunning model inference on the user's local hardware (phone, laptop, embedded device), enabled by smaller models, FP8 quantization, and runtimes like llama.cpp and MLX. Open full entry use; Moonshot’s Kimi K2 Thinking applies QAT to its mixture of expertsweightsA model architecture where each token activates only a fraction of total parameters by routing through learned expert subnetworks, decoupling capacity from compute. Open full entry weights for native int4 inferenceruntimeRunning a trained model to produce outputs (tokens, images, embeddings) from inputs at serving time, as distinct from the gradient updates of training. Open full entry ; Apple’s 2025 on-device model uses QAT at two bits per weight; and Mistral NeMo was trained with quantization awareness for FP8.
This is distinct from native low-precision pretraining, where the training math itself runs in a low-precision format (DeepSeek-V3’s FP8 training is the well-known case). QAT keeps the training in higher precision and only simulates the eventual inference format. Both reduce deployment cost, but they are different techniques and should not be labelled the same.
For open and local AI the practical upshot is reach: QAT is often what decides whether a capable model runs on a laptop or a single consumer card at close to full quality, rather than only in a data center. It sits between the weights a lab publishes and the runtime that serves them, and in 2026 it moved from a research technique to something the major labs ship by default for their small and edge models.