quantization

The set of techniques for representing model weights and activations in fewer bits than the training precision. A 70B parameter model in FP16 is 140 GB; in INT4 it is 35 GB; in many low-bit formats it runs on a single consumer GPUⓘGPUsiliconA massively parallel processor originally designed for graphics, repurposed since the 2010s as the dominant compute substrate for both training and inference of large neural networks. Chat about this → Open full entry →. The cost is some quality loss, which modern post-trainingⓘpost-trainingtrainingEverything that happens after pretraining ends: supervised fine-tuning, preference optimization, red-teaming, distillation, and safety work that turns a base into a shippable assistant. Chat about this → Open full entry → quantization (GPTQ, AWQ, GGUFⓘGGUFweightsA binary container format for quantized model weights used by llama.cpp and its ecosystem; the dominant on-device LLM file format since 2023. Chat about this → Open full entry →/llama.cppⓘllama.cppruntimeGeorgi Gerganov's C++ inference engine optimized for CPUs and consumer GPUs, the on-device standard and the engine behind Ollama, LM Studio, and most local-first AI products. Chat about this → Open full entry →’s k-quant formats) has reduced to a fraction of a percent on most benchmarks for 4-bit and barely measurable for 8-bit.

The format choice depends on deployment. FP8ⓘFP8siliconAn 8-bit floating-point format used for AI inference and increasingly for training, halving memory and bandwidth versus FP16 with minimal quality loss on most workloads. Chat about this → Open full entry → is the data-center default on Hopper/Blackwell and MI300X, supported by mature kernels and minimal quality loss. INT4 (GPTQ, AWQ) is common for inferenceⓘinferenceruntimeRunning a trained model to produce outputs (tokens, images, embeddings) from inputs at serving time, as distinct from the gradient updates of training. Chat about this → Open full entry → on older hardware without FP8ⓘFP8siliconAn 8-bit floating-point format used for AI inference and increasingly for training, halving memory and bandwidth versus FP16 with minimal quality loss on most workloads. Chat about this → Open full entry → support. GGUFⓘGGUFweightsA binary container format for quantized model weights used by llama.cpp and its ecosystem; the dominant on-device LLM file format since 2023. Chat about this → Open full entry → k-quants (Q4_K_M, Q5_K_M, etc.) target on-deviceⓘon-devicesovereignty-decentralizationRunning model inference on the user's local hardware (phone, laptop, embedded device), enabled by smaller models, FP8 quantization, and runtimes like llama.cpp and MLX. Chat about this → Open full entry → deployment via llamaⓘLlamaweightsMeta's open-weight model family, the most widely deployed open release through 2024 to 2026, released under the source-available Community License with an MAU cap and acceptable-use clause. Chat about this → Open full entry →.cpp and balance speed and quality.

Training in low precision is harder than inferenceⓘinferenceruntimeRunning a trained model to produce outputs (tokens, images, embeddings) from inputs at serving time, as distinct from the gradient updates of training. Chat about this → Open full entry →. FP8 mixed- precision training has become production-ready on Hopper; 4-bit training and 2-bit training remain research. Inference quantization is mature; training quantization is still an active frontierⓘfrontierweightsThe current capability envelope of AI, defined by the most capable models in deployment at any given time; an evolving label rather than a fixed threshold. Chat about this → Open full entry → in 2026.