Glossary
FP16
A 16-bit floating-point format used as the default precision for deep learning training and inference, halving memory versus FP32 with small quality cost on most workloads.
A 16-bit floating-point number with 1 sign bit, 5 exponent bits, and 10 mantissa bits. The format was standardized in IEEE 754-2008 as binary16 and adopted broadly in deep learning starting with the NVIDIA Pascal generation in 2016. FP16 occupies half the bytes of FP32 (4 bytes per parameter) and twice the bytes of FP8siliconAn 8-bit floating-point format used for AI inference and increasingly for training, halving memory and bandwidth versus FP16 with minimal quality loss on most workloads. Open full entry (1 byte per parameter).
For most of the past decade, FP16 was the default precision for both LLM weights and activations. The Mixed Precision Training paper (Micikevicius et al., 2017) showed that combining FP16 storage with FP32 master weights and a loss scaler could match FP32 final quality while doubling training throughput. By 2023, FP16 had become the shipping precision for most open-weights models: Llama 2, Mistral, Qwen, and the earlier DeepSeek releases all used it as the base format.
By 2026 the precision frontier has shifted lower. FP8 and FP4 are preferred for inference on hardware that supports them, and many training runs use mixed FP8 with BF16siliconA 16-bit floating-point format with FP32's exponent range and only 7 mantissa bits. Designed for neural-network training; standard across 2026 accelerators alongside FP16. Open full entry . FP16 remains the fallback when the target accelerator lacks FP8 kernels, the reference precision against which lower-bit formats are quality-tested, and a common upload format for community fine-tunes on Hugging Face.