GPU

A parallel processor with thousands of small cores grouped into streaming multiprocessors, paired with high-bandwidth memory (HBM3ⓘHBMsiliconStacked DRAM used as the main memory of every modern AI accelerator, with bandwidth in TB/s rather than GB/s and capacity per stack in tens of GB. Chat about this → Open full entry → or HBM3eⓘHBMsiliconStacked DRAM used as the main memory of every modern AI accelerator, with bandwidth in TB/s rather than GB/s and capacity per stack in tens of GB. Chat about this → Open full entry → on current data-center parts). The instruction set is built around dense linear algebra: matrix-multiply units (tensor cores on NVIDIA, matrix cores on AMD) execute thousands of fused multiply-adds per cycle.

The GPU is what trained every modern frontier modelⓘfrontierweightsThe current capability envelope of AI, defined by the most capable models in deployment at any given time; an evolving label rather than a fixed threshold. Chat about this → Open full entry → and serves nearly every production inferenceⓘinferenceruntimeRunning a trained model to produce outputs (tokens, images, embeddings) from inputs at serving time, as distinct from the gradient updates of training. Chat about this → Open full entry → workload. The architecture’s per-chip memory capacity (80 to 192 GB on 2026 datacenter parts) and per-chip bandwidth (3 to 8 TB/s) are the constraints that shape model architecture choices like GQA, MoEⓘmixture of expertsweightsA model architecture where each token activates only a fraction of total parameters by routing through learned expert subnetworks, decoupling capacity from compute. Chat about this → Open full entry → expert sizing, and KV-cacheⓘKV cacheruntimeThe stored key and value vectors from previously processed tokens, reused at each generation step so an autoregressive model does not recompute attention over the entire prefix. Chat about this → Open full entry → layouts.

NVIDIA dominates the datacenter GPU market in 2026 with the Hopper (H100/H200) and Blackwell (B100/B200) generations. AMD’s MI300X is the main credible competitor for inferenceⓘinferenceruntimeRunning a trained model to produce outputs (tokens, images, embeddings) from inputs at serving time, as distinct from the gradient updates of training. Chat about this → Open full entry →; Apple Silicon is the consumer local-firstⓘlocal-firstsovereignty-decentralizationAn architecture stance where inference (and increasingly memory and agent state) runs on the user's own device rather than a remote API, prioritizing privacy, latency, and offline operation. Chat about this → Open full entry → parallel; the rest of the ecosystem is hyperscaler in-house parts (TPUⓘTPUsiliconGoogle's custom AI accelerator family, used internally for training Gemini and externally via Google Cloud, designed around dense matrix multiplication with a systolic array architecture. Chat about this → Open full entry →, Trainium) or startups (Cerebras, Groq, Tenstorrent).