continuous batching

The serving pattern that made high-throughputⓘthroughputcomputeThe rate at which a model produces output tokens, usually quoted as tokens-per-second per GPU or aggregate, the headline number for serving-cost economics. Chat about this → Open full entry → LLM inferenceⓘinferenceruntimeRunning a trained model to produce outputs (tokens, images, embeddings) from inputs at serving time, as distinct from the gradient updates of training. Chat about this → Open full entry → economically viable. Naive batchingⓘbatchingcomputeGrouping multiple requests or training examples into a single forward or backward pass, the lever that turns GPU compute density into throughput. Chat about this → Open full entry → waits for every request in a batch to finish before starting the next, which is wasteful: a 500-token completion holds the batch while a 20-token completion sits idle. Continuous batchingⓘbatchingcomputeGrouping multiple requests or training examples into a single forward or backward pass, the lever that turns GPU compute density into throughput. Chat about this → Open full entry → schedules at the token level, swapping new requests in as others finish, so the GPUⓘGPUsiliconA massively parallel processor originally designed for graphics, repurposed since the 2010s as the dominant compute substrate for both training and inference of large neural networks. Chat about this → Open full entry → is never idle waiting on the longest generation.

Combined with PagedAttention (so per-request KV cache can grow independently) and prefill chunkingⓘchunkingretrieval-memorySplitting source documents into smaller passages for embedding and retrieval, where the chunk size and overlap directly affect retrieval quality and context efficiency. Chat about this → Open full entry → (so a new request’s prefill does not stall the decode loop), continuous batching pushes a server’s effective throughputⓘthroughputcomputeThe rate at which a model produces output tokens, usually quoted as tokens-per-second per GPU or aggregate, the headline number for serving-cost economics. Chat about this → Open full entry → per GPUⓘGPUsiliconA massively parallel processor originally designed for graphics, repurposed since the 2010s as the dominant compute substrate for both training and inference of large neural networks. Chat about this → Open full entry → 5 to 20 times higher than naive batching at similar latencyⓘlatencycomputeThe time from request submission to response completion, broken down for LLMs into time-to-first-token and time-per-output-token, the user-facing speed metric. Chat about this → Open full entry → targets.

It is the default scheduling mode in vLLMⓘvLLMruntimeAn open-source inference engine introduced by UC Berkeley in 2023, built around PagedAttention to manage KV cache memory and serve tokens efficiently under load. Chat about this → Open full entry →, SGLangⓘSGLangruntimeAn open inference engine from the LMSYS team featuring RadixAttention for prefix sharing and a structured-generation frontend, particularly strong on agent and tool-calling workloads. Chat about this → Open full entry →, TensorRT-LLMⓘTensorRT-LLMruntimeNVIDIA's closed-source inference engine for NVIDIA GPUs, the fastest runtime on Hopper and Blackwell but tied to NVIDIA's proprietary kernel stack and CUDA. Chat about this → Open full entry →, TGIⓘTGIruntimeHugging Face's production inference server, an early peer of vLLM that ceded throughput leadership in 2024 and now sits in maintenance mode behind vLLM and SGLang. Chat about this → Open full entry →, and llama.cppⓘllama.cppruntimeGeorgi Gerganov's C++ inference engine optimized for CPUs and consumer GPUs, the on-device standard and the engine behind Ollama, LM Studio, and most local-first AI products. Chat about this → Open full entry →’s server mode.