Hardware

Will it fit, and how fast?

Two questions decide whether you can run an open model on a given box. Memory capacity decides what fits. Memory bandwidthⓘmemory bandwidthsiliconThe rate (GB/s or TB/s) at which an accelerator reads its memory. It sets the ceiling on decode tokens/sec, since each token streams the active weights once. Chat about this → Open full entry → decides how fast it runs. They are not the same number, and a box that holds a model can still be too slow to serve it.

Pick a model, a quantizationⓘquantizationweightsStoring or computing model weights in lower-precision number formats (FP8, INT8, INT4) to reduce memory and bandwidth, accepting small quality loss. Chat about this → Open full entry →, and a context length, then compare specific boxes, find every box a model fits on, or pick a box and list everything it can run. The explorer shows whether the model fits, a theoretical rooflineⓘrooflineruntimeA performance model that bounds throughput by either compute or memory bandwidth, whichever is the limiting resource for an operation's arithmetic intensity. Chat about this → Open full entry → tokens/sec ceiling with the formula, a realistic range, a measured anchor where one exists, and a plain-language reason that traces back to bandwidth, capacity, and KV cacheⓘKV cacheruntimeThe stored key and value vectors from previously processed tokens, reused at each generation step so an autoregressive model does not recompute attention over the entire prefix. Chat about this → Open full entry →. The method comes straight from the self-host course.

Theoretical numbers are labeled as estimates and show their inputs; measured anchors carry a date and source. Every hardware spec flows through the same verification gate as /models: an unverified value renders as —, never a guess.