The Open-Source AI Stack
RSS
All models

Models · deepseek

DeepSeek-R1-Distill-Llama-70B

Open DeepSeek · 2025-01-20 · MIT

Dense Llama 3.3 70B Instruct base supervised-fine-tuned on reasoning traces sampled from DeepSeek-R1. Released alongside the R1 paper under MIT, the largest of the R1 distill series and the closest open dense reasoning model to R1-grade math performance.

Cost

$0.70 / Mtok input
$1.05 / Mtok output

Median across providers · as of 2026-05-21

via Artificial Analysis ↗

Speed

43.5 tok/sec output
342 ms TTFT

Median across providers · as of 2026-05-21

via Artificial Analysis ↗

Architecture

tokens in Embedding vocab not disclosed · llama3 tokenizer × N layers Grouped-Query Attention RoPE (Llama 3 scaling) context 128,000 tokens Dense MLP SwiGLU activation (standard) 70B active params Output projection tokens out
Schema-generated from data/models.yaml. Every label is auditable against the model's sources.

Specs

Architecture
dense
Total params
70B
Active params
70B
Context window
128K tokens
Attention
gqa
Position encoding
rope-llama3
Post-training
sft
OSI-approved
yes
Data released
no
Training code
not released

Benchmarks

Each score carries the date it was published; we never infer or interpolate missing scores.

General reasoning

MMLU-Pro 79.5 as of 2026-05-21 source ↗
GPQA-Diamond 65.2 as of 2025-01-22 source ↗

Code

LiveCodeBench 57.5 as of 2025-01-22 source ↗

Math

MATH 93.5 as of 2026-05-21 source ↗
AIME 2024 70.0 as of 2025-01-22 source ↗
AIME 2025 53.7 as of 2026-05-21 source ↗

Recommended use cases

  • open-weights reasoning at dense 70B
  • math and code reasoning where MoE serving is impractical
  • reasoning inside Llama-compatible stacks

Available quantizations

GGUF llama.cpp's container; the common local format, k-quants from Q2 to Q8. runs on llama.cpp, Ollama
AWQ Activation-aware 4-bit weight quantization for GPU serving. runs on vLLM, SGLang
GPTQ Post-training 4-bit weight quantization for GPU serving. runs on vLLM, SGLang, Transformers
EXL2 ExLlamaV2's variable-bitrate format for consumer GPUs. runs on ExLlamaV2
MLX Apple MLX 4/8-bit layout for Apple silicon. runs on Apple MLX
FP8 8-bit float, frequently a native release on Hopper / Blackwell GPUs. runs on vLLM, SGLang, TensorRT-LLM
bitsandbytes On-the-fly NF4 / INT8 weight quantization inside Transformers. runs on Transformers

Verified via the Hugging Face model tree ↗. Community quantizations change over time; the families shown are those with published weights at audit time.

Notable innovations

  • · MIT-licensed dense reasoning model at 70B
  • · Reasoning capability transferred via SFT from R1 traces
  • · Closes much of the reasoning gap to the MoE R1

Lineage

Distilled from R1 reasoning traces into a Llama 3.3 70B Instruct base.

Derived from

DeepSeek-R1 2025-01-20

Sources