Model Distillation: 32B Beats o1-Mini at Half the Cost

A 32-billion-parameter student model fine-tuned on DeepSeek-R1’s reasoning traces scored 72.6% Pass@1 on AIME 2024, beating OpenAI’s o1-mini (63.6%) while costing roughly an order of magnitude less per token to serve. Released with the DeepSeek-R1 checkpoints, it is the strongest production evidence yet that knowledge distillation—not larger GPUs—is the dominant lever for cutting reasoning-model inference cost.

A 32B Model Beat o1-Mini

The benchmark that reframed distillation from a compression trick into a strategy decision comes straight from DeepSeek’s own evaluation table. Six dense checkpoints—1.5B through 70B—were fine-tuned on reasoning samples generated by the full DeepSeek-R1 teacher, then published openly on the project’s repository.

Model	AIME 2024 (Pass@1)	MATH-500 (Pass@1)	GPQA Diamond
OpenAI o1-mini	63.6	90.0	60.0
R1-Distill-Qwen-7B	55.5	92.8	49.1
R1-Distill-Qwen-14B	69.7	93.9	59.1
R1-Distill-Qwen-32B	72.6	94.3	62.1
R1-Distill-Llama-70B	70.0	94.5	65.2

A 32B dense model outscoring a purpose-built frontier reasoning model on competition mathematics is not a rounding error. The DeepSeek authors state it plainly: reasoning patterns distilled from a large teacher yield better small-model performance than running reinforcement learning directly on the small model (arXiv:2501.12948). That single finding inverts the default build instinct—you no longer need to RL-train every model that ships.

Soft-Label Distillation Fails on Reasoning

The technique most engineers picture when they hear “distillation” is Hinton’s original formulation: the student learns from the teacher’s softened output distribution, the so-called soft targets, which carry more signal than a hard label (Hinton et al., 2015). A temperature parameter exposes the teacher’s confidence across alternatives; Hinton’s own experiments used temperatures as high as 20.

That recipe works for classification and short-form generation. It does not transfer reasoning. The teacher’s next-token probabilities encode what to say, not how to think. A chain-of-thought trajectory is a sequence of intermediate decisions—verification, backtracking, strategy switching—that a softmax over the vocabulary cannot compress. Run standard logit distillation on a reasoning teacher and the student reliably learns to produce confident, fluent, wrong derivations.

The practical consequence: teams that copy the DistilBERT playbook onto an o1-class teacher get a faster model that has lost the capability they actually paid for. The Redis distillation guide notes DistilBERT retains 97% of BERT-base accuracy at 40% smaller, and TinyBERT-4 hits 13.3% of BERT parameters—but those are encoders on bounded tasks, not open-ended reasoners.

Trace Distillation: What Actually Transfers

DeepSeek’s contribution was not a new loss function. It was treating the teacher’s full reasoning traces as the training signal. The teacher generates complete chain-of-thought solutions—with its self-reflection, error-checking, and recovery intact—and the student is supervised to reproduce that trajectory token by token. The student inherits the teacher’s process, not just its outputs.

This is closer to behavior cloning than to classical distillation, and it has a sharp engineering implication: data quality is the entire game. DeepSeek curated hundreds of thousands of verified reasoning traces from R1, filtering out the endless repetition and language-mixing that plagued the earlier R1-Zero (DeepSeek-R1 README). A student trained on noisy, unverified traces will faithfully reproduce the noise. Garbage in, fluent garbage out.

For platform teams, the takeaway is that distillation is a data pipeline problem before it is a modeling one. You need the teacher to attempt, verify, and self-correct at scale, then a rejection sampler that keeps only trajectories reaching a checked answer. Skip that filter and your 32B student inherits the teacher’s hallucinations without the teacher’s capacity to recover from them.

The Cost Math: Teacher vs Student

Why the economics are suddenly compelling: reasoning models are expensive precisely because they spend tokens thinking. A frontier reasoning model might emit thousands of hidden chain-of-thought tokens before the final answer, and you pay for every one. A distilled 7B or 14B student running locally on a single GPU produces the same final answer with no remote reasoning tax—the same self-host economics that make NVIDIA NIM deployments beat per-call API pricing at volume.

The spending backdrop makes this hard to ignore. Enterprise GenAI spend climbed from $11.5B in 2024 to $37B in 2025, with 80% of companies overshooting their forecasts by 25% or more, and an estimated 40–60% of token budgets classified as waste (2026 cost-reduction analysis). Distillation attacks the most expensive slice of that waste: paying frontier-model prices for reasoning that a smaller, specialized student handles within tolerance.

Compared to other levers, distillation is the only one that permanently changes the unit economics—and unlike hardware bets like custom silicon that halves compute cost, it works on the GPUs you already own. Prompt caching saves 50–90% on repeated prefixes but does nothing for novel queries. Routing cuts 40–70% by dispatching easy queries to cheap models but still routes the hard ones to the expensive teacher. Distillation replaces the teacher for an entire capability band—and the saving compounds every request, forever.

A Practical Distillation Pipeline

The workflow that consistently produces deployable students, drawn from the Redis guide and the DeepSeek recipe, is five steps. Each one has a failure mode that quietly degrades the result.

Select a frozen teacher. Pick a frontier model that is strong on your target distribution. It stays frozen—its weights never change. The teacher’s only job is to generate high-quality traces.
Design the student. Choose a backbone (Qwen, Llama, Mistral) small enough to serve cheaply but large enough to absorb the teacher’s reasoning. A 7B–32B range covers most production needs.
Generate and filter traces. Run your real workload through the teacher, then rejection-sample: keep only traces whose final answer passes a verifier. This filter is where most teams lose quality.
Train with combined loss. Supervise the student on the teacher’s traces. Where soft labels are available, blend them: Total Loss = α × Distillation Loss + (1−α) × Student Loss, with temperature controlling distribution sharpness (Redis).
Validate on held-out distribution. Benchmarks like AIME and MATH-500 are necessary but not sufficient. You must eval on your own traffic, because distribution drift is where distilled students collapse.

Where Distillation Silently Breaks

Three failure modes account for most of the production disappointments I have seen, and none of them show up on the benchmark leaderboard.

Distribution drift. A student distilled on competition math excels at competition math. Route it customer-support tickets and accuracy collapses. The teacher’s reasoning is domain-specific; the student inherits that specificity. Re-distill per workload, or accept the quality cliff at the domain boundary.

Eval leakage. Public benchmarks are contaminated. A student that looks great on MATH-500 may have memorized the test set during trace generation. Hold out an internal eval set the teacher never saw, and gate deployment on that.

Lost self-correction. The teacher recovers from its own mistakes mid-trace; the student, with less capacity, often cannot. On hard multi-step problems the student commits to an early wrong step and never recovers. Measure accuracy and recovery rate, not just final-answer correctness.

Stacking Distillation With Routing and Caching

Distillation is powerful alone but brutal in combination. The right architecture layers three independent cost reductions, each attacking a different segment of the traffic distribution.

Distillation handles the reasoning band—the queries that genuinely need chain-of-thought—on a cheap local student.
Routing sends trivial queries (classification, extraction, FAQ) to a $0.14/M-token model, saving 40–70% on the easy tail—the same dispatch logic that cuts 85% of API spend in production (cost analysis).
Prompt caching eliminates re-processing of static prefixes, cutting cached-token cost 50–90% (cost analysis).

Stacked, these levers routinely clear 70% total reduction on real enterprise traffic. Distillation is the only one that requires upfront training investment—but it is also the only one whose savings do not decay as your query distribution shifts toward novel, hard prompts.

What to Measure Before Committing

Before you replace a frontier reasoning endpoint with a distilled student, instrument three numbers on your live traffic for at least two weeks. They tell you whether distillation will actually pay off, or merely move cost into a less visible place.

Metric	What it tells you	Green-light threshold
Teacher vs student accuracy gap	Capability loss on your distribution	< 5% on internal eval
P95 latency per request	User-visible speed gain	Student ≤ 0.5× teacher
Cost per correct answer	True unit economics, errors included	Student ≤ 0.2× teacher

The last metric is the one teams skip. A student that is 10× cheaper per token but 20% less accurate can cost more per correct answer once retries, escalations, and human review are counted. Distillation only wins when cost-per-correct-answer drops—not when cost-per-token does. Measure the denominator.

Cloud AI