Why Everyone Runs a Model They Can't Afford

GPT-5-class models are brilliant. They're also like hiring a Nobel laureate to answer your customer support tickets.

The laureate will do a great job. Every ticket gets a thoughtful, nuanced, correct answer. But you're paying laureate rates for questions like "how do I reset my password?" — and somewhere in your finance team, someone is quietly having a breakdown.

Model distillation is how you train the intern. Not a random intern off the street — an intern who spent six months shadowing the laureate, absorbing how they think, and can now handle 95% of the tickets at 3% of the cost.

This is the first post in a 7-part series on model distillation: what it is, the math that makes it work, how to actually do it, and how the industry quietly uses it everywhere. Today's job is just to convince you the problem is real — because until the problem is real, the solution looks like an academic curiosity.

The gap nobody talks about in demos

Every AI demo you've seen runs on a frontier model. Every production budget you've seen wishes it didn't.

Here's the gap in concrete numbers. As of August 2026, API prices per million tokens look roughly like this:

• Flagship models: GPT-5.6-Sol at $5 in / $30 out, Claude Opus 5 at $5 / $25, and specialist tiers going as high as $30 / $180
• Small siblings: GPT-5.4-nano at $0.20 / $1.25, Claude Haiku 4.5 at $1 / $5, Gemini 3 Flash at $0.50 / $3
• Aggressively priced small models: DeepSeek V4 Flash at $0.14 / $0.28

Read that spread again. Between a flagship and its smallest sibling, the price difference is 25x to 100x — for what is, on many everyday tasks, a barely noticeable quality difference.

Now scale it. Say your product processes 50 million tokens a day — a mid-sized chatbot, a document pipeline, nothing exotic. On a flagship model at $30 per million output tokens, heavy months land in the tens of thousands of dollars. On a nano-class model, the same traffic costs less than your team's coffee budget.

And cost is only half the gap. The other half is latency. A big model thinking hard is a user watching a spinner. If your feature needs to respond in 300 milliseconds — autocomplete, classification, routing, anything interactive — a frontier model over an API is often disqualified before pricing even enters the conversation. Add the third constraint, privacy (some data simply cannot leave the building, or the device), and you get the deployment gap:

The model that's smart enough is too expensive, too slow, or not allowed in the room. The model that fits the budget isn't smart enough.

Distillation exists to close exactly this gap.

"So just use a smaller model" — why that fails

The obvious response: fine, train a small model on your data. A 1–3B parameter model runs on a single GPU, sometimes on a phone. Problem solved?

Here's the part that surprises people: a small model trained from scratch on your data performs much worse than the same small model taught by a big one. Same architecture, same size, same dataset — different teacher, very different results.

This feels wrong at first. The small model saw the same data. Why does it matter who taught it?

Back to our laureate. Imagine two interns. Intern A is handed the company's entire ticket archive and told: "figure it out." Intern B gets the same archive, but sits next to the laureate for six months — watching not just what answer the laureate gives, but how confident they are, which cases they treat as similar, when they hesitate, and which almost-right answers they reject and why.

Intern B will be dramatically better. Not because they saw more data — they didn't — but because they absorbed the laureate's judgment, which is a far more compressed and useful signal than raw examples.

That's the core insight of distillation, and it's worth stating precisely:

A large model has already spent enormous compute learning the structure of a problem. Distillation transfers that learned structure to a small model directly — instead of making the small model rediscover it from scratch, which it usually can't.

Why can't it? Because a small model has limited capacity, and capacity spent on rediscovering structure is capacity not spent on being good at the task. The big model's knowledge acts as a shortcut: here's what matters, here's what's similar to what, here's how confident to be. The small model gets to skip the expensive part of learning.

(How exactly do you transfer "judgment" from one neural network to another? That's Post 2, and it's the most beautiful idea in this whole series. Teaser: the secret is in the wrong answers.)

You've been using distilled models all along

Distillation sounds like a niche optimization trick. It's actually one of the most-deployed techniques in AI — it's just invisible, because it hides inside product names.

DistilBERT is the classic proof. In 2019, Hugging Face took BERT and distilled it into a model 40% smaller and 60% faster that kept about 97% of the performance. That one result turned distillation from a research paper into an industry standard. (We'll dissect exactly how they did it in Post 6.)

Every "mini", "nano", "flash", and "turbo" model you've used almost certainly had its bigger sibling somewhere in its training story. AI labs don't publish full recipes, but the pattern is an open secret: you don't get a model that cheap and that good by training it from scratch in isolation. The flagship teaches; the small one learns; you get the 25–100x price cut.

Your phone keyboard predicting your next word, on-device speech recognition, the photo app that finds "beach" pictures without uploading anything — these run tiny models on hardware with the memory budget of a decade-old laptop. Small models capable of this are almost always distilled from larger ones.

Cars, wearables, and edge devices are the extreme case: hard latency limits, no reliable network, strict power budgets. Distillation isn't an optimization there — it's the entry ticket.

So the honest framing isn't "distillation is a cool trick you might use someday." It's: distillation is already the quiet engine behind most AI that runs fast, cheap, or on-device — and if you're building with AI, you're either using it or paying the laureate for password resets.

The trade you're actually making

Let's be precise about what distillation buys and what it costs, because it's not magic.

What you give up: the student will not fully match the teacher. On broad, open-ended tasks, the gap is real. Distillation shines when the job is scoped — classification, extraction, routing, domain-specific Q&A, a support bot that handles your product and not the sum of human knowledge. On scoped tasks, students routinely reach 95–99% of teacher quality.

What you get: a model that's 10–100x cheaper to run, fast enough for interactive use, small enough to self-host or ship on-device, and fully under your control — no API terms changing under you, no data leaving your infrastructure.

The trade, in one sentence: you exchange a few points of generalist brilliance for an order of magnitude (or two) of cost, speed, and control — on the specific tasks you actually care about.

For most production use cases, that's not a hard decision. It's barely a decision at all. The hard part is doing it well — which is what the rest of this series is for.

Where this series goes

Over the next six posts, we'll go from intuition to production:

Post 2 — Soft labels and dark knowledge. Why the teacher's wrong answers carry more teaching signal than its right ones. The single most important idea in distillation.

Post 3 — The three flavors. Copying answers vs. copying thought process vs. copying how knowledge is organized — and when each applies.

Post 4 — The math, demystified. KL divergence, temperature, and the one scaling factor most tutorials silently skip. With the full loss in ~20 lines of PyTorch.

Post 5 — Hands-on. We distill a real model, publish the numbers, and watch a distilled student beat an identical model trained without a teacher.

Post 6 — Distillation in the wild. DistilBERT's recipe, how modern LLMs teach their "mini" siblings with synthetic data, and the legal gray zone nobody likes to mention.

Post 7 — Production. Pipelines, monitoring, distill-then-quantize, and where this is all heading — including whether you can distill reasoning itself.

The laureate is brilliant. But the future of deployed AI belongs mostly to well-taught interns — and by the end of this series, you'll know exactly how to train one.