Distillation in the Wild: DistilBERT, Mini Models, and Synthetic Data

Every "mini", "turbo", "flash", and "nano" model you've used was almost certainly taught by its bigger sibling. The labs don't publish full recipes — but the techniques are an open secret, and today we walk through them: the classic that started it all, the modern LLM-era twist where teachers generate their own textbooks, and the awkward legal question hovering over the whole field.

Recap for new readers: Parts 1–4 built the theory (the deployment gap, dark knowledge, the three flavors, the loss function); Part 5 distilled a real model and published the receipts. Now: how this looks at industrial scale.

DistilBERT: the recipe that made distillation mainstream

In 2019, a four-person team at Hugging Face compressed BERT into something 40% smaller and 60% faster that kept ~97% of its language-understanding performance. The numbers made distillation an industry default; the recipe is worth reading closely because it uses all three flavors from Part 3 at once.

The triple loss:

1. Soft labels (response-based). The classic Hinton term — match the teacher's softened output distribution over the vocabulary. The dark-knowledge channel: after "The cat sat on the ___", BERT's spread of probability across mat, sofa, floor, chair teaches more than the answer alone. (Parts 2 and 4.)
2. Masked language modeling (the answer key). Plain hard-label pretraining loss on real text — the textbook keeping the student honest.
3. Cosine alignment on hidden states (feature-based). Pull the student's internal representations into the same directions as the teacher's — shadowing in the kitchen, not just copying plated dishes.

Plus two moves you can steal directly. Smart initialization: the student is born from the teacher — 6 layers taken from BERT's 12 (we used the same trick in Part 5; it's free accuracy). Distill during pretraining, not just fine-tuning: the student inherits general language understanding first, then fine-tunes on whatever task you like. That one decision is why a single distilled checkpoint could replace BERT across dozens of downstream tasks.

TinyBERT (2020) pushed feature-matching further — distilling attention maps so the student learns where the teacher looks, plus a two-stage general-then-task pipeline. MobileBERT re-architected the student around a bottleneck design so feature-matching would fit a phone. Different bets, same theme: when you own the teacher's internals, white-box distillation with multiple simultaneous losses is the power tool.

The LLM-era twist: the teacher writes the textbook

Then models grew 1000x, moved behind APIs — and broke the classic recipe. GPT-class teachers won't hand you logits over the wire (and even when the raw numbers exist internally, providers don't expose them at usable fidelity). No logits means no KL term; no weights means no feature matching. Every white-box technique above: off the menu.

The workaround reshaped the field. If the teacher won't share its probability distributions, have it do something better: generate entire training examples.

This is sequence-level distillation: prompt the teacher to produce inputs, outputs, explanations — tens or hundreds of thousands of them — then fine-tune the student on the synthetic corpus with ordinary hard-label training. The 2023 proof was Alpaca: Stanford took a 7B open model, fine-tuned it on ~52,000 instruction-response pairs generated by an OpenAI model (seeded from just 175 hand-written examples — the "self-instruct" bootstrap), and got startlingly capable instruction-following for a few hundred dollars of API calls. The wave that followed — and the "trained on synthetic data" lines in today's model cards — runs on descendants of this idea.

Squint and it's still dark knowledge, delivered differently. Per-token soft labels whisper "here's the shape of my doubt"; a generated dataset demonstrates the teacher's judgment — which phrasings it chooses, how it structures reasoning, what a good answer even looks like. The knowledge arrives as behavior instead of probabilities. The mentor stopped annotating your exam and wrote you a personalized textbook.

Black-box vs. white-box, the honest comparison. White-box (own the teacher): logits + features + geometry, maximum signal per example — this is how labs build their own mini/flash/nano siblings, and why their smalls are so strong. Black-box (API teacher): behavior only, but with a compensating superpower — you steer the curriculum. Need a student great at your domain? Have the teacher generate exactly that distribution of examples, including the edge cases production keeps surfacing. Remember Part 5's lesson about distilling on the wrong data distribution: synthetic generation is the cure, because you can manufacture the right distribution. Quality control is the tax: teachers generate duplicates, blandness, and confident nonsense, so real pipelines filter aggressively — dedup, heuristics, and increasingly a second model grading the first one's output. Curating the synthetic textbook is most of the actual work.

The paragraph nobody likes to write

Now the awkward part, stated plainly.

Most frontier API providers' terms of service prohibit using their outputs to train competing models. Distilling a commercial teacher into your commercial student, via exactly the synthetic-data pipeline above, sits somewhere between "violates the ToS" and "genuinely contested," depending on jurisdiction, purpose, and lawyer. The industry's open secret is that the practice is widespread — accusations between labs have made headlines, which is precisely why model cards say "synthetic data" and leave the sentence unfinished. This isn't settled law; it's an active fault line in AI's economics, because distillation transfers in weeks what cost hundreds of millions to create.

Practical guidance, not legal advice (I'm an engineer, not your lawyer): for commercial work, distill from teachers whose licenses explicitly allow it. The strongest open-weight model families now ship with licenses permitting distillation into your own products — and give you logits and hidden states, unlocking every white-box technique the APIs lock away. In 2026, "open-weight teacher + white-box distillation + synthetic data you control" is not the compromise option; it's frequently the technically superior pipeline and the one that lets you sleep.

The quiet engine of on-device AI

Zoom out and notice where all of this lands: not in data centers — in your pocket.

Phone keyboards predicting your next word. On-device speech recognition that works in airplane mode. The photo app finding "beach" without uploading anything. Cars making lane decisions with no network. Watches flagging heart rhythms. Every one of these runs a model squeezed into a memory and power budget that would make a server laugh — and distillation is, overwhelmingly, how they got there (usually with quantization stacked on top; Part 7 covers how those combine).

This is the forward-looking bet hiding in this whole series: as capability keeps migrating from cloud to device — for latency, privacy, and cost all at once — distillation stops being an optimization trick and becomes the distribution channel for intelligence itself. Frontier models are where capability is born. Distilled models are how it ships.

What's next

We've covered how models get distilled once. The finale covers what nobody blogs about: keeping a distilled model good — pipelines, monitoring the tails, distill-vs-quantize-vs-prune (and why the answer is "stack them"), the cost math that justifies all of this to your CFO, and whether you can distill reasoning itself. That's Part 7.