The Three Flavors of Distillation: Response, Feature, and Relation

You can learn from a master chef three ways.

You can collect their recipes — the finished outputs. You can shadow them in the kitchen — watching how they hold the knife, when they taste, how they adjust the heat. Or you can study how they pair flavors across every dish they've ever made — the underlying map of what goes with what.

Same chef, three completely different kinds of learning. Distillation has an exact version of each, and choosing the right one is the first real engineering decision you'll make when you distill a model.

Quick recap for anyone joining now: Part 1 made the case that big models are too expensive to deploy and small models need a teacher. Part 2 showed what the teacher transfers — dark knowledge, the structure hidden in how a model distributes doubt across wrong answers. Today: the three places in the teacher you can extract that knowledge from.

Flavor 1: Response-based — copy the answers

This is the classic setup from Hinton's 2015 paper, and everything in Part 2 was describing it: run examples through the teacher, collect its softened output distributions, train the student to match them.

The loss, in plain words before symbols:

"Student, how surprised would the teacher be by your probability distribution? Minimize that surprise."

Formally, it's the KL divergence between the teacher's softened outputs and the student's:

L_response = KL( teacher_probs(T) ‖ student_probs(T) )

(One equation per flavor, promised. The full training loss with temperature and the T² factor gets its own post — that's Part 4.)

Why it's the default. Response-based distillation only needs the teacher's outputs. It doesn't care whether the teacher is a transformer and the student is something else entirely, whether the teacher has 96 layers and the student has 4, or even — crucially — whether you have access to the teacher's insides at all. The teacher is a black box that answers questions. That's the whole requirement.

Where it falls short. Recipes tell you what the dish should taste like, but nothing about technique. If the student is much smaller than the teacher, matching final outputs can be too little guidance arriving too late — the student sees where it should end up but not how to get there. Researchers call this the capacity gap: a student too small to follow the teacher's reasoning ends up imitating its confidence without inheriting its competence. (Sound like anyone you've worked with?)

Flavor 2: Feature-based — copy the thinking

So open the box. A neural network is a stack of layers, and each layer's intermediate output — its hidden representation — is a snapshot of the model mid-thought. Feature-based distillation trains the student to match those snapshots, layer by layer: not just "end at the same answer" but "pass through similar intermediate conclusions along the way."

The plain-words loss:

"Student, at your layer 3, produce a representation that looks like the teacher's at layer 12. And at your layer 6, match its layer 24..."

L_feature = MSE( project(student_hidden_L) , teacher_hidden_L' )

This is the shadowing-in-the-kitchen flavor, and it's how the famous small models were actually made. FitNets introduced the idea in 2015 ("hints" from teacher layers); TinyBERT pushed it furthest, matching not just hidden states but attention maps — teaching the student where the teacher looks, not only what it concludes. DistilBERT's recipe includes a cosine loss pulling student and teacher hidden states into alignment. We'll dissect those recipes properly in Part 6.

The price you pay. Notice the two extra pieces of machinery in that equation. project(...): the teacher's hidden states are typically wider than the student's (say, 1024 numbers vs 384), so you can't compare them directly — you bolt on a small learned projection layer just to translate between the two, scaffolding you throw away after training. And the pairing L → L': which student layer should imitate which teacher layer? A 4-layer student learning from a 24-layer teacher has no natural alignment — every 6th layer? Spread evenly? This layer mapping is a genuine design decision, it measurably affects results, and there's no universally right answer.

The bigger constraint: you need the teacher's internals. White-box access. If your teacher lives behind an API, this flavor is simply off the menu — a constraint that shapes the entire modern LLM distillation landscape, as we'll see in Part 6.

Flavor 3: Relation-based — copy the worldview

The subtlest flavor, and the one people usually meet last.

Response-based matches what the teacher says about one example. Feature-based matches how it processes one example. Relation-based matches something neither of them touches: how the teacher organizes examples relative to each other.

Concretely: take a batch of inputs — a cat photo, a dog photo, a truck photo. In the teacher's embedding space, cat and dog sit close together; truck sits far away, off at some angle. That geometry — the distances and angles between examples — is the teacher's worldview: its sense of what resembles what. Relation-based distillation trains the student to reproduce the geometry:

"Student, I don't care if your internal coordinates match the teacher's. But whatever is close together in the teacher's head should be close together in yours."

L_relation = Σ  loss( d_student(xᵢ, xⱼ) , d_teacher(xᵢ, xⱼ) )

where d measures distance (or angle) between pairs of examples in each model's embedding space.

The elegant part. The student doesn't imitate the teacher's coordinates — only the shape of its knowledge. That makes this flavor indifferent to architecture and dimensionality in a way even response-based isn't: a 384-dimensional student can mirror the relational map of a 4096-dimensional teacher with no projection layers at all. It's the natural choice when the geometry is the product — embedding models, retrieval systems, semantic search, recommendation. If your RAG pipeline's retriever was distilled (many are), odds are something relation-shaped was in the recipe.

The catch. It transfers structure, not answers. For a straight classification task, relation-based alone usually underperforms — the student learns what's similar to what but never quite learns what anything is. It shines as a companion: answers from one flavor, worldview from this one.

Choosing your flavor (and why you usually don't choose just one)

The decision usually collapses to two questions.

Question 1: Can you see inside the teacher? If no — API-only teacher, no logits, no hidden states — response-based is your only flavor, and in the LLM era there's a further twist even on that (Part 6). If yes, keep reading.

Question 2: How big is the size gap? Teacher and student reasonably close — response-based alone is probably enough; it's the simplest to build and debug, and simplicity in training pipelines is worth real money. Gap is large — a 4-layer student drinking from a 24-layer teacher — add feature-based hints so guidance arrives throughout the network, not just at the exit. And if what you're shipping is an embedding space rather than answers, make relation-based the main course.

In practice, the strongest recipes are combinations. DistilBERT uses three losses at once — soft labels (response) + language modeling (hard labels) + cosine alignment (feature). TinyBERT stacks attention transfer on hidden-state matching on response matching. The three flavors aren't competitors; they're courses in the same meal. The real skill — and most of the tuning pain — is in the blending weights, which is exactly where the math in Part 4 earns its keep.

One forward-looking note: the flavor taxonomy is also a map of leverage. Response-based is being reinvented for the API era (teachers that generate training data instead of sharing logits), feature-based is having a renaissance in on-device model families where labs control both teacher and student, and relation-based quietly powers the embedding models underneath every RAG system. Eleven years after Hinton's paper, all three flavors are still compounding.

What's next

We've now toured the what (dark knowledge), and the where (outputs, hidden layers, geometry). Time for the how: the actual loss function — KL divergence, the α blending knob, and the T² scaling factor that most tutorials silently drop and then wonder why their distillation underperforms. With the full thing in ~20 lines of PyTorch. That's Part 4.