Soft Labels: The Secret Ingredient

A hard label says: "this is a cat."

A soft label says: "this is 90% cat, 7% dog, 2.9% fox, 0.001% truck."

That second sentence is where all the learning lives. If you understand why, you understand distillation — everything after this post is implementation detail.

In Part 1, we established the problem: frontier models are too expensive for most production work, and small models trained from scratch aren't good enough. The fix is to have the big model teach the small one. Today we answer the question Part 1 left hanging: how do you transfer "judgment" from one neural network to another?

The answer, discovered by Geoffrey Hinton and his colleagues in their 2015 paper, is almost embarrassingly simple. You train the student on the teacher's full probability distribution instead of the right answer. And the magic — Hinton literally called it dark knowledge — hides in the probabilities the teacher assigns to the wrong answers.

What a model actually knows

When a classifier looks at a photo of a cat, it doesn't output "cat." It outputs a score for every class it knows — raw numbers called logits — which get converted into probabilities. Something like:

cat:    95.2%
dog:     4.7%
fox:     0.09%
truck:   0.004%

Normally, we throw almost all of this away. We take the argmax — "cat" — and call it the answer. For using the model, that's fine.

But look at what we discarded. The model didn't just say "cat." It said dog is roughly a thousand times more likely than truck. It said fox is more plausible than truck, but far less than dog. None of these are the answer — and yet each one is a tiny confession about how the model sees the world: cats resemble dogs, resemble foxes a little, and resemble trucks not at all.

No one explicitly taught the model that. It emerged from millions of training examples. The similarity structure of the entire visual world, compressed into the ratios between wrong-answer probabilities.

That's dark knowledge: the information encoded in how a model distributes its doubt.

The answer key vs. the mentor

Here's the analogy that made this click for me.

Imagine learning for a difficult exam with only an answer key. Question 7: B. Question 8: D. You can memorize the answers, but you learn nothing about why B, or what made C tempting but wrong. Next exam, different questions — you're lost. That's training on hard labels.

Now imagine a mentor going through the same exam with you: "It's B — but I see why you'd pick C, they look similar, and here's the difference. A and D you can eliminate instantly, they're not even close." Ten minutes with the mentor beats a hundred answer keys, because near-misses carry the structure of the subject.

The teacher model's soft labels are the mentor. The dataset's hard labels are the answer key. Distillation lets the student study with both.

And this explains the puzzle from Part 1 — why a distilled student beats an identical model trained from scratch on the same data. Per training example, the from-scratch model receives one bit of guidance: right or wrong. The distilled student receives a full ranked map of how plausible every option was. Same exam, radically richer feedback. More signal per example also means the student needs fewer examples — one of distillation's most underrated gifts.

The classic example: when a 2 looks like a 7

Hinton's original illustration was handwritten digits. Take two images, both labeled "2" — one is a clean, textbook 2; the other has a long flat top and a shallow curve, nearly a 7.

Hard labels treat these identically: "2" and "2." Every trace of the difference is erased before the student sees it.

The teacher's soft labels don't. For the clean one: 2 at 99.9%. For the ambiguous one: 2 at 78%, 7 at 19%. The teacher is saying: this example sits near the boundary between 2 and 7 — and here's exactly how near. The student inherits the teacher's map of where the boundaries are and which examples live close to them — which is most of what "knowing" a classification task even means.

One problem — and one dial to fix it

There's a catch. Well-trained models are confident. In practice the teacher's distribution often looks like: cat 99.97%, everything else microscopic. The dark knowledge is technically in there, but squeezed into numbers like 0.0001 — far too faint to shape the student's training.

The mentor knows which wrong answers were tempting, but only ever mutters it under their breath. We need them to speak up.

The fix is one parameter: temperature, written T. Before converting the teacher's logits into probabilities, divide them by T. At T = 1, nothing changes. As T rises, the distribution softens — the gap between the top answer and the rest shrinks, and the wrong-answer structure becomes visible.

Here it is in code — this is the entire trick:

import torch
import torch.nn.functional as F

# Teacher's raw scores (logits) for: [cat, dog, fox, truck]
logits = torch.tensor([8.0, 5.0, 1.0, -2.0])

for T in [1, 4]:
    probs = F.softmax(logits / T, dim=0)
    print(f"T={T}: " + ", ".join(f"{p:.4f}" for p in probs))

Output:

T=1: 0.9517, 0.0474, 0.0009, 0.0000
T=4: 0.5786, 0.2733, 0.1006, 0.0475

Same logits, same model, same knowledge. At T = 1, the teacher says "cat, obviously" — one bit of information. At T = 4, it says "cat, but I want you to notice it's dog-like, somewhat fox-like, and nothing like a truck." The similarity structure was always in the logits; temperature just turns up the volume on it.

A physical intuition for the name: in chemistry, heating a substance spreads its molecules across more energy states. Heating a probability distribution spreads its mass across more classes. Distillation — actual distillation, the kind with flasks — uses controlled temperature to separate and extract what matters. Hinton knew what he was doing when he named this.

How the student actually trains

So the full recipe, at intuition level (the precise math is Post 4):

1. Run each training example through the frozen teacher, at high temperature, to get soft labels.
2. Train the student to match those soft distributions — its own softmax also at the same high temperature during this comparison.
3. Simultaneously, train the student on the ground-truth hard labels as usual — the mentor is brilliant but not infallible, and the answer key keeps the student honest.
4. Blend the two objectives with a weighting factor, and — one subtlety almost every tutorial silently drops — scale the soft-label loss by T², or the two signals fall out of balance. (Why T²? Post 4. It's a good story about gradients.)

At inference time, temperature goes back to 1. The student answers normally. T is scaffolding for learning, not part of the final building.

Notice what steps 2 and 3 mean together: the student is learning from two teachers at once — the dataset, which knows what's true, and the big model, which knows what's similar to what. Neither alone is as good as both.

Why this idea aged so well

Hinton's paper is from 2015 — geological time in ML. The reason it still anchors every distillation pipeline in 2026 is that "wrong answers carry structure" turned out to generalize far beyond image classifiers.

A language model's next-token distribution is dark knowledge at massive scale: after "The capital of France is", the model puts most mass on " Paris" but meaningful mass on " the" and " a" — and how it spreads doubt across 100,000+ tokens encodes grammar, facts, and style all at once. Every "mini" and "flash" model learning from its flagship sibling is drinking from that stream. And when API-only teachers won't share their logits at all, the workaround — having the teacher generate whole training examples instead — is dark knowledge by other means. (That story is Post 6.)

One idea, three sentences long, still load-bearing eleven years later: don't just copy the teacher's answers — copy the shape of its doubt.

What's next

We now know what gets transferred (dark knowledge) and the dial that makes it visible (temperature). But matching output probabilities is only one way to learn from a teacher. You can also peek inside its head — matching its intermediate representations, or the way it organizes examples relative to each other. That's Part 3.