Model Strategy
How big models teach small models.
The most capable models are often too expensive, too slow, or too large for the place where the product needs intelligence. Knowledge distillation is one way to move capability from a large model into a smaller model that is easier to deploy.
The small model is not a compressed copy. It is a separate student model trained to imitate useful behavior from a larger teacher model.
Practical Takeaway
Distillation turns a capable model into a training source.
A teacher model can provide richer signals than ordinary labels. The student learns from answers, probabilities, explanations, generated examples, or internal representations, depending on what the training process can access.
Core Idea
Distillation is different from compression.
Compression starts with an existing model and reduces its size through techniques like quantization or pruning. The identity of the model stays largely the same.
Distillation trains a new model. The teacher helps shape training, but the deployed student no longer needs the teacher beside it. That difference matters because the student can have a different architecture, cost profile, and deployment target.
Training Signal
Soft labels carry more information than one right answer.
A hard label says the answer is cat. A soft label says the teacher thinks cat is likely, dog is somewhat plausible, and fox is unlikely. That distribution teaches relationships between categories.
The student does not merely memorize the final answer. It learns the teacher's pattern of confidence. That can make each example more useful than a single label and can improve generalization on narrow tasks.
{
"hard_label": "cat",
"teacher_soft_label": {
"cat": 0.70,
"dog": 0.25,
"fox": 0.05
}
}Methods
There are three common forms.
Output distillation trains the student to match the teacher's final probabilities. Feature distillation trains the student to match internal representations. Synthetic data distillation has the teacher generate examples, then the student trains on that generated dataset.
Synthetic data is common because many strong teacher models are only available through an API. If probabilities and internal activations are hidden, generated examples are still accessible.
Output
Copy probability patterns when logits are available.
Feature
Copy intermediate representations when internals are available.
Synthetic data
Use the teacher to create examples for fine-tuning.
Limits
Small students work best when the task is narrow.
Distillation can produce strong small models for code, math, extraction, classification, routing, or a domain-specific workflow. It is less likely to turn a tiny model into a broad general-purpose assistant.
The teacher still matters. A bad teacher passes bad habits. A teacher that is too far above the student can also be difficult to copy. Sometimes a middle-sized teacher assistant makes the transfer smoother.
Architecture matters too. Parameter count alone does not decide whether the student will learn well. The base model, training data, target task, and evaluation set all shape the result.
Deployment Fit
Distillation is strongest when the deployment target is clear.
A small distilled model is useful when the product has a repeatable task and a hard constraint: low latency, low cost, offline operation, private data, mobile hardware, or high request volume.
The team should define the target before training. A model intended for an iPhone, an Android tablet, a support triage service, and a batch analytics job may need different size, quantization, context, and accuracy tradeoffs.
The final student still needs normal software discipline: versioned datasets, held-out tests, bias checks, regression tests, observability, rollback plans, and clear ownership after release.
Good use case
A narrow task with many similar examples and a measurable success metric.
Weak use case
A broad assistant expected to match the teacher across every domain.
Release rule
Compare the student against the teacher, the old system, and real user examples.
Related Reading