AI Daddy › Training & Adaptation
Knowledge Distillation · Training & Adaptation
Knowledge distillation is the process of transferring the intelligence from a large, complex model ("Teacher") to a smaller, more efficient one…
Knowledge Distillation
Knowledge distillation is the process of transferring the intelligence from a large, complex model ("Teacher") to a smaller, more efficient one ("Student"). It is a large part of why today's small open-weight models punch well above their parameter count.
Table of Contents
The Teacher-Student Paradigm
Many small models are not trained on raw web data alone. They are trained on outputs, logits, or reasoning traces from a much larger model. The documented cases are open models:
- Gemma 2 (2B and 9B) and all Gemma 3 sizes were pretrained with distillation from a larger teacher.
- Llama 3.2 1B/3B incorporated logits from Llama 3.1 8B and 70B during pretraining; Llama 4 Maverick was codistilled from the unreleased Llama 4 Behemoth.
- DeepSeek-R1's distilled checkpoints are Qwen and Llama models fine-tuned on about 800K samples generated by R1.
- NVIDIA's Nemotron-Labs competitive-coding model (September 2026), which scored above the IOI 2026 gold threshold in an unofficial run, was trained on 477,642 reasoning traces distilled from Z.ai's MIT-licensed GLM-5.2. Teacher lineage now crosses vendors and borders, so record it.
Closed labs do not publish small-model recipes. "Claude Haiku is distilled from Opus" or "Gemini Flash is distilled from Pro" is plausible, not documented.
| Model | Role | Intelligence Source |
|---|
| Teacher | Large (100B+ params, often MoE) | Frontier-scale pretraining (15T-36T+ disclosed tokens) plus post-training |
| Student | Small (1B - 30B params) | Teacher's filtered outputs, logits, or reasoning traces |
How Distillation Works
1. Hard Label Distillation
The student learns from the teacher's final predictions (e.g., the answer to a question).
2. Soft Label Distillation (Temperature Scaling)
The student learns from the teacher's probability distribution (Logits). This is much richer because it tells the student not just the right answer, but which wrong answers were "almost" right.
# Distillation Loss (KL Divergence):
Loss = KL_Div(Teacher_Logits / T, Student_Logits / T)
Where T is the Temperature (typically 2.0 - 5.0).
3. On-Policy Distillation
The student samples its own completions and the teacher scores every token, combining RL's on-policy relevance with SFT's dense signal. It became a popular cheap path to reasoning models in 2025-2026, but its mechanism is now contested: an August 2026 paper (arXiv:2608.31046, revised in September, single group) reports the teacher signal is noisy and that most of the gain comes from suppressing low-probability tokens, which a teacher-free method matched and beat. Run a teacher-free baseline before paying for a large teacher. Details in RLVR and Reasoning Models.
Feature vs. Output Distillation
Output Distillation (Standard)
Student matches the teacher's text responses.
- Pros: Easy to implement via API.
- Cons: Only learns behavioral surface patterns. Closed APIs increasingly return summarized or encrypted reasoning, so raw chain-of-thought is usually not available to distill.
- Platform risk: Vendor-hosted distillation pipelines are shrinking. OpenAI's stored-completions-to-fine-tune path depends on two products being wound down: Evals goes read-only October 31, 2026 and shuts down November 30, and self-serve fine-tuning stops accepting new jobs from active customers on January 6, 2027 (see Fine-Tuning Strategies). Keep the dataset, eval suite, and student training on infrastructure you control.
Feature/Hidden State Distillation
Student matches the inner Hidden States (vector representations) of the teacher.
- Requirement: You need access to the teacher's weights (Open Weights).
- Pro: The student learns the teacher's "internal conceptual map," leading to much higher reasoning depth.
Self-Distillation via Rejection Sampling
A core reasoning-training technique. The literature calls it rejection-sampling fine-tuning, expert iteration, or STaR. The model improves without new human data:
- Generation: The model generates many candidate solutions to a hard math/code problem.
- Verification: A rule-based system (compiler, test suite, calculator, proof checker) keeps only the correct ones.
- Distillation: The model is fine-tuned on the chains of thought that led to correct solutions.
Result: The model "distills itself" by keeping only the high-quality reasoning paths. DeepSeek-R1 documents this step: after its RL stage, it collected about 600K verified reasoning samples by rejection sampling (plus about 200K non-reasoning samples) for the next SFT round. OpenAI and Anthropic have not published their recipes. The RL version of the same idea is covered in RLVR and Reasoning Models.
Quantization-Aware Distillation
Standard quantization (e.g., 16-bit to 4-bit) causes a small drop in accuracy.
The Fix: Use Knowledge Distillation during the quantization process. The 16-bit model acts as the teacher, guiding the 4-bit model to minimize its error. Together with quantization-aware training, this is how vendors ship low-bit builds that stay close to 16-bit quality, such as Google's QAT int4 Gemma builds. See Quantization Deep Dive.
If you serve a model through an API, someone will try to distill it. This is now documented at scale:
- Anthropic's February 2026 disclosure attributed over 16M exchanges, through about 24,000 fraudulent accounts, to distillation campaigns by three Chinese labs. Proxy services ran "hydra cluster" account networks (one managed 20,000+ accounts at once), and prompts asked Claude to write out the reasoning behind a completed answer to harvest chain-of-thought training data.
- Anthropic's September 2026 threat report (covered September 10) described nearly 200M exchanges across five attributed campaigns, the largest over 151M exchanges in three months from more than 3,500 accounts.
- In August, researchers showed that encrypted reasoning blocks from several vendors could be replayed across models because each model family shared one encryption key; the vendors closed the hole.
- Hiding reasoning is leaky in other ways too: a September 2026 paper (arXiv:2609.26637) extracted hidden reasoning from frontier models, including GPT-6 Astra, by registering a simple custom tool through a standard API feature.
Design implications for API operators: rate shaping and per-account behavioral fingerprinting (including CoT-elicitation detection), stronger account verification, intelligence sharing, and limiting what reasoning you expose, then red-teaming every path that can surface it (tool calls, structured outputs, replayed blocks), not just the response text. Those countermeasures explain several API changes you will meet as a consumer: summarized thinking, and thinking blocks bound to the model and conversation that produced them (Claude Fable 5.1, Opus 5.5 and Sonnet 5.5; Sonnet 5.5 blocks are also bound to the producing account).
Interview Questions
Q: Why is a distilled 8B model better than an 8B model trained from scratch on the same tokens?
Strong answer:
Training from scratch (Pretraining) on raw web data is noisy; the model spends a lot of capacity learning to navigate that noise. A distilled model, however, is trained on a "purified" curriculum. The teacher model acts as a high-quality filter, providing structured logic, clear explanations, and a cleaner distribution of language. Essentially, the teacher provides "hints" through its logit distribution that tell the student exactly which features of the language are most important to learn.
Q: What are the risks of distilling a closed API model into an open-weight student?
Strong answer:
- Model Collapse: If the student only sees the teacher's output, it may lose the "long tail" of creative or diverse knowledge and only learn the teacher's narrow biases.
- License Violations: Most proprietary models (OpenAI, Anthropic) have clauses forbidding the use of their outputs to train "competing" models. This is a major legal risk for enterprises distilling their own models from API outputs.
- Detection and enforcement: Vendors now actively detect distillation traffic. Anthropic's 2026 reports describe fingerprinting CoT-elicitation prompts and banning account networks, so a pipeline built on bulk teacher calls can be cut off mid-project.
- Less signal than you expect: Closed APIs expose summarized or encrypted reasoning, not raw chains of thought or logits, so you are doing output distillation only.
- Linguistic Mimicry: The student might learn to sound confident (like the teacher) without actually having the same level of logical depth, leading to confident but incorrect hallucinations.
The safer pattern is an open-weight teacher with a permissive license (MIT or Apache 2.0), with teacher identity and license recorded per training row.
References
- Hinton et al. "Distilling the Knowledge in a Neural Network" (2015)
- Gou et al. "Knowledge Distillation: A Survey" (2021)
- DeepSeek. "DeepSeek-R1: Incentivizing Reasoning Capability" (2025)
- Zelikman et al. "STaR: Bootstrapping Reasoning With Reasoning" (2022)
- Ding and Zhang. "Does On-Policy Distillation Really Distill?" arXiv:2608.31046 (2026)
- Anthropic. "Detecting and preventing distillation attacks" (2026)
Next: Synthetic Data Generation