AI Daddy › Training & Adaptation
Synthetic Data Generation · Training & Adaptation
The industry has hit the "Data Wall", the exhaustion of high-quality human text on the web. Synthetic data is now the primary engine for model improvement…
Synthetic Data Generation
The industry has hit the "Data Wall", the exhaustion of high-quality human text on the web. Synthetic data is now the primary engine for model improvement, sitting at the core of every modern frontier-model recipe.
Table of Contents
After the "Data Wall": The Synthetic Shift
Frontier models (Llama 4, GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro) are trained on 100T+ tokens. There simply isn't enough human text to sustain this scaling.
The reality: More than 50% of the training mixture for frontier fine-tuning (and 10% of pretraining) is now synthetic.
| Source | Human Data | Synthetic Data |
|---|
| Volume | Fixed (Finite) | Infinite |
| Quality | Variable (Noisy) | Controllable (Purified) |
| Cost | High (Human Labelers) | Cheap (Inference/GPU) |
| Bias | Mirror of internet | Can be manually balanced |
Evol-Instruct Pattern
Evol-Instruct is a recursive process where an LLM takes a simple instruction and evolves it into a more complex one.
The Evolution Directions:
- Breadth: Increase the number of tasks.
- Depth: Add constraints, complicating factors, or multi-step logic.
- De-noising: Clean up phrasing to remove "AI-isms."
# Simple Instruction: "Write a function to add two numbers."
# Evolved Instruction: "Write a thread-safe Python class that performs
# matrix addition with error handling and unit tests, adhering to PEP8."
Constitutional AI & AI Feedback (RLAIF)
Developed by Anthropic and widely adopted across the industry, RLAIF uses a "Constitution" (a set of rules) to guide a model in evaluating and improving its own data.
The Loop:
- Propose: Model A generates a response.
- Critique: Model B (the constitutional judge) identifies flaws based on guidelines.
- Revise: Model A produces a better version based on the critique.
- Train: The final (Prompt, Revise) pair is added to the SFT set.
Verifiable Synthetic Data
The biggest risk of synthetic data is Model Collapse (the model learning its own mistakes).
The 2025 Solution: Focus on domains where the "Truth" is verifiable without an LLM.
- Math: Use Formal Verification (Lean/Isabelle) or Python execution to verify answers.
- Code: Run generated code against test cases (Unit Tests).
- RAG: Use "Gold Context" to generate questions where the answer is explicitly in the text.
De-biasing and Diversity
Synthetic data is used to "fill the gaps" in human data.
- Languages: Generating high-quality text in low-resource languages (e.g., Swahili, Marathi) by translating conceptual templates.
- Logic: Creating 1,000,000 variations of a specific logical fallacy to "harden" the model against it.
Interview Questions
Q: What is the risk of "Model Collapse" when training on synthetic data?
Strong answer:
Model Collapse occurs when a model is trained on data generated by an earlier version of itself. Because the model's distribution is narrower than the real world (it has preferences/biases for certain words and patterns), the training loop becomes a "positive feedback loop" of errors and blandness. By 2025, we mitigate this by:
- Mixing in 5-20% "Golden" human-authenticated data.
- Using "Verifiable" rewards (Math/Code) so mistakes are never learned.
- Using more powerful "Teacher" models to generate data for "Student" models.
Q: How do you ensure the quality of a synthetic dataset of 10 million rows?
Strong answer:
We use a Multi-Stage Filtering Pipeline:
- Semantic Deduplication: Using embeddings to remove near-identical clusters.
- LLM-as-Judge: Sampling 1% of the data and having a stronger model (e.g., GPT-5.2) score it for logic and safety.
- Perplexity Filtering: Using a small model to calculate the perplexity of the text. If it's too high (nonsense) or too low (repetitive/simple), it's discarded.
- Verifiable Execution: If the data contains code or math, it must pass a local compiler/interpreter check.
References
- Xu et al. "WizardLM: Empowering Large Language Models to Follow Complex Instructions" (2023)
- Bai et al. "Constitutional AI: Harmlessness from AI Feedback" (2022)
- OpenAI. "Weak-to-Strong Generalization" (2023)
Next: Quantization Deep Dive