AI Daddy › Training & Adaptation
Fine-Tuning Strategies · Training & Adaptation
Fine-tuning adapts a pretrained model to specific tasks, domains, or styles. Today, fine-tuning is less about "teaching facts" and more about "teaching…
Fine-Tuning Strategies
Fine-tuning adapts a pretrained model to specific tasks, domains, or styles. Today, fine-tuning is less about "teaching facts" and more about "teaching format and behavior."
Table of Contents
When to Fine-Tune
Before fine-tuning, ask: Can this be solved with Prompt Engineering or RAG?
| Requirement | Better Solution | Why |
|---|
| New Facts / Knowledge | RAG | LLMs are bad at memorizing facts from FT; RAG is easier to update. |
| Specific Output Format | Fine-Tuning | Teaches the model to reliably output JSON/XML without complex prompting. |
| Tone / Persona | Fine-Tuning | Much more consistent than system prompts. |
| Latency Reduction | Fine-Tuning | Reduces the need for long few-shot prompts. |
| Private Domain Language | Continued Pretraining | Teaches specialized vocabulary (medical, legal, custom code). |
Supervised Fine-Tuning (SFT)
The first step after pretraining. The model is trained on (Prompt, Response) pairs.
The Quality Hierarchy
1,000 "Perfect" examples beat 1,000,000 noisy examples.
- Golden Sets: Hand-curated by domain experts (PhD level for technical tasks).
- Negative Constraint Training: Including examples of what the model should not do (e.g., "Don't apologize," "Don't mention you are an AI").
Continued Pretraining (Domain Adaptation)
Also known as "Second-stage Pretraining."
- How: Train on raw text from a specific domain (e.g., all SEC filings for a finance model).
- Objective: Learn the statistical distribution of the domain language.
- Nuance: Requires a much lower learning rate (~1/10th of original) to prevent "catastrophic forgetting."
PEFT vs. Full-Parameter
| Feature | Full-Parameter FT | PEFT (LoRA, QLoRA) |
|---|
| GPU VRAM | Very High (Model Size * 4-12) | Low (Model Size * 1.5) |
| Speed | Base | 2x-3x Faster |
| Risk | High (Catastrophic Forgetting) | Low |
| Deployment | One model per task | One base model + multiple adapters |
| Verdict | Reserved for foundation training | The Production Standard |
Hyperparameter Tuning
1. Learning Rate (LR)
- SFT:
1e-5 to 5e-5 is standard.
- Too high: Model "collapses" and starts repeating or speaking gibberish.
2. Rank (r) for LoRA
- Higher ranks (
r=64 to r=256) for complex reasoning tasks.
- Lower ranks (
r=8) for simple style/tone changes.
3. Packaged Training (Packing)
To maximize throughput, we "pack" multiple short examples into a single 4k or 8k sequence, separated by EOS tokens.
- Challenge: Self-attention might leak across examples.
- Solution: FlashAttention with block-masking to prevent cross-example attention.
Interview Questions
Q: Why use Continued Pretraining instead of just putting domain data in the SFT set?
Strong answer:
SFT is "expensive" in terms of data creation—you need prompt/answer pairs. Continued Pretraining allows you to leverage massive amounts of raw, unlabeled domain text to teach the model's inner representations the specialized vocabulary and style. Once the model "speaks the language," you use a small SFT set to teach it the "tasks" (e.g., classification, summarization) in that language.
Q: How do you prevent a model from "unlearning" general capabilities during fine-tuning?
Strong answer:
This is "Catastrophic Forgetting." Two main mitigations:
- Rehearsal: Mix in 5-10% of the original pretraining data into your fine-tuning set.
- PEFT (LoRA): Since we only train a small percentage of weights, the original "knowledge" remains frozen in the base model weights, significantly reducing the risk of forgetting.
References
- Hu et al. "LoRA: Low-Rank Adaptation of Large Language Models" (2021)
- Ouyang et al. "Training language models to follow instructions" (InstructGPT, 2022)
- Dettmers et al. "QLoRA: Efficient Finetuning of Quantized LLMs" (2023)
Next: LoRA, QLoRA, and PEFT