AI Daddy › Inference Optimization
Cost Optimization Playbook · Inference Optimization
AI costs are no longer "magic." They are measurable, predictable, and highly optimizable. With API pricing down 30-60% over the past year, the cost lever…
Cost Optimization Playbook
AI costs are no longer "magic." They are measurable, predictable, and highly optimizable. With API pricing down 30-60% over the past year, the cost lever is now mostly about routing and caching, not just picking a cheaper provider. This chapter covers the strategies to reduce inference costs by 10x without sacrificing quality.
Table of Contents
The Unit Economics of AI
We measure success by Tokens per Dollar ($).
| Component | Cost Driver | Optimization |
|---|
| Compute | GPU Time ($/hr) | Better utilization (Batching). |
| VRAM | KV Cache Size | GQA, Quantization. |
| Network | Payload Size | Compression, Local serving. |
| API | Per-token pricing | Caching, Model selection. |
Model Cascading (Efficiency Tiers)
The most effective cost-saving strategy is to use the cheapest model capable of the task.
The cascade pattern:
- Classifier: A tiny model (0.5B) determines query complexity ($0.00).
- Tier 1 (SLM): 90% of queries (greetings, simple Q&A) go to an 8B model ($).
- Tier 2 (Frontier): 9% of queries (complex reasoning) go to a 405B/Claude Sonnet 4.6 / GPT-5.5 / Gemini 3.1 Pro tier model ($$).
- Tier 3 (Reasoning): 1% of queries (expert-level) go to thinking models like Claude Opus 4.7 or GPT-5.5 with extended thinking ($$$).
Net result: 80% cost reduction vs. sending all traffic to Tier 2.
Small Language Models (SLMs) for Production
3B-8B models (Llama 4 8B, Gemini 3.1 Flash, Claude Haiku 4.5) now match or beat the original GPT-4 from 2023 on most benchmarks.
- Use Case: Entity extraction, sentiment analysis, simple RAG.
- Cost: 100x cheaper to run than frontier models.
- Latency: < 100ms response times.
The DeepSeek V4 Floor
DeepSeek V4 Flash (released April 24, 2026) reset the floor for cheap frontier-class inference at 0.14/0.28 per 1M tokens with a 1M context window and cache-hit input at 0.0028/M.DeepSeekV4Proisroughly10xcheaperthanClaudeOpus4.7(0.435 / 0.87vs5 / $25 per 1M) after the 75% discount was made permanent on May 22, 2026. For cache-heavy, high-volume workloads where the prefix is reused often (RAG with shared knowledge bases, batch classification, codebase agents), V4 Flash or V4 Pro is now the dominant cost-optimization lever before you even start cascading. Verify on the DeepSeek pricing page before committing.
Spot Instance Strategies
For non-real-time workloads (batch processing, data extraction), use GPU Spot Instances (AWS Spot, Azure Spot, Lambda Labs).
- Risk: GPU can be reclaimed with 30-sec notice.
- Mitigation: Live KV-Cache Migration. Serving frameworks can stream the KV cache of ongoing requests to another node as soon as the "Reclamation Signal" is received, ensuring no work is lost.
The "Token Tax" Optimization
- System Prompt Caching: Hard-code common prefixes to get 90% discounts.
- Output Truncation: Strictly limit
max_tokens.
- Negative Prompting: "Don't be wordy" saves ~15% in output tokens (and thus cost).
Interview Questions
Q: How do you justify the cost of an AI system to a CFO?
Strong answer:
I focus on the ROI of Efficiency. First, I implement "Model Cascading" to ensure that 90% of our traffic is handled by sub-cent-per-million-token models. Second, I implement "Semantic Caching" to prevent paying for the same answer twice. Third, I set up "Inference Quotas" and "Chargeback Models" so each business unit is accountable for their usage. By treating AI as a "Commodity Resource" with tiered pricing, we can transition from "unbounded experimentation" to a "predictable OpEx" model.
Q: When is a self-hosted individual GPU cluster cheaper than an API?
Strong answer:
The "Crossover Point" usually happens at constant high throughput. If your application has a baseline of 5-10 requests per second, 24/7, the fixed cost of an H100 reservation becomes cheaper than the variable token cost of an API. However, if your traffic is "spiky" or heavily weighted toward business hours, API providers are usually cheaper because they allow you to "pay for the silence" during off-peak hours. For most enterprises, the break-even is around 500 million tokens per month for a 70B-tier model.
References
- Google Cloud. "Cost Optimization for Generative AI" (2024)
- Anyscale. "LLM Inference: API vs. Self-Hosted Costs" (2024)
Next: Diffusion Language Models