Case Study: Real-Time AI Search Engine
The Problem
A fintech startup needs to build a real-time market intelligence platform that lets analysts ask natural language questions about live market data, news, and company filings.
Constraints given in the interview:
- Data freshness: queries must reflect information from the last 5 minutes
- Scale: 10,000 concurrent users, 50,000 queries/hour
- Accuracy: financial data cannot be hallucinated
- Latency: p95 response time under 3 seconds
The Interview Question
"Design a system that lets users ask 'What is the sentiment around Tesla in the last hour?' and get an accurate, sourced answer in under 3 seconds."
Solution Architecture
Diagram
Key Design Decisions
1. Why Kafka for Ingestion?
The interviewer wants to know you understand streaming vs batch.
Answer: Kafka provides exactly-once delivery and allows multiple consumers. We have one consumer writing to the vector DB and another to Elasticsearch. If the vector indexing falls behind, the full-text index still serves queries. This is the dual-write pattern for resilience.
2. Why Hybrid Search (Vector + Full-Text)?
Answer: Financial queries mix semantic ("sentiment around Tesla") with keyword ("TSLA 10-K filing"). Pure vector search would miss exact ticker matches. We use Reciprocal Rank Fusion (RRF) to combine results.
3. Why GPT-4o-mini Instead of GPT-4o?
Answer: For a 3-second p95 latency target at 50K queries/hour, we need fast generation. GPT-4o-mini gives us 100+ tokens/second vs 40 tokens/second for GPT-4o. The reranker handles accuracy; the LLM only synthesizes already-verified content.
Handling the Freshness Requirement
The hardest part of this problem is ensuring the index reflects data from the last 5 minutes.
Solution: TTL-Based Indexing
# Each document gets a timestamp field
doc = {
"content": "Tesla announces new factory...",
"timestamp": datetime.now(UTC),
"source": "Reuters",
"ttl_hours": 24 # Auto-delete after 24 hours
}
# Query filters to last N minutes
def search_recent(query: str, minutes: int = 60):
cutoff = datetime.now(UTC) - timedelta(minutes=minutes)
return vector_db.search(
query=query,
filter={"timestamp": {"$gte": cutoff}}
)
Cost Analysis
| Component | Monthly Cost (at 50K queries/hour) |
|---|
| Kafka (MSK) | $2,500 |
| Qdrant (managed) | $1,800 |
| Elasticsearch | $2,000 |
| GPT-4o-mini (generation) | $3,500 |
| Cross-encoder reranking | $800 |
| Total | $10,600/month |
Interview Follow-Up Questions
Q: How do you prevent hallucinated financial data?
A: Three layers: (1) The LLM only summarizes retrieved content, never generates facts. (2) Every claim must cite a source document. (3) A post-generation validator checks that any number in the response exists verbatim in a source.
Q: What if Kafka falls behind during a news spike?
A: We implement backpressure with consumer lag monitoring. If lag exceeds 2 minutes, we shed load on the ingestion side using sampling. Real-time queries hit a "recent" index with only the last hour of data; batch jobs backfill the full index.
Key Takeaways for Interviews
- Real-time AI search requires streaming infrastructure, not batch ETL
- Hybrid search (semantic + keyword) outperforms pure vector for structured domains
- Latency budgets drive model selection: use fast models for synthesis, save expensive models for reasoning
- Freshness is a filter, not a feature: implement at the index level, not the prompt level
Related chapters: Hybrid Search, Serving Infrastructure