The Problem
A B2B startup is building an AI-powered document analysis platform where each customer uploads their own contracts, and the AI answers questions about them. Customers include competitors who must never see each other's data.
Constraints given in the interview:
- 500 enterprise customers, each with 10,000-100,000 documents
- Absolute data isolation: Customer A's data cannot leak to Customer B
- Shared infrastructure for cost efficiency
- Compliance: SOC 2 Type II, GDPR
- Query latency under 2 seconds
The Interview Question
"Design a multi-tenant RAG system where Coca-Cola and Pepsi can both be customers, and there is zero risk of cross-tenant data leakage."
Solution Architecture
Diagram
Key Design Decisions
1. Hybrid Isolation: Namespace vs Physical
Answer: Pure physical isolation (one database per tenant) is expensive. Pure namespace isolation (shared database with tenant_id filter) has leakage risk if a filter bug occurs. We use a tiered approach:
| Tier | Tenant Size | Isolation Method | Why |
|---|
| Standard | <50K docs | Namespace in shared Qdrant | Cost efficient |
| Premium | 50K-500K docs | Dedicated Qdrant collection | Performance isolation |
| Enterprise | >500K docs | Dedicated Qdrant pod | Physical + regulatory |
2. Defense in Depth for Data Isolation
Answer: We never trust a single layer. Our isolation stack:
- API Gateway: Validates tenant_id from JWT, rejects cross-tenant requests
- Database Layer: Row-level security (RLS) enforces tenant_id filter at DB level
- Application Layer: ORM wrapper automatically injects tenant filter
- LLM Layer: System prompt explicitly states "You are answering for Tenant X only"
- Output Layer: Post-generation filter scans for any document IDs not belonging to tenant
3. Why Not One Vector DB Per Tenant?
Answer: 500 tenants × 100/monthpermanagedinstance=50K/month just for databases. By using namespace isolation for 80% of tenants, we reduce this to $8K/month. The remaining 20% on dedicated infrastructure pay a premium tier price.
The Data Ingestion Pipeline
Diagram
Critical: The tenant_id is attached at the earliest possible point (upload validation) and travels with the document through every stage. It is not derived or looked up later.
Handling the Compliance Requirements
SOC 2 Type II
| Control | Implementation |
|---|
| Access logging | Every query logged with tenant_id, user_id, timestamp |
| Encryption at rest | AES-256 for blob storage, database-native for vector DB |
| Encryption in transit | TLS 1.3 everywhere |
| Access reviews | Automated quarterly reports from audit logs |
GDPR Right to Deletion
async def delete_tenant_data(tenant_id: str):
# 1. Delete from vector DB
await vector_db.delete(filter={"tenant_id": tenant_id})
# 2. Delete from blob storage
await blob_storage.delete_prefix(f"tenants/{tenant_id}/")
# 3. Anonymize audit logs (cannot delete for compliance)
await audit_log.anonymize(tenant_id=tenant_id)
# 4. Generate deletion certificate
return generate_deletion_certificate(tenant_id)
Cost Analysis (500 Tenants)
| Component | Monthly Cost |
|---|
| Shared Vector DB (Qdrant Cloud) | $2,500 |
| Dedicated pods (20 enterprise tenants) | $4,000 |
| LLM costs (pooled, GPT-4o-mini) | $8,000 |
| Blob storage (S3) | $1,500 |
| Audit logging (CloudWatch) | $500 |
| Total | $16,500/month |
| Per tenant average | $33/month |
Interview Follow-Up Questions
Q: What if a bug in your ORM bypasses the tenant filter?
A: Defense in depth. Even if the ORM fails, the database enforces RLS (Row-Level Security). The query SELECT * FROM documents internally becomes SELECT * FROM documents WHERE tenant_id = current_tenant(). This is enforced at the Postgres level, not the application level.
Q: How do you handle a tenant who wants to export all their data?
A: We provide a data portability API that streams all documents with their embeddings and metadata. The export is triggered by an admin, logged in the audit trail, and delivered to a customer-controlled S3 bucket (not our infrastructure).
Q: What if the LLM hallucinates information from its training data that matches a competitor's confidential info?
A: This is a real risk. We mitigate by: (1) Using only retrieval-grounded generation (the LLM cannot answer without retrieved docs). (2) Filtering outputs for any content that does not trace back to the tenant's uploaded documents. (3) Offering a "private model" tier where we fine-tune a tenant-specific model on their data only.
Key Takeaways for Interviews
- Multi-tenancy is about layers: never rely on a single isolation mechanism
- Tiered isolation balances cost and security: not all tenants need dedicated infrastructure
- Tenant ID must be immutable and early: tag at upload, not at query time
- Compliance is an architecture concern: design for audit, deletion, and portability from day one
Related chapters: LLM Security, Access Control & Multi-Tenant Isolation