Case Study: Multi-Tenant AI SaaS Platform¶

The Problem¶

A B2B startup is building an AI-powered document analysis platform where each customer uploads their own contracts, and the AI answers questions about them. Customers include competitors who must never see each other's data.

Constraints given in the interview: - 500 enterprise customers, each with 10,000-100,000 documents - Absolute data isolation: Customer A's data cannot leak to Customer B - Shared infrastructure for cost efficiency - Compliance: SOC 2 Type II, GDPR - Query latency under 2 seconds

The Interview Question¶

"Design a multi-tenant RAG system where Coca-Cola and Pepsi can both be customers, and there is zero risk of cross-tenant data leakage."

Solution Architecture¶

flowchart TB
    subgraph Gateway["API Gateway Layer"]
        REQ[User Request] --> AUTH[Auth Service]
        AUTH --> TENANT_ID[Extract Tenant ID]
        TENANT_ID --> VALIDATE[Validate Tenant Access]
    end

    subgraph Isolation["Tenant Isolation Layer"]
        VALIDATE --> ROUTER{Isolation Strategy}
        ROUTER -->|Small Tenant| SHARED[(Shared Vector DB<br/>Namespace Isolation)]
        ROUTER -->|Large Tenant| DEDICATED[(Dedicated Pod<br/>Physical Isolation)]
    end

    subgraph Query["Query Execution"]
        SHARED --> FILTER[Tenant Filter<br/>WHERE tenant_id = X]
        DEDICATED --> DIRECT[Direct Query]
        FILTER --> LLM[LLM with Tenant Context]
        DIRECT --> LLM
    end

    subgraph Audit["Audit Layer"]
        LLM --> RESPONSE[Response]
        LLM --> LOG[(Audit Log<br/>Immutable)]
    end

Key Design Decisions¶

1. Hybrid Isolation: Namespace vs Physical¶

Answer: Pure physical isolation (one database per tenant) is expensive. Pure namespace isolation (shared database with tenant_id filter) has leakage risk if a filter bug occurs. We use a tiered approach:

Tier	Tenant Size	Isolation Method	Why
Standard	<50K docs	Namespace in shared Qdrant	Cost efficient
Premium	50K-500K docs	Dedicated Qdrant collection	Performance isolation
Enterprise	>500K docs	Dedicated Qdrant pod	Physical + regulatory

2. Defense in Depth for Data Isolation¶

Answer: We never trust a single layer. Our isolation stack:

API Gateway: Validates tenant_id from JWT, rejects cross-tenant requests
Database Layer: Row-level security (RLS) enforces tenant_id filter at DB level
Application Layer: ORM wrapper automatically injects tenant filter
LLM Layer: System prompt explicitly states "You are answering for Tenant X only"
Output Layer: Post-generation filter scans for any document IDs not belonging to tenant

3. Why Not One Vector DB Per Tenant?¶

Answer: 500 tenants × $100/month per managed instance = $50K/month just for databases. By using namespace isolation for 80% of tenants, we reduce this to $8K/month. The remaining 20% on dedicated infrastructure pay a premium tier price.

The Data Ingestion Pipeline¶

flowchart LR
    subgraph Upload["Document Upload"]
        DOC[Customer Document] --> VALIDATE[Validate Format]
        VALIDATE --> TAG[Tag with Tenant ID]
    end

    subgraph Process["Processing Pipeline"]
        TAG --> PARSE[Parse Document]
        PARSE --> CHUNK[Chunk + Embed]
        CHUNK --> ENCRYPT[Encrypt Metadata]
    end

    subgraph Store["Storage"]
        ENCRYPT --> VECTOR[(Vector DB)]
        ENCRYPT --> BLOB[(Blob Storage)]
        ENCRYPT --> AUDIT[(Audit Log)]
    end

Critical: The tenant_id is attached at the earliest possible point (upload validation) and travels with the document through every stage. It is not derived or looked up later.

Handling the Compliance Requirements¶

SOC 2 Type II¶

Control	Implementation
Access logging	Every query logged with tenant_id, user_id, timestamp
Encryption at rest	AES-256 for blob storage, database-native for vector DB
Encryption in transit	TLS 1.3 everywhere
Access reviews	Automated quarterly reports from audit logs

async def delete_tenant_data(tenant_id: str):
    # 1. Delete from vector DB
    await vector_db.delete(filter={"tenant_id": tenant_id})

    # 2. Delete from blob storage
    await blob_storage.delete_prefix(f"tenants/{tenant_id}/")

    # 3. Anonymize audit logs (cannot delete for compliance)
    await audit_log.anonymize(tenant_id=tenant_id)

    # 4. Generate deletion certificate
    return generate_deletion_certificate(tenant_id)

Cost Analysis (500 Tenants)¶

Component	Monthly Cost
Shared Vector DB (Qdrant Cloud)	$2,500
Dedicated pods (20 enterprise tenants)	$4,000
LLM costs (pooled, GPT-4o-mini)	$8,000
Blob storage (S3)	$1,500
Audit logging (CloudWatch)	$500
Total	$16,500/month
Per tenant average	$33/month

Interview Follow-Up Questions¶

Q: What if a bug in your ORM bypasses the tenant filter?

A: Defense in depth. Even if the ORM fails, the database enforces RLS (Row-Level Security). The query SELECT * FROM documents internally becomes SELECT * FROM documents WHERE tenant_id = current_tenant(). This is enforced at the Postgres level, not the application level.

Q: How do you handle a tenant who wants to export all their data?

A: We provide a data portability API that streams all documents with their embeddings and metadata. The export is triggered by an admin, logged in the audit trail, and delivered to a customer-controlled S3 bucket (not our infrastructure).

Q: What if the LLM hallucinates information from its training data that matches a competitor's confidential info?

A: This is a real risk. We mitigate by: (1) Using only retrieval-grounded generation (the LLM cannot answer without retrieved docs). (2) Filtering outputs for any content that does not trace back to the tenant's uploaded documents. (3) Offering a "private model" tier where we fine-tune a tenant-specific model on their data only.

Key Takeaways for Interviews¶

Multi-tenancy is about layers: never rely on a single isolation mechanism
Tiered isolation balances cost and security: not all tenants need dedicated infrastructure
Tenant ID must be immutable and early: tag at upload, not at query time
Compliance is an architecture concern: design for audit, deletion, and portability from day one

Related chapters: LLM Security, Access Control & Multi-Tenant Isolation