Your Architecture at 100 Users Will Not Survive 10,000
Accept this early. The single Postgres instance that hums along for your first hundred users will buckle under ten thousand.
This isn’t a failure of planning. It’s the natural lifecycle of every SaaS platform. The question isn’t whether you’ll re-architect, but when.
If you haven’t read our complete multi-tenant SaaS guide, start there. This post assumes you have a working multi-tenant platform and need to scale it.
Stage 1: Optimize Before You Add Hardware
The cheapest scaling strategy is making existing infrastructure work harder. Most SaaS platforms have 3-5x headroom hiding in unoptimized queries.
Fix the database first
N+1 queries are the silent killer. Your ORM fetches a list of tenants, then fires a separate query for each tenant’s data. That’s 101 queries instead of 2.
Add indexes on tenant_id plus your most common filter columns. We’ve seen a single composite index cut page load from 4 seconds to 200 milliseconds. Not exaggerating.
Analyze your slow query log weekly. PostgreSQL’s pg_stat_statements extension shows exactly which queries consume the most time. Fix the top five, repeat.
Connection pooling
Each database connection costs memory. Without pooling, every concurrent request opens a new connection. At 500 concurrent users, your database chokes.
PgBouncer sits between your app and the database. It maintains a pool of connections and reuses them across requests. The difference is night and day.
Caching the obvious stuff
Redis as a read cache for frequently accessed data. Tenant configs, feature flags, user sessions, permission sets. These rarely change but get read on every request.
Tenant-aware cache keys are critical. dashboard:{tenant_id} works. Just dashboard doesn’t. Forget the tenant ID in your cache key and you’ll serve one tenant’s data to another.
Cache invalidation on webhook events. When a tenant’s plan changes, invalidate their cached data immediately. Stale billing data is a support ticket waiting to happen.
Stage 2: Go Horizontal
When vertical scaling (bigger server) runs out of runway, go horizontal (more servers).
Stateless APIs first
If your server stores session state in memory, horizontal scaling breaks. Request 1 hits server A and creates a session. Request 2 hits server B and the session is gone.
Move all state to external stores. Sessions to Redis. File uploads to S3. Background jobs to a queue. Your API servers should be disposable.
Load balancing
Put a load balancer in front of your API layer. Route based on latency, not round-robin. Health checks every 10 seconds.
Unhealthy instances get removed from the pool automatically. For WebSocket connections, use sticky sessions or a dedicated gateway.
Auto-scaling
Scale on request latency and queue depth, not just CPU. A server at 40% CPU with a 5-second request queue is overloaded. A server at 80% CPU with instant responses is fine.
Reserved instances for base load save 30-70% compared to on-demand. Auto-scaling groups handle spikes. Budget for 2x your average load as burst capacity.
Stage 3: Tame the Noisy Neighbor
At scale, tenant diversity becomes your biggest challenge. Tenant A runs a 2-minute report. Tenant B’s dashboard crawls because they share the same database.
Sound familiar?
Per-tenant rate limits
Set request limits per tenant. A tier-1 tenant gets 100 requests/second. Free tier gets 10. Enforce at the API gateway.
But rate limiting alone isn’t enough. A single expensive query within the rate limit can still monopolize database resources.
Queue heavy work
Reports, exports, bulk imports: anything over a few seconds runs asynchronously. Queue it, process in the background, notify when done.
Dedicate separate worker pools for heavy operations. Your real-time API and your batch processing should never compete for the same resources.
Resource quotas
Storage quotas per tenant. API call quotas per integration. Query timeout limits per request.
The nuclear option: a tenant-level kill switch. Throttle or disable one tenant without affecting others. Build it before you need it.
Stage 4: Scale the Database
When optimizations and caching aren’t enough, scale the database itself.
Read replicas
Route read queries to replicas, writes to the primary. Most SaaS workloads are 80-90% reads. Replicas handle that load while the primary handles writes.
Replication lag is usually under a second. For tenant writes that must be immediately visible, route those reads to the primary.
Sharding
Shard by tenant ID. Each shard holds a subset of tenants. When a shard gets hot, split it.
Sharding adds significant complexity. Cross-shard queries don’t work. Your application needs a routing layer that maps tenant IDs to shards. Don’t shard until you absolutely must.
For how data isolation patterns affect scaling, see multi-tenancy patterns: shared vs. database-per-tenant.
Monitoring at Scale
System averages hide per-tenant problems. Include tenant_id in every log line, every metric, every trace.
When tenant 300 reports “it’s slow,” you need their p95 latency. Not the system average.
Alert on per-tenant anomalies. API call volume dropping 50% for a tenant? They’re about to churn. Storage growing 10x overnight? Something broke.
Track cost per tenant. Your top 5% of tenants by usage are probably consuming 40-60% of your infrastructure. If they’re not paying accordingly, that’s a pricing problem.
See SaaS metrics that matter for how to track this.
Infrastructure as Code
At 100 users, you can set up infrastructure by clicking around in the AWS console. At 10,000, that’s a recipe for inconsistency and untracked changes.
Define your infrastructure in Terraform, Pulumi, or CloudFormation. Every server, every database, every load balancer described in code. Reviewed in pull requests. Versioned in git.
This solves the “it works in staging but not production” problem. Your staging environment is defined by the same code as production. Same configs, same architecture, different scale.
One team we worked with had three months of infrastructure drift between staging and production. Deployments would work in staging and fail in production because someone had manually changed a security group. Terraform fixed it in a week.
When to Rearchitect vs. Optimize
Not every performance problem needs new architecture. Most need better code.
If response times are slow but CPU is low, you have a query problem. Optimize queries. If CPU is maxed but response times are fine per-request, you need horizontal scaling. More servers.
If individual tenants cause problems for others, you have an isolation problem. Rate limits, queue isolation, or data isolation changes.
The expensive mistake is rearchitecting when optimization would have been enough. The other expensive mistake is optimizing when the architecture fundamentally doesn’t support your scale.
A good rule of thumb: if you can get 3x more headroom from optimization, optimize. If you need 10x, rearchitect.
The Scaling Checklist
Database queries are indexed and optimized. Connection pooling is in place. Caching handles hot data with tenant-aware keys.
API servers are stateless and horizontally scalable. Rate limiting is per-tenant. Heavy operations run asynchronously.
Monitoring includes tenant-level metrics and anomaly alerting. Skip any of these and you’ll discover the gap in production. Probably on a Friday evening.
Scaling your SaaS platform and hitting performance walls? Let’s diagnose it together. We’ve scaled multi-tenant platforms from early-stage to thousands of users and know where the bottlenecks hide.
