How to Scale to 1 Million Users: A Practical Guide

The most common mistake engineers make is reaching for distributed systems before they have exhausted what a single, well-tuned server can do. Before you add a second server, a cache, or a message queue — measure. If you don't measure, you don't know what's actually slow.

What to Measure

Before you touch anything, set up basic observability. You need to know three numbers at all times:

• Server CPU % — if this is consistently above 80%, your application code or database queries are doing too much computation
• Database connection count — each connection to your database consumes memory on both sides; running out of connections is one of the most common causes of outages
• P99 Latency — the 99th percentile response time. Your average latency can look fine at 80ms while 1% of requests are timing out at 10 seconds. Those users exist and they're frustrated.

Add Database Indexes

A database without proper indexes is like a book without a table of contents. Every query does a full table scan — it reads every single row to find the ones you asked for. On a table with 10 million rows, that's 10 million comparisons for a single request.

When you add an index to a column, the database builds a separate data structure (usually a B+ Tree) that stores the column values in sorted order, with pointers back to the full rows. Instead of reading every row, the database can now do a binary search — jumping directly to the range you need. The same query that scanned 10 million rows now touches perhaps 200.

The rule: any column you filter by (WHERE), sort by (ORDER BY), or join on should have an index. Start there before doing anything else.

Use a Connection Pooler

Every time your application server opens a database connection, the database has to do meaningful work: authenticate the request, allocate memory, set up the session. This takes 20–100ms. Opening a new connection for every request is like hiring a new employee, training them, and then firing them at the end of every task.

A connection pooler like PgBouncer (for PostgreSQL) maintains a fixed pool of long-lived connections. Your application asks for a connection, uses it, and returns it to the pool. The actual database connection never closes. This is especially critical in horizontally scaled setups — if you run 10 application servers and each has 100 connection threads, you're asking the database to maintain 1,000 simultaneous connections, which can exhaust its memory before any query runs.

Vertical scaling eventually hits a ceiling. You can't buy an infinitely large server. At some point — when your single server's CPU is pegged at 90% even after indexing and tuning — you need to run your application on multiple servers simultaneously. This is called horizontal scaling, and it introduces your first real systems problem: state.

Stateless Servers

For multiple servers to work interchangeably, none of them can hold data that belongs to a specific user. If Server A stores a user's session in its local memory and the next request goes to Server B, Server B has no idea who that user is.

The solution is to make every server stateless. Session data, user authentication tokens, and temporary state all live in a shared external store — typically Redis. Any server can pick up any request because the data it needs is always in the same place, not tied to a specific machine.

Stateless servers also give you something valuable: if one crashes, users are rerouted to another with no data loss. The system degrades gracefully instead of completely failing.

The Load Balancer

Once you have multiple servers, you need something sitting in front of them to distribute incoming requests. You can't give users two different IP addresses and ask them to pick one. That's what a load balancer does.

All traffic hits the load balancer at a single public address. It decides — based on an algorithm — which server to send each request to, and it continuously monitors server health via heartbeat checks. If a server stops responding, the load balancer removes it from rotation immediately. Users never see the failure.

  Users
    │
    ▼
[Load Balancer]  ←── health checks every 5s
  /    |    \
 ▼     ▼     ▼
[A]   [B]   [C]   ← stateless app servers
  \    |    /
    ▼  ▼  ▼
  [Shared Redis]  ← sessions, cache
       │
       ▼
  [Primary DB]

The most common routing strategies are:

• Round Robin — requests cycle through servers in order: 1, 2, 3, 1, 2, 3. Simple and works well when all servers are identical.
• Least Connections — the request goes to whichever server has the fewest active connections right now. Better when requests have wildly different processing times.
• Weighted Round Robin — you assign weights to servers so a larger machine gets proportionally more traffic. Useful when your servers aren't identical.

Web servers scale horizontally well. Databases do not — at least not as easily. Once your app servers are distributed, every single one of them connects to the same database. The database becomes the bottleneck. It doesn't matter how many application servers you run; they all serialize through one database server.

The first thing to understand is what kind of load your database is under. Most web applications are read-heavy — they read far more data than they write. A social feed, a product catalog, a search page — all of these generate tens or hundreds of reads for every write. If that's your pattern, you have two main tools available before you need to do anything radical.

Read Replicas

A read replica is a copy of your database that gets all writes forwarded to it automatically from the primary, and serves read queries independently. Your application sends all INSERT,UPDATE, and DELETE operations to the primary. All SELECT queries go to one of the replicas.

  App Servers
      │
   ┌──┴──────────────┐
   │                 │
   ▼ (writes)        ▼ (reads)
[Primary DB] ──► [Replica 1]
               ──► [Replica 2]
               ──► [Replica 3]

Primary replicates to all replicas asynchronously

This setup buys you significant headroom. You can add more replicas as read traffic grows. The primary only handles writes, which are typically a small fraction of total database load.

But this comes with a trade-off you must understand: replication lag. Replication is asynchronous by default, which means a write to the primary doesn't instantly appear on the replica. There's a delay — usually milliseconds, but it exists. A user who writes data and immediately reads it back might read from a replica that hasn't caught up yet and see stale data.

Caching with Redis

For data that is read extremely frequently but changes infrequently, a cache is far more effective than adding more replicas. Instead of hitting the database at all, the application checks Redis first.

Redis stores data in RAM — not on disk. Reading from RAM is orders of magnitude faster than reading from a database that has to hit disk. A query that takes 40ms from PostgreSQL often takes under 1ms from Redis. That's a 40× improvement in the hottest paths of your application.

The most common pattern is called cache-aside (or lazy loading):

1. Application receives a request for some data
2. Application checks Redis for that data using a key
3. If the data is there (cache hit) → return it immediately
4. If the data is not there (cache miss) → query the database, store the result in Redis with a TTL, return it

The TTL (Time To Live) determines how long the cached value is valid before Redis automatically deletes it. Once deleted, the next request fetches fresh data from the database and the cycle repeats.

The Thundering Herd Problem

There's a subtle failure mode that catches many teams off guard. When a very popular cache key expires, and hundreds of requests arrive simultaneously, all of them see a cache miss at the exact same moment. All of them then fire off identical database queries at the same time. This is called a cache stampede, and it can bring a database to its knees.

The standard mitigation is request coalescing: the first request that sees a cache miss takes a lock and fetches from the database. All other concurrent requests for the same key wait for that lock to be released and then read the newly populated cache. Only one database query fires, regardless of how many requests arrived simultaneously.

Not everything a user triggers needs to happen before you respond to them. When a user uploads a video, they don't need to wait for transcoding to finish. When they place an order, they don't need the email confirmation to be sent before they see the confirmation page. When they export a PDF report, they certainly don't need to watch a loading spinner for three minutes.

All of these are examples of work that can be done asynchronously — put in a queue, processed later by a background worker, and reported back to the user when done.

How a Message Queue Works

A message queue is a buffer that sits between the part of your system that receives work (producers) and the part that does the work (consumers). The producer adds a message to the queue and immediately returns an acknowledgement to the user — usually HTTP 202 Accepted, meaning "we got it, we'll handle it." The consumer picks up the message at its own pace and does the actual work.

User Request
     │
     ▼
[API Server] ──► [Message Queue] ──► [Worker 1]
     │               │              [Worker 2]
     ▼               │              [Worker 3]
HTTP 202 Accepted    │
(instant response)   └── workers process at their own pace

This pattern gives you two things:

• Burst absorption — if 50,000 users upload videos simultaneously, the queue accepts all 50,000 immediately. Workers process them at whatever rate they can handle without crashing. Without a queue, all 50,000 requests would hit your processing servers at once.
• Independent scaling — you can scale the number of workers up or down based on queue depth, completely independently of your API servers.

Dead Letter Queues

What happens when a worker crashes halfway through processing a message? Or when the message contains data that causes a bug every single time a worker tries to process it? Without handling this, the message gets retried forever, blocking other messages behind it. This is called a poison pill message.

The standard pattern is to configure a Dead Letter Queue (DLQ). After a message fails N times (typically 3–5), the queue system automatically moves it to the DLQ instead of retrying. Your engineers get alerted, can inspect the failed message, fix the underlying bug, and replay it. The DLQ prevents one bad message from blocking your entire processing pipeline.

Up to this point, you've been scaling your application tier horizontally and your database with read replicas and caching. All writes still go to a single primary database server. Eventually — and for most systems this point comes much later than engineers expect — that single primary becomes the bottleneck. Either the data volume grows beyond what fits on one machine's disk, or write throughput exceeds what one server can handle.

The solution is sharding: splitting your data across multiple independent database servers, each responsible for a portion of the total dataset. Each server is called a shard, and each shard is a full, independent database — not a replica. Shards don't mirror each other; they hold different data.

All users data:
┌──────────────────────────────┐
│ user_id 1 – 10,000,000       │ ← Shard 1 (DB Server A)
├──────────────────────────────┤
│ user_id 10,000,001 – 20M     │ ← Shard 2 (DB Server B)
├──────────────────────────────┤
│ user_id 20,000,001 – 30M     │ ← Shard 3 (DB Server C)
└──────────────────────────────┘

Your application code decides which shard to query.
The database servers have no awareness of each other.

Sharding Strategies

How you decide which data goes to which shard is called your sharding keystrategy. The choice matters enormously.

• Range-based sharding — user IDs 1–1M go to Shard 1, 1M–2M to Shard 2, and so on. Simple to understand, easy to add new shards at the top of the range. The problem: early users are almost always more active than new users, so Shard 1 receives disproportionately more traffic than Shard 3.
• Hash-based sharding — apply a hash function to the user ID and use the result to determine the shard: shard = hash(user_id) % number_of_shards. This distributes data much more evenly. The problem: when you add a new shard, the modulo changes and nearly every piece of data now belongs on a different shard. You have to migrate the entire dataset.
• Geographic sharding — data for users in North America goes to servers in the US, European users to servers in Frankfurt, etc. This minimizes latency for users and helps with data residency laws (GDPR requires EU user data to stay in the EU). The risk: some regions grow much faster than others, creating hot shards.

Why Sharding is a Nightmare

Sharding is one of the most complex things you can do to a production database. The list of problems it introduces is long:

• No cross-shard JOINs. If a user's profile is on Shard 1 and their orders are on Shard 3, you cannot join these tables with a SQL query. You have to fetch from both shards in your application code and join the data in memory. This is slower, more complex, and forces you to rethink your entire data access layer.
• Application-level routing. Your application code must know which shard to query for every single database operation. This logic gets embedded throughout your codebase and becomes an ongoing maintenance burden.
• The hot shard problem. If one enterprise customer generates 80% of your write traffic and they're assigned to Shard 2, Shard 2 runs hot while all other shards sit idle. Hash-based sharding helps, but doesn't eliminate this with highly skewed data.
• Rebalancing is painful. Adding a new shard after launch means migrating terabytes of live data while the system stays up. This requires careful coordination, double-writes during migration, and extensive testing to avoid data loss.
• Transactions don't span shards. If you need to atomically update data that lives on different shards, standard SQL transactions don't work. You need distributed transaction protocols (like two-phase commit), which are slow and complex.