How Discord Stores Billions of Messages

Discord launched on a single MongoDB replica set. Every message was a document in a single collection, indexed by channel_id and message_id. Reading message history for a channel was a range query: find all messages with this channel ID, sort by message ID descending, limit to 50.

This worked because MongoDB keeps frequently accessed data in memory via its WiredTiger storage engine. As long as the working set — the portion of data actively being read — fits in RAM, reads are fast. But Discord was growing rapidly, and the total dataset eventually exceeded the available memory on the replica set.

Once that happened, reads started hitting disk. MongoDB uses memory-mapped I/O, so when a page of data isn't in memory, the OS has to load it from disk — a page fault. Discord's read pattern was roughly 50/50 reads and writes with highly random access (users open different channels at unpredictable times), which is the worst case for page faults. Read latencies spiked from milliseconds to seconds. By late 2015, with around 100 million stored messages, the system was unusable at peak traffic.

Snowflake IDs

One design decision from this era carried forward through every subsequent migration: Discord adopted Snowflake IDs for messages. These are 64-bit integers where the most significant 42 bits encode a millisecond timestamp (relative to Discord's custom epoch of January 1, 2015). The remaining bits encode a worker ID, process ID, and a per-millisecond sequence counter.

Snowflake ID (64 bits):

 ┌──────────────────────────────────────┬──────┬──────┬────────────┐
 │  Timestamp (ms since Discord epoch)  │Worker│ PID  │  Sequence  │
 │              42 bits                 │ 5b   │ 5b   │   12 bits  │
 └──────────────────────────────────────┴──────┴──────┴────────────┘

 Sorting by Snowflake ID = sorting by time.
 No separate timestamp column or index needed.

Because the timestamp occupies the highest-order bits, sorting by Snowflake ID sorts by time automatically. This eliminates the need for a separate timestamp column or a secondary index on creation time. It also means the message ID itself encodes when the message was sent — you can extract the timestamp from any message ID with a single bit shift.

Discord needed a database that could partition data across many machines (horizontal scaling) while maintaining fast writes. They chose Apache Cassandra, a wide-column NoSQL database designed for exactly this workload: high write throughput distributed across a cluster of commodity nodes.

The data model

The key design decision was how to partition the data. In Cassandra, the partition key determines which node stores the data. All rows with the same partition key live on the same node and can be read in a single query without cross-node coordination.

Discord chose a composite partition key: (channel_id, bucket). Thebucket is a time window — roughly 10 days. Within each partition, messages are sorted by message_id in descending order (the clustering key), so reading the most recent messages is a sequential read from the beginning of the partition.

Cassandra schema:

  CREATE TABLE messages (
    channel_id  bigint,
    bucket      int,         -- time window (~10 days)
    message_id  bigint,      -- Snowflake ID (clustering key)
    author_id   bigint,
    content     text,
    PRIMARY KEY ((channel_id, bucket), message_id)
  ) WITH CLUSTERING ORDER BY (message_id DESC);

Query path:  "Give me the latest 50 messages in channel X"
  → partition key = (channel_id, current_bucket)
  → sequential read of first 50 rows (already sorted DESC)

Why bucketing matters

Without the bucket, the partition key would be just channel_id. A busy channel accumulating millions of messages over years would produce a single enormous partition. In Cassandra, large partitions cause problems: GC pressure increases, compaction takes longer, and read latency degrades because the database has to scan through more data structures to find the requested range.

By bucketing into 10-day windows, Discord capped partition size. Each partition held at most 10 days of messages for one channel — a bounded, predictable amount of data. Reading recent messages almost always hits a single partition. Scrolling further back crosses a bucket boundary, which requires querying the previous bucket's partition, but this is a predictable extra query rather than a scan of an unbounded dataset.

What went wrong

Cassandra handled Discord's write volume well. The problems showed up in operations and tail latency, and they got worse as the cluster grew. By 2022, the cluster was 177 nodes storing trillions of messages.

JVM garbage collection. Cassandra runs on the JVM. Under heavy load, garbage collection pauses would freeze a node for hundreds of milliseconds — sometimes long enough for the cluster to mark the node as down, triggering data rebalancing. Engineers had to manually intervene: restart the node, let it rejoin the cluster, wait for it to catch up. This happened frequently enough to generate significant on-call burden.

Compaction storms. Cassandra uses an LSM-tree storage engine. Writes go to an in-memory table that gets flushed to disk as sorted files (SSTables). Periodically, these files are merged — a process called compaction. On nodes with heavy write traffic, compaction would fall behind, leading to a growing backlog of SSTables. Reads became slower because the database had to check more files. Engineers developed a manual procedure they called the "gossip dance": take a node out of the cluster, let it compact in isolation, then reintroduce it.

Hot partitions. Despite the bucketing strategy, some channels generated extreme traffic. A popular public server with thousands of simultaneous users produces far more reads than a quiet private channel. These hot partitions saturated individual nodes, and because Cassandra routes by partition key, there was no way to spread that load without changing the data model.

Discord's solution had two parts: replace the database engine and add a service layer in front of it.

ScyllaDB: Cassandra without the JVM

ScyllaDB is a C++ reimplementation of Cassandra. It uses the same data model, the same query language (CQL), and the same partition/clustering key concepts. Discord migrated without changing their schema. The critical difference is architectural: ScyllaDB uses a shard-per-core model. Each CPU core owns a dedicated shard of data and runs its own event loop. There is no JVM, no garbage collector, and no stop-the-world pauses.

The shard-per-core model also improves workload isolation. A hot partition on one core doesn't affect the latency of queries hitting other cores on the same machine. In Cassandra, a single GC pause or compaction storm affected the entire node.

Rust data services: request coalescing

Replacing the database engine solved the GC and compaction problems, but it didn't solve hot partitions. When a popular channel gets opened by thousands of users simultaneously, thousands of identical queries hit the database at the same moment — all asking for the same 50 most recent messages in the same channel.

Discord built a Rust-based service that sits between the application layer and ScyllaDB. When multiple concurrent requests arrive for the same channel, the service coalesces them: it sends one query to the database and fans out the result to all waiting callers. This is conceptually similar to request coalescing in CDNs or the singleflight pattern in Go.

Request flow with data services layer:

 User A ──┐
 User B ──┤                                    ┌──────────┐
 User C ──┼── [Rust Data Service] ──(1 query)──►│ ScyllaDB │
 User D ──┤    coalesces concurrent             └──────────┘
 User E ──┘    reads for same channel                │
               ◄──────────(1 response)───────────────┘
               fans result to all 5 callers

Without coalescing, 5,000 users opening the same channel generates 5,000 identical database queries. With coalescing, it generates one.

The results

After the migration, Discord reduced the cluster from 177 Cassandra nodes to 72 ScyllaDB nodes. The P99 read latency dropped from a variable 40–125ms range to a consistent 15ms. On-call incidents related to GC pauses and compaction storms effectively disappeared.