How Discord Stores Trillions of Messages

Discord shares their journey of migrating from Cassandra to ScyllaDB to store trillions of messages. Initially, Discord used Cassandra, but as their data grew to 177 nodes and trillions of messages, they faced significant operational toil. Issues included unpredictable latency due to "hot partitions" (where high-traffic channels overwhelmed specific nodes), garbage collection (GC) pauses causing latency spikes, and expensive maintenance operations like compaction.

To solve these problems, Discord decided to migrate to ScyllaDB, a Cassandra-compatible database written in C++. ScyllaDB offered key advantages: it is written in C++ (avoiding Java GC pauses), uses a shard-per-core architecture for better workload isolation, and promises faster repairs. The migration reduced their cluster size from 177 Cassandra nodes to just 72 ScyllaDB nodes, significantly improved tail latencies (p99 read latency dropped from 40-125ms to 15ms), and reduced maintenance overhead.

In addition to the database switch, Discord architected a protection layer upstream. They built "data services" using Rust and the Tokio ecosystem to sit between their API monolith and the database. The primary feature of these services is request coalescing: if multiple users request the same message row simultaneously, the service queries the database once and returns the result to all subscribers. This middleware, combined with consistent hash-based routing, shields the database from traffic spikes on hot partitions.

Key Concepts

• Hot Partitions: A performance pitfall in distributed databases where specific data partitions receive disproportionate traffic (e.g., a large Discord server), causing latency and node overload.
• Request Coalescing: A pattern where multiple concurrent requests for the same resource are grouped into a single backend request to reduce load. Discord implemented this in intermediary Rust services.
• ScyllaDB vs. Cassandra: ScyllaDB is a C++ rewrite of Cassandra that eliminates Java Garbage Collection pauses and uses a thread-per-core model for better performance and lower latency.
• Super-disk Topology: A storage strategy using Local SSDs for high-speed access combined with RAID mirroring to persistent disks for durability, offering the best of both worlds.
• Custom Data Migrator: Instead of using slow, standard tools (like Spark), Discord wrote a custom high-performance data migrator in Rust to move trillions of records in days rather than months.