Scaling data / Intermediate / 8 min
Sharding: splitting data across servers
Hash mod N vs consistent hashing, keys that move, and why virtual nodes even out the load.
Start the simulation → Change the load, break a part, watch what happens. Free, no sign-up.
In short
- What it is
- Sharding splits data across machines by a key; consistent hashing places keys and servers on a ring so adding a server moves few keys.
- Why it matters
- One database can only hold and serve so much; sharding spreads both the data and the load.
- The trade-off
- Queries across shards are slow and joins are hard, and a badly chosen key creates a hot shard.
- Say it in an interview
- "I would shard by user id with consistent hashing and virtual nodes, so adding a node moves only about 1/N of the keys."
Check yourself
With hash(key) mod N, roughly what share of keys move when you go from 4 to 5 servers?
Answer: About 80%. Most remainders change when N changes, so about (N−1)/N of keys, here around 80%, land on a different server.
With consistent hashing, adding one server to N moves roughly how many keys?
Answer: About 1/N of them. The new server takes over part of one neighbour’s slice; everything else stays put.
Why add virtual nodes?
Answer: To spread each server over many small slices so the load evens out. One point per server gives random-sized slices. Many points per server average out.