Replication: What We've Learned

Part one opened with a deceptively small question: how do you make sure every replica ends up with the same data? Fourteen posts later it is worth going back to it, because the answer turned out to be less a technique than a series of trade-offs, and they are easier to see all at once.

Why Replicate At All

Part one gave three reasons. The series turned up a fourth.

High availability. Keep the system serving when a machine — or a whole datacenter — goes down. Single-leader replication delivers this only with a working failover story, and failover is where most of the sharp edges live.

Reduced latency. Put data geographically near the people using it. This is what pushes systems toward multiple leaders or leaderless clusters spanning datacenters, because a single leader means somebody is always paying for a long round trip.

Read throughput. Spread reads across replicas. The cheapest win of the four, and the one that quietly introduces replication lag as a user-visible problem the moment you start reading from followers.

Disconnected operation. Let an application keep working with no network at all, and reconcile later. Part one did not list this one — it only became visible with multi-leader replication, where an offline client is really just a leader that has been partitioned for a while. Calendar apps and collaborative editors are the everyday case.

The Three Approaches

Read left to right and the progression is one of trading away coordination. Single-leader keeps a single point of ordering, which is why conflicts cannot happen and why that node is both a bottleneck and a failure domain. Multi-leader gives up the single point and immediately inherits write conflicts and topology-dependent causality problems. Leaderless gives up designated writers entirely and gets quorums instead of ordering — along with the job of telling an overwrite from a conflict.

None of them is more advanced than the others. Single-leader is the default for good reasons, and most systems that reach for something else do so because of geography or offline requirements rather than scale.

The Trade-Off Underneath All of It

One thing recurs in every post, and it is worth naming directly.

Asynchrony is what makes replication useful, and it is the entire source of the difficulty.

If every write waited for every replica, replication would be simple and largely pointless — you would have the availability of the least available node and the latency of the slowest one. So real systems let replicas fall behind. Everything hard in this series comes out of that gap:

• Replication lag and the three consistency guarantees that patch over its symptoms.
• Write conflicts, which exist because two nodes accepted writes without consulting each other.
• The limits of quorums, where w + r > n turns out to be a probability rather than a promise.
• Version vectors and siblings, which exist to reconstruct an ordering the network destroyed.

Every guarantee in this series is bought with either latency or availability. Read-your-writes costs you a trip to the leader or a timestamp to track. Synchronous replication costs you a blocked write. A strict quorum costs you availability during a partition. There is no configuration that gives you all of it, and the useful skill is knowing which one you are spending.

The corollary is a practical one: do not assume your database gives you a guarantee because it feels like it should. Quorums look like they promise fresh reads and don’t. A failed write looks like it didn’t happen and may still be readable. Read the documentation for the system you actually run, and test the behaviour you are relying on.

What This Series Left Out

Replication is one chapter of a larger problem, and several things were deliberately deferred:

• Partitioning. This series assumed every replica holds the whole dataset. Once data is too big for one machine it gets split across nodes, and replication and partitioning have to be reasoned about together.
• Transactions. Conflict resolution here worked one row at a time. A transaction that atomically writes five rows is resolved as five independent conflicts, which is rarely what anyone wants.
• Consensus and linearizability. The guarantees quorums cannot provide come back as a much stronger — and more expensive — set of tools.
• Clocks. Last write wins leans entirely on comparing timestamps across machines, which deserves its own treatment rather than the warning it got here.

Each is its own subject, and each is a candidate for where this series goes next.

Key Takeaways

• Four reasons to replicate: high availability, reduced latency, read throughput, and disconnected operation — the last of which only becomes visible once multiple nodes accept writes.
• Single-leader has no conflicts by construction, and pays with failover complexity and one write path.
• Multi-leader and leaderless both trade away a single ordering point, and both inherit conflicts as the direct consequence.
• No approach is more advanced than another; geography and offline requirements drive the choice more often than scale does.
• Asynchrony is both the point and the problem. Lag, conflicts, quorum limits and version vectors all trace back to letting replicas fall behind.
• Every guarantee costs latency or availability. The skill is knowing which you are spending.
• Do not assume a guarantee you have not verified — quorums in particular promise less than they appear to.
• Partitioning, transactions, consensus and clocks are the natural next subjects.

References

• Martin Kleppmann, Designing Data-Intensive Applications, chapter 5 — the canonical treatment of replication this series works through.
• DeCandia et al. — Dynamo: Amazon’s Highly Available Key-value Store (SOSP 2007).
• PostgreSQL manual — Log-Shipping Standby Servers.
• Apache Cassandra — Dynamo-style architecture.