Ismail Enegadi

Multi-Leader Replication: When One Leader Isn't Enough

6 min readSystems
On this page6

In the previous posts we covered single-leader replication in depth — how changes flow from leader to followers, how followers catch up after failure, and how lag creates consistency problems for readers. Single-leader replication has one structural limit that all of this assumes away: every write must go through a single node. If that node is unavailable, or just geographically far from a user, writes are blocked or slow.

Multi-leader replication (also called master–master or active/active replication) removes that constraint. More than one node can accept writes. The tradeoff is that writes can now conflict, and somebody has to resolve them.

Multi-Datacenter Operation

The most common and most defensible use case for multi-leader replication is running a database across multiple datacenters — for fault tolerance, geographic distribution, or both.

In a single-leader setup, the leader lives in one datacenter. Every write from any user anywhere must travel to that datacenter. Reads can be served locally from followers, but writes can’t. If that datacenter has a network problem or goes down entirely, writes stop until the leader is recovered or failed over.

In a multi-leader setup, each datacenter has its own leader. Writes from users in that region go to the local leader. Between datacenters, leaders replicate to each other asynchronously over the internet.

Multi-leader replication across two datacenters

Each datacenter has its own leader. Writes are accepted locally, then replicated asynchronously to the other datacenter's leader.

The performance difference is significant for write-heavy workloads. A user in Tokyo writing to a leader in Frankfurt adds hundreds of milliseconds of round-trip latency. With a local leader, that write completes at local network speed — the cross-datacenter replication happens in the background, hidden from the user.

Comparing the two configurations on the axes that matter most:

Single-leaderMulti-leader
Write latencyHigh for remote usersLow — always local
Datacenter outageLeader DC down = no writesOther DCs continue independently
Network partitionWrites blocked to leader DCTemporary partition = brief local writes, sync when restored
ComplexitySimpleWrite conflicts must be resolved

Some databases support multi-datacenter operation natively. Others use external replication tools: Tungsten Replicator for MySQL, BDR for PostgreSQL, GoldenGate for Oracle. Either way, the conflict resolution problem doesn’t go away — it’s just handled at different layers.

Clients with Offline Operation

A less obvious but structurally identical use case: applications that need to accept writes when the device has no internet connection.

Consider a calendar app running on a phone, a laptop, and a tablet. The user creates a meeting on their phone while on a plane. When the phone comes back online, that meeting needs to sync to the server and to all other devices. Meanwhile, the user also edited a different meeting on their laptop. Both changes need to merge.

From a database perspective, every device running the app is a leader. Each accepts writes locally (reads and writes happen against a local database, with no network required). When connectivity is restored, each device replicates its changes to the server and other devices — just like leaders replicating to each other across datacenters.

The replication lag in this scenario can be extreme: hours or days if the user is truly offline. But the model is the same as the multi-datacenter case, just taken further. The replication is asynchronous, the conflicts are real, and they need to be resolved.

This is exactly what CouchDB is designed for — it treats every client as a peer with its own local database, and provides a sync protocol built around resolving concurrent edits. The broken history of calendar sync across platforms is a testament to how hard this problem actually is in practice.

Collaborative Editing

Real-time collaborative editing — Google Docs, Notion, Figma — is a variant of the same problem at finer granularity.

When two people edit the same document simultaneously, each user’s client maintains its own local replica. Keystrokes are applied instantly to the local copy (which is why the editor never feels laggy), then replicated to the server and other clients.

If you want to guarantee no conflicts, you can lock the document before editing — only one user can hold the write lock at a time. This is equivalent to single-leader replication: it prevents conflicts, but only one person can write at a time, and anyone trying to edit waits.

For faster collaboration without locking, you let multiple users write concurrently. This brings all the challenges of multi-leader replication: changes made at the same time to the same part of the document need to be merged somehow. Collaborative editors deal with this using specialized algorithms — operational transformation (OT) in older systems, CRDTs (conflict-free replicated data types) in newer ones — which we’ll look at more closely in the next post.

The Price: Write Conflicts

All three use cases above share the same structural property: more than one node accepts writes to the same data, and those writes can arrive in any order.

In a single-leader database, this situation can’t arise. If two clients try to write the same record at the same time, one of them blocks and waits; the leader serializes them. The final state is unambiguous.

In a multi-leader database, both writes succeed locally. The conflict is only detected later, when replication tries to reconcile them. By then it may be too late to ask the user — the client that made the write has already moved on. The database needs a strategy for resolving this automatically.

How to detect conflicts, the different strategies for resolving them (last-write-wins, merge, custom logic), and the topologies that connect multiple leaders to each other — that’s where the series goes next.

Key Takeaways

  • Multi-leader replication allows more than one node to accept writes. Each node forwards its writes to all the others, creating a network of peers.
  • The most common use case is multi-datacenter deployments: each datacenter has a local leader, writes are accepted locally, and replication between datacenters is asynchronous. This improves write latency for geographically distributed users and lets each datacenter continue operating independently during outages.
  • Offline-capable clients follow the same pattern: each device is effectively a datacenter, syncing when connectivity is available. CouchDB is built around this model.
  • Collaborative editing is multi-leader replication at keystroke granularity. Locking to prevent conflicts degrades it to single-leader semantics; fine-grained concurrency requires conflict resolution algorithms.
  • The fundamental downside is write conflicts: two leaders can accept concurrent writes to the same record, and the database must reconcile them after the fact. This is the core problem of multi-leader replication.

References

Subscribe via RSS