Write Conflicts in Multi-Leader Replication
On this page9
In the previous post we established why multi-leader replication exists — multi-datacenter deployments, offline clients, collaborative editing. Each use case buys something real: lower write latency, better fault isolation, the ability to work without a network connection. The price is always the same: write conflicts.
In a single-leader database, conflicts can’t happen by design. Two writers racing to update the same record are serialized at the leader — one goes first, the other waits or retries. The final state is always unambiguous.
In a multi-leader database, both writes succeed locally and immediately. The conflict is discovered later, asynchronously, when replication tries to reconcile two leaders that have diverged. By then, the clients have already moved on.
How a Conflict Arises
Consider a wiki page whose title is currently “A”. User 1, connected to Leader 1, renames it to “B”. At the same moment, User 2, connected to Leader 2, renames it to “C”. Both writes land on their respective leaders, both get an immediate success response.
A write conflict across two leaders
When replication runs, Leader 1 ships its change (title = B) to Leader 2 — but Leader 2 already has title = C. Conflict. Leader 2 ships its change (title = C) to Leader 1 — which now has title = B. Conflict. Both leaders have diverged, and there is no “correct” answer that falls out automatically.
Why You Can’t Just Detect Conflicts Synchronously
The obvious fix: before acknowledging a write, wait for it to replicate to all other leaders first, and only confirm success once all leaders have agreed. Any conflicting write would be detected immediately and rejected.
This works, but it destroys the main advantage of multi-leader replication. If Leader 1 in Tokyo has to wait for acknowledgment from Leader 2 in Frankfurt before confirming a write to the user, the write latency is now the inter-datacenter round trip — exactly what multi-leader was supposed to eliminate. Synchronous conflict detection degrades multi-leader back to single-leader semantics, with more complexity and no benefit.
The core tension is fundamental: multi-leader replication gives each node the ability to accept writes independently, and conflicts are the price of that independence.
Conflict Avoidance
The simplest strategy for dealing with conflicts is to not have them at all.
If the application can ensure that all writes for a particular record always go through the same leader, conflicts cannot occur. From any one user’s perspective, the system looks like single-leader replication even though multiple leaders exist.
A concrete implementation: route all writes from a given user to their “home” datacenter. User A’s requests always go to the datacenter nearest to them; User B’s requests go to theirs. As long as users don’t share records, there’s nothing to conflict.
The limitation is that this avoidance breaks down. A user moves to a different region. A datacenter fails and traffic gets rerouted. The designated leader for a record needs to change. In any of these situations, writes for the same record can temporarily land on different leaders, and the conflict problem comes back.
Conflict avoidance is a practical default — it handles the common case well — but it can’t be the complete answer.
Converging Toward a Consistent State
When a conflict can’t be avoided, it has to be resolved. And regardless of how it’s resolved, every replica must eventually agree on the same final value. A database where Leader 1 has title = B and Leader 2 has title = C forever is unacceptable — any replication scheme must converge.
There are four main ways to achieve convergent conflict resolution:
Last Write Wins (LWW). Attach a timestamp to each write. When a conflict is detected, the write with the higher timestamp wins; the other is discarded. Simple, widely implemented (Cassandra uses it by default), and dangerously prone to data loss. Clocks across distributed nodes are never perfectly in sync, so “higher timestamp” doesn’t reliably mean “more recent.” A write that happened later can lose to one that happened earlier if the clocks differ by even a few milliseconds. LWW is appropriate when data loss is acceptable — analytics events, ephemeral state — not for records that users expect to be durable.
Replica ID priority. Assign each replica a unique ID. When a conflict is detected, the write from the higher-numbered replica always wins. Deterministic and consistent, but arbitrary: there’s no reason the replica with ID 5 should be right more often than replica 2. This also implies data loss.
Merge the values. Instead of picking one winner, combine both. For the title conflict, the merged result might be "B/C" — preserving both values concatenated or sorted alphabetically. This avoids data loss but produces a value that might be nonsensical for the application. It works well for some data types (sets, counters) and poorly for others (a page title, a price, a status field).
Explicit conflict record. Record the conflict in a data structure that preserves all versions, and surface it to the application for resolution later. The application might prompt the user to pick the correct value, or resolve it using domain logic. This is the most flexible approach and the only one that genuinely avoids data loss — but it requires the application to handle the conflict, which most applications aren’t designed to do.
Side by side:
| Strategy | Converges | Loses data | Needs app logic | Reach for it when |
|---|---|---|---|---|
| Last write wins | ✓ | ✓ silently | ✗ | writes are disposable (metrics, events) |
| Replica ID wins | ✓ | ✓ silently | ✗ | you need determinism and nothing else |
| Merge values | ✓ | ✗ | ✗ | the type merges cleanly (sets, counters) |
| Conflict record | ✓ | ✗ | ✓ | the value is worth a user’s attention |
Every row converges — that part is non-negotiable. The columns that actually decide the choice are the middle two: whether you can afford to lose a write, and whether there is application code willing to resolve one.
Custom Conflict Resolution Logic
Most multi-leader replication tools let you write conflict resolution code that runs at one of two points:
On write. As soon as the database detects a conflict in the replication log, it calls your handler. The handler runs in the background, synchronously with replication — it can’t prompt the user, and it must run fast. Bucardo (a PostgreSQL replication tool) lets you write Perl handlers this way.
On read. The database stores all conflicting versions of a record without resolving them. On the next read, it returns all versions to the application. The application resolves the conflict — by prompting the user, applying domain logic, or picking by some rule — and writes the resolved value back. CouchDB works this way.
One important nuance: conflict resolution applies at the level of an individual row or document, not at the level of a transaction. If a transaction atomically makes five writes, each write is resolved independently. A conflict handler has no visibility into the fact that those five writes were originally atomic.
Automatic Conflict Resolution
Writing correct conflict handlers is hard, and the business logic required is often subtle. Amazon famously ran a shopping cart that preserved all concurrent additions to the cart but silently dropped concurrent removals — customers would see items reappear after they’d deleted them. The conflict handler was technically correct (it preserved all writes) but wrong for the application semantics.
Three research-backed approaches to avoiding hand-rolled conflict handlers:
CRDTs (Conflict-Free Replicated Data Types). Data structures for sets, maps, ordered lists, and counters that are mathematically guaranteed to merge correctly when concurrent edits are combined, with no application-level conflict handler required. Riak 2.0 implemented them; they’re increasingly appearing in collaborative tools. The constraint is that not every data type has a natural CRDT form — CRDTs work best for append-heavy or commutative operations.
Mergeable persistent data structures. Track full edit history explicitly, like Git does, and use a three-way merge function (current state, common ancestor, incoming state) to resolve conflicts. More powerful than CRDTs because the ancestor gives the merge algorithm context, but heavier-weight.
Operational transformation (OT). The conflict resolution algorithm behind Google Docs and Etherpad. Designed specifically for ordered lists of characters — i.e., text documents. OT transforms concurrent operations against each other so they can be applied in any order and still converge on the same result. Complex to implement correctly; the literature is full of OT algorithms with subtle bugs.
None of these are plug-and-play solutions for a general-purpose database. But they represent the direction the field is moving, and CRDTs in particular are making their way into production systems.
What Actually Counts as a Conflict
Some conflicts are obvious: two concurrent writes set the same field to different values. But conflicts can be more subtle than same-field writes.
Consider a meeting room booking system. Two groups each try to book Room A for 2pm on Tuesday. The system checks availability before writing — both checks happen at the same time, both find Room A free, both proceed to write a booking record. These writes touch different rows (two separate booking records), but they represent a logical conflict: there can’t be two groups in the same room at the same time.
This kind of conflict — where the constraint is on the combination of records rather than on a single field — is much harder to detect and resolve at the database level. The application knows the constraint, but the replication system doesn’t. Standard conflict resolution strategies (LWW, CRDTs, merge) don’t help here.
The only real solution is to avoid the conflict: route all bookings for a given room through a single leader, or use a distributed lock. Which brings us back to conflict avoidance as the safest default when the conflict semantics are complex.
Key Takeaways
- Write conflicts arise when two leaders accept concurrent writes to the same record. Both succeed locally; the conflict is detected only during async replication.
- Making conflict detection synchronous defeats the purpose of multi-leader replication — it re-introduces single-leader write latency.
- Conflict avoidance (routing all writes for a record through one leader) is the simplest and most reliable strategy when it’s feasible.
- When conflicts do occur, resolution must be convergent — all replicas must eventually agree on the same final value.
- The four convergent strategies are: Last Write Wins (simple, lossy), replica ID priority (deterministic, lossy), merge (lossless, can produce odd values), and explicit conflict records (lossless, requires application logic).
- On-write and on-read are the two hooks for custom conflict resolution logic; on-read is more flexible, on-write must be fast and non-interactive.
- CRDTs, mergeable persistent data structures, and operational transformation are research-backed paths toward automatic conflict resolution that avoids hand-written handlers.
- Subtle conflicts — like double-booking a resource — involve constraints across multiple records, not just same-field writes, and are harder to resolve automatically.
References
- Martin Kleppmann, Designing Data-Intensive Applications, chapter 5 — the canonical treatment of replication this series works through.
- Shapiro, Preguiça, Baquero & Zawirski — Conflict-free Replicated Data Types (2011), the paper CRDTs come from.
- CouchDB manual — Replication and conflicts, on-read conflict resolution in practice.
- Riak data types — CRDTs exposed as a database feature.
- Bucardo — on-write conflict handlers for PostgreSQL.
- Operational transformation — the algorithm family behind Google Docs and Etherpad.