Consistency Guarantees Under Lag
On this page5
In the previous post we saw that replication lag breaks read-your-writes consistency — users can’t always see their own changes. But lag creates two more subtle problems, both of which feel like bugs from the user’s perspective but aren’t bugs in the code at all. They’re properties of the distributed system.
Monotonic Reads
The first problem: a user can observe data going backwards in time.
This happens when a single user makes multiple reads and those reads are served by different followers — followers that are at different replication offsets. A second read, hitting a follower that’s further behind, may not show something the first read already showed.
Monotonic reads — the time-travel problem
From the user’s perspective: they saw a post. They refreshed the page. The post is gone. Not because the post was deleted — it still exists on the leader and on Follower A — but because the load balancer happened to route the second request to a follower that hasn’t replicated it yet.
This is a violation of monotonic reads: the guarantee that a user will never read older data after having already read newer data. In a single-node database this is trivially satisfied — the data only moves forward. In a replicated system with async lag, it takes deliberate engineering.
The fix: pin each user’s reads to the same replica for the duration of a session. The simplest implementation is hashing the user ID to pick a specific follower. That follower might still be behind the leader, but the user sees a consistent view from a single point in time — they never go backwards, even if they occasionally see slightly stale data.
The tradeoff is availability: if the pinned follower goes down, the session has to be rerouted, and the user might briefly see a jump in either direction.
Consistent Prefix Reads
The second problem is more subtle and shows up specifically with causally related writes.
Imagine this exchange on a Q&A platform:
- A user writes: “If Mr. Poons turns the handle, Mr. Cake will fall into the river.”
- Another user replies: “Mr. Cake always falls into the river.”
These two writes are causally ordered: the reply depends on the question. The leader processes them in order. But if the reply gets replicated to one follower faster than the question does, a reader hitting that follower might see the reply first — before the question it’s answering — which makes no sense at all.
Consistent prefix reads — causal order violated
This violates consistent prefix reads: the guarantee that any read that sees a write also sees all writes that causally preceded it.
The problem is most visible in sharded or partitioned databases where different partitions have independent replication lag. Writes that are causally related might land on different partitions, and each partition replicates independently.
The fix: ensure causally related writes always go to the same partition. For conversations or threads, partition by conversation ID. If the writes land on one partition, they’ll be replicated together and their order is preserved.
When you can’t control partitioning, some systems use vector clocks or hybrid logical clocks to track causal dependencies explicitly — any read waits until the replica has advanced past the causally preceding write before returning results. This adds latency but enforces the ordering guarantee strictly.
Solutions to Replication Lag: The Full Picture
Looking at all three consistency problems together, a pattern emerges. Each one has a pragmatic fix and a general fix:
| Guarantee | What breaks | Pragmatic fix | General fix |
|---|---|---|---|
| Read-your-writes | User can’t see own changes | Route own reads to leader | Track per-user write timestamps |
| Monotonic reads | Data appears to go backwards | Pin user to one replica | Track per-user read position |
| Consistent prefix reads | Causally ordered writes appear out of order | Partition by causal key | Vector clocks / causal ordering |
The pragmatic fixes work well for most applications. The general fixes are more complex and often not necessary unless you’re building a system that requires strict ordering guarantees — a financial ledger, a collaboration tool, or an audit log.
There’s also a higher-level solution: transactions. If you need all three guarantees by default, without per-feature engineering, use a database that provides strong consistency guarantees across replicas. Distributed transactions or consensus protocols (Raft, Paxos) can provide linearizability — reads always see the latest committed write, everywhere, always. The cost is latency and complexity.
Most applications sit in the middle: they accept eventual consistency for the majority of reads, and apply targeted fixes only to the interactions where order or recency actually matters to the user. A social feed can tolerate slight staleness; a post editor cannot.
Key Takeaways
- Monotonic reads: a user making multiple reads can see older data on a later read if the load balancer routes them to a more-lagging follower. Fix by pinning users to a single replica per session.
- Consistent prefix reads: causally related writes can appear out of order if they land on different partitions with different replication lag. Fix by co-locating causally related writes on the same partition.
- The three consistency guarantees — read-your-writes, monotonic reads, consistent prefix reads — each require deliberate design in an async-replicated system.
- Transactions with strong consistency (via consensus protocols) solve all three at once, at the cost of latency.
- Most real-world systems apply targeted fixes: leader routing for own data, replica pinning for sessions, and partition-by-causality for conversations or threads.
That closes out single-leader replication. We’ve covered how leaders and followers work, how new nodes are bootstrapped, what replication logs actually contain, and the three concrete ways that lag can make your system behave unexpectedly. From here the series removes the single-writer constraint entirely — first with multi-leader replication, then with leaderless replication.
References
- Martin Kleppmann, Designing Data-Intensive Applications, chapter 5 — the canonical treatment of replication this series works through.