Replication Lag: The Hidden Cost

Leader-based replication works well when followers keep up with the leader. In normal operation, the lag is tiny — milliseconds. But under load, or when a follower restarts after a crash, that gap can grow to seconds or minutes.

This creates a problem: if you read from a lagging follower, you might get data that’s out of date. The write happened. The leader confirmed it. But the follower hasn’t seen it yet.

Eventual Consistency

The standard term for this situation is eventual consistency: given enough time without new writes, all replicas will converge on the same data. But the word “eventual” does a lot of work here. It makes no promise about how long convergence takes.

Under normal conditions, replication lag is short enough that you’d never notice. But push the system hard — high write volume, a network hiccup, a follower catching up after downtime — and the lag window can open wide. Any reads that happen during that window see stale data as if the recent writes never happened.

This isn’t a theoretical edge case. It’s the day-to-day reality of running any system with asynchronous replication, which is to say, almost every system at scale.

Reading Your Own Writes

The most immediately jarring manifestation of replication lag is when a user can’t see their own change.

The scenario plays out like this:

A user submits a comment. The write goes to the leader and is confirmed.
The page refreshes. The read goes to a follower — a different machine.
The follower hasn’t received that write yet. The comment isn’t there.
The user stares at their empty post, wondering if it went through.

Read-your-writes violation

Click ▶ Run to see how replication lag causes a user to miss their own write.

This violates a guarantee called read-your-writes consistency (also called read-your-own-writes): a user should always be able to see their own writes, even before other users can. From the user’s perspective, this is simply the application working correctly — anything less feels like a bug.

Solutions

There’s no single universal fix. The right solution depends on your access patterns.

Read your own data from the leader

If you can identify which data a user might have just written, route those reads to the leader.

The classic example: a social profile. When a user views their own profile page, always read from the leader — they may have just updated it. When they view someone else’s profile, a follower is fine (they have no recent writes there).

This works well when a clear boundary exists between “my data” and “everyone else’s data.” It breaks down when users can modify many different things, and those reads need to be fast everywhere.

Read own data from the leader

Click ▶ Run to see how routing own-data reads to the leader keeps writes immediately visible.

Track timestamps on the client

When the user makes a write, record the timestamp of that write on the client (or in a session cookie). On each subsequent read, include that timestamp. The system only routes the read to a follower whose replication log has advanced past that timestamp — otherwise it falls back to the leader.

This guarantees the user always sees at least their own most recent write, without routing all reads to the leader.

The timestamp needs to be a logical one — a log sequence number (LSN) or a database transaction ID — not wall-clock time, which can drift between machines.

Tracking the write timestamp

Click ▶ Run to see how a session LSN prevents stale reads without routing everything to the leader.

Cross-device complications

The problem gets harder when a user switches devices. The write happened on a phone, and the tablet has no session cookie from the phone. The tablet has no idea what timestamps to send. You’d need to centralize “last write” state per user — which works but adds complexity.

This is the form of the problem that trips up most teams the first time: they assume “same user, different device” is fine, and then discover that user-level state isn’t being shared.

Cross-device: session state isn't shared

Click ▶ Run to see why the session-LSN fix breaks when the user switches devices.

Cross-datacenter reads

If your replicas span multiple datacenters — one in US-East, one in EU-West — and a user can be routed to either one depending on the load balancer, the lag window can be much larger than within a single datacenter. Routing the read to the leader means crossing the datacenter boundary on every read, which adds latency.

Some systems solve this by pinning a user’s reads to the datacenter where they most recently wrote. Others accept the occasional inconsistency for non-critical reads and apply the leader-routing rule only for writes the user just made in the current session.

Key Takeaways

Asynchronous replication creates a lag window during which reads from followers may return stale data — even after the write was confirmed.
“Eventual consistency” sounds reassuring but says nothing about how long the lag lasts.
Read-your-writes consistency is the guarantee that a user always sees their own writes, even before replication has propagated them.
The practical fix is either routing the user’s own reads to the leader, or tracking per-user write timestamps and using them to decide which replica to read from.

In the next post, we’ll look at two more subtle consistency problems that lag introduces: monotonic reads and consistent prefix reads — and the approaches that address all three.

References

Martin Kleppmann, Designing Data-Intensive Applications, chapter 5 — the canonical treatment of replication this series works through.