Ismail Enegadi

Running a Replicated Cluster

7 min readSystems
On this page11

In the previous post, we covered how leader-based replication works in steady state: writes flow to the leader, the replication log fans out to followers, and clients read from any node. Now for the harder part — what happens when you need to grow the cluster, or when nodes fail?

Setting Up New Followers

You might want to add a follower to increase read capacity, replace a failed node, or improve fault tolerance. The problem: you can’t simply copy the leader’s data files to a new machine while traffic is running. The data is changing constantly, and a naive copy will end up with some tables captured at one point in time and others at another — an inconsistent mess.

You also can’t just lock the database for the duration of the copy. That defeats the point of having a distributed system.

The solution is more elegant: use the replication log position as the bridge between a static snapshot and the live data stream.

Adding a new follower

Click ▶ Run to see how a new follower bootstraps without taking the cluster offline.

Here’s how it works in practice:

  1. Take a consistent snapshot of the leader’s data at a specific point in time. The database records exactly where in the replication log this snapshot corresponds to — PostgreSQL calls this the log sequence number (LSN); MySQL calls it the binlog coordinates. Most databases can do this without locking the entire database, using the same snapshot isolation mechanism that makes transactions work.

  2. Transfer the snapshot to the new node. This can take a while for large datasets, but the leader keeps running normally the entire time.

  3. Connect and request the log gap. The new follower connects to the leader and says: “give me everything that happened after log position X.” The leader has been logging writes the whole time, so it can replay exactly what the new follower missed while the snapshot was being transferred.

  4. Catch up. The follower processes the backlog. Once it reaches the leader’s current log position, it’s fully in sync and joins the cluster as a normal replica, receiving new changes in real time.

This process works because the replication log is both a durable record of history and the live channel for new writes — the same log that replays the past also carries the present.

Handling Node Outages

Nodes fail. Disks corrupt, networks partition, machines crash for maintenance. A resilient replicated system needs a plan for each case.

Follower failure: easy

When a follower crashes and restarts, it knows exactly where it left off — it keeps a local record of the last log entry it processed. On reconnect, it asks the leader for everything it missed and catches up. This is self-healing by design.

Follower failure: self-healing

Click ▶ Run to see how a follower recovers after a crash without any manual intervention.

Leader failure: hard

Leader failure is a different problem entirely. One of the followers needs to be promoted to leader, clients need to be reconfigured to send writes to the new node, and the remaining followers need to switch to replicating from the new leader. This whole process is called a failover.

Leader failover

Click ▶ Run to watch automatic failover: detect → elect → redirect → rejoin.

Automatic failover breaks down into four steps:

  1. Detect the failure. Followers monitor the leader with periodic health checks. If the leader doesn’t respond within a timeout, it’s declared dead. Choosing the right timeout is genuinely hard — too short and a slow network causes false positives that trigger unnecessary failovers; too long and the cluster stays leaderless for longer during a real outage.

  2. Elect a new leader. The remaining nodes need to agree on which follower becomes the new leader. The usual choice is the node with the most up-to-date copy of the data — the one that had received the most replication log entries before the leader went down.

  3. Redirect clients. The system reconfigures clients and other followers to route writes to the new leader. This can happen automatically through a controller process or via service discovery, or it can be done manually by an operator.

  4. Rejoin the old leader. If the original leader comes back online, it must recognize it’s no longer in charge and become a follower. This sounds simple, but it’s where things can get subtle.

The Edge Cases That Make Failover Dangerous

Even a well-implemented automatic failover has failure modes worth understanding.

Data loss from asynchronous replication

If the old leader was using asynchronous replication, some writes that it acknowledged to clients may not have reached any follower before it failed. When a follower is promoted, those writes are simply gone — even though the clients received a success response. From the client’s perspective, data they thought was saved has disappeared. This is one of the most jarring failure modes in distributed databases.

Some systems deal with this by discarding the old leader’s unreplicated writes when it comes back — accepting the data loss. Others prefer to keep those writes and try to reconcile, but that can cause conflicts with writes that happened on the new leader in the meantime.

Split-brain

What if the old leader didn’t actually die — it just lost network connectivity temporarily, and the rest of the cluster couldn’t reach it? It might still be accepting writes. Now you have two nodes both believing they are the leader, both accepting writes, and no coordination between them. This is called split-brain, and it can silently corrupt your data.

Split-brain

Click ▶ Run to see what happens when the old leader never actually died.

To guard against this, some systems use a mechanism called STONITH (“Shoot The Other Node In The Head”) — when in doubt, forcibly shut down the other node before accepting writes. It’s not subtle, but it prevents split-brain.

The right timeout

There’s no universally correct timeout for declaring a leader dead. A timeout that’s too short causes frequent unnecessary failovers, each of which is disruptive. A timeout that’s too long means the cluster stays degraded for longer during a real failure. The right value depends on your network stability, your write volume, and your tolerance for each kind of disruption.

Why Teams Often Prefer Manual Failover

Given these edge cases, many teams operating critical databases choose manual failover — they let an operator make the call on when and how to promote a new leader, rather than automating it. The automation handles the mechanics (log transfer, client reconfiguration), but a human decides when it’s safe to trigger.

This is especially common when the stakes of data loss or split-brain are high: financial systems, inventory databases, anything where a phantom write reappearing or disappearing has real-world consequences.

Key Takeaways

  • Adding a new follower is safe and non-disruptive: take a snapshot at a log position, transfer it, then replay the log gap until caught up.
  • Follower failures are self-healing — the follower picks up from its last recorded log position on reconnect.
  • Leader failover has four steps: detect, elect, redirect, rejoin — each step can fail in subtle ways.
  • Async replication means a new leader may be missing writes the old leader already acknowledged to clients.
  • Split-brain (two nodes thinking they’re leader) is a real risk with serious consequences.
  • Many teams prefer manual failover over automated failover for critical systems.

In PostgreSQL the snapshot-and-catch-up dance is a single command run on the new follower:

# -Xs streams WAL during the copy, so changes made while the base backup
#     runs are captured instead of leaving a gap to fill afterwards.
# -R  writes standby.signal and the primary_conninfo settings, so the node
#     starts up as a follower and begins streaming on its own.
pg_basebackup \
  --host=leader.internal \
  --username=replicator \
  --pgdata=/var/lib/postgresql/17/main \
  --wal-method=stream \
  --write-recovery-conf \
  --progress

The leader stays writable throughout — this is a read of a consistent snapshot, not a lock.

In the next post, we’ll look under the hood at how the replication log itself is implemented — the four different approaches databases use to represent and ship changes.

References

Subscribe via RSS