Ismail Enegadi

Sloppy Quorums, Hinted Handoff, and Multiple Datacenters

6 min readSystems
On this page6

The previous post picked apart what w + r > n does and doesn’t promise while the cluster is basically healthy. This one is about the case where the arithmetic can’t run at all, because the client cannot reach enough nodes to form a quorum in the first place.

When a Client Can’t Reach Its Quorum

Quorums already handle the failures you’d expect. A node crashing is absorbed without failover, because the write only ever needed w acknowledgements and there are others. A node running slowly is absorbed too — the client stops waiting once w have responded, so one degraded machine doesn’t set the latency for everyone. That combination is most of why leaderless replication is attractive.

Network partitions are a different shape of problem. A partition can cut a client off from a large set of database nodes that are perfectly healthy — alive, serving other clients, holding current data. From this client’s side, though, they might as well be down. If fewer than w or r of the right nodes are reachable, the client cannot assemble a quorum, and every operation on those keys fails.

What makes this worth solving is that in a cluster meaningfully larger than n, the client can usually still reach plenty of nodes. Just not the ones that happen to hold this particular key. There is capacity available; it is simply the wrong capacity.

So the database designer has a choice:

  • Return an error for every request whose quorum can’t be reached, or
  • accept the write on whichever nodes are reachable, even though they aren’t among the n where this value normally lives.

Sloppy Quorums and Hinted Handoff

The second option is a sloppy quorum. Reads and writes still demand w and r successful responses — that part is unchanged — but the responding nodes are allowed to include ones outside the value’s designated home set.

Each such write is tagged with a hint recording where it actually belongs. When the partition heals, those nodes forward the writes to their proper homes and drop their temporary copies. That forwarding step is hinted handoff.

Sloppy quorum and hinted handoff

A partition makes home nodes A and B unreachable. The write is accepted by C and a temporary node D — a sloppy quorum. When the partition heals, D delivers the write to A via hinted handoff.

The everyday version: locked out of your house, you ask a neighbour whether you can sleep on their couch. You are still housed. You are not home, and when your keys turn up the arrangement ends.

Why It Isn’t Really a Quorum

Here is the part that matters, and it is easy to skim past because the name says “quorum”.

A sloppy quorum does not give you the overlap guarantee. It gives you durability: after the write returns, the data is stored on w nodes somewhere. That is a genuinely useful property — the write is not lost — but it is a different property from the one a strict quorum provides.

The consequence is concrete. Even with w + r > n, a read of r home nodes can miss the latest value entirely, because the latest value is sitting on a stand-in node that isn’t in the home set and won’t be until hinted handoff completes. The arithmetic still holds; it is just being applied to two different sets of nodes.

Which means a sloppy quorum is best read as an availability feature that happens to reuse the quorum machinery, not as a consistency mechanism. It raises write availability — as long as any w nodes are reachable, writes keep succeeding — and it lowers the confidence you can place in a subsequent read.

That the trade is genuinely contested shows up in the defaults. Riak enables sloppy quorums by default; Cassandra and Voldemort ship with them disabled. Same mechanism, opposite judgement about which risk is worse, from teams who have all thought hard about it. It is a configuration decision, not a best practice.

Stretching Across Datacenters

Leaderless replication turns out to suit multi-datacenter deployment well, and for a reason worth noticing: it was already built to tolerate conflicting concurrent writes, network interruptions and latency spikes. A cross-datacenter link is mostly just a slower, flakier version of conditions the model already assumes.

Compare that with multi-leader replication, where spanning datacenters meant adding a second leader and inheriting write conflicts as the price. Here there is no new failure mode to take on — the existing one simply stretches.

Two approaches are in use:

Extend n across datacenters. Cassandra and Voldemort keep the ordinary leaderless model and let n span every datacenter, with configuration for how many of the n replicas live in each. A write goes to all replicas everywhere, but the client typically waits only for a quorum of nodes in its local datacenter. Cross-datacenter latency and interruptions therefore don’t block the client, and the remote writes are usually asynchronous.

Keep n local and replicate between clusters. Riak keeps all client-to-node traffic inside one datacenter, so n counts replicas within that datacenter only. Replication between datacenters happens asynchronously in the background — which is, structurally, the multi-leader arrangement from part 7 with leaderless clusters at each end.

The distinction is where the datacenter boundary sits relative to the quorum. In the first, it’s inside n and the local-quorum trick keeps it off the critical path. In the second, it’s outside n entirely and handled by a separate replication stream.

Key Takeaways

  • Quorums already absorb crashed and slow nodes without failover, because a write only ever waits for w acknowledgements.
  • A partition is different: it can cut a client off from healthy nodes, leaving it unable to assemble a quorum even though much of the cluster is reachable.
  • A sloppy quorum accepts writes on reachable nodes outside the value’s home set; hinted handoff forwards them home once the partition heals.
  • It buys durability, not overlap. The data is on w nodes somewhere, but a read of r home nodes can still miss it until handoff completes — even with w + r > n.
  • Treat it as an availability feature, not a consistency one.
  • The defaults disagree: on by default in Riak, off in Cassandra and Voldemort. It’s a configuration decision, not a best practice.
  • Leaderless suits multi-datacenter work because it already assumes conflicting writes and unreliable links.
  • Cassandra and Voldemort span n across datacenters and wait on a local quorum; Riak keeps n local and replicates between clusters asynchronously.

References

Subscribe via RSS