Ismail Enegadi

What "Concurrent" Actually Means

6 min readSystems
On this page6

Quorums look like they should settle the question of which write is current. If every read set overlaps every write set, surely the read sees the latest value. But “latest” quietly assumes the writes can be put in an order, and in a leaderless system that assumption is exactly what’s missing.

This post is about the word we have been leaning on without defining: concurrent. It turns out to have a precise meaning, and it is not the one most people reach for.

Conflicts Survive Strict Quorums

Two clients write to the same key at the same moment. Both send to all n replicas, both collect w acknowledgements, both are told they succeeded. But the writes reach the replicas in different orders — network delays are not uniform, and a node that is briefly unresponsive may see one write and miss the other entirely.

Concurrent writes: both clients succeed, nodes disagree

Two clients write to overlapping quorums at the same time. Both claim success (w=2), but Node B receives conflicting values.

The result is that replicas disagree about the final value, and no amount of quorum arithmetic fixes it. The quorum guarantees an overlap, not an order. A read that touches every replica still finds two candidate values and no rule for choosing between them.

There is a sharper version of this that catches people out. Conflicts do not only arrive from clients writing at the same time — they are also produced by the repair machinery itself. Read repair writes a value back to a replica it believes is stale, and hinted handoff delivers a write to its home node long after the fact. Both are writes, both can land alongside another value, and both can therefore manufacture the conflict they were meant to clean up.

Last Write Wins, and Its Real Safety Rule

The simplest way out is to impose an order that does not naturally exist. Attach a timestamp to every write, and when a replica holds two values, keep the one with the larger timestamp. Convergence is guaranteed: every replica applies the same rule to the same pair and lands on the same answer.

This is last write wins (LWW), and it is what Cassandra does — it is the only conflict resolution the database offers. Riak supports it as an option.

The cost is durability. If several writes were concurrent, every one of them was acknowledged as successful, and all but one are silently discarded. The client that lost was told it had won.

The usual advice is that LWW is fine “when losing writes is acceptable”, which is true but too vague to act on. The precise rule is narrower and more useful:

LWW is safe when a key is written exactly once and never updated again.

If a key is immutable after creation, there is no second write to lose. This is why the recommended pattern with Cassandra is to generate a fresh UUID for every write and use it as the key, so no two writes ever contend for the same one. You are not resolving conflicts; you are arranging for them to be impossible.

Any other use of LWW is a decision to lose data. That can be the right decision — caches, telemetry, anything where the newest sample is the only one that matters — but it should be a decision, not an accident.

There is a second problem lurking in “larger timestamp”, which is that it depends on clocks agreeing across machines. That is a much bigger subject than replication, and it is not one this series will settle here.

Happens-Before

To do better than an arbitrary order, we need to know when an order genuinely exists.

The relationship we care about is happens-before. An operation A happens before an operation B if B knew about A: B read the value A wrote, or built on it, or depended on it in some other way. Order is defined when one operation is downstream of the other.

When neither operation knew about the other, they are concurrent.

That is the whole definition, and the important thing about it is what it leaves out. It says nothing about clocks.

Happens-before vs concurrent

Two operations are causally related only when one knew about the other. Overlapping in time has nothing to do with it — the lower pair never overlap and are still concurrent.

The temptation is to read “concurrent” as “at the same time”, and it is wrong. Two writes that overlap to the microsecond are concurrent, yes — but so are two writes ten minutes apart, if the network kept each client ignorant of the other for those ten minutes. A partition that lasts an hour produces writes an hour apart that are still, formally, concurrent. Simultaneity is not the test. Knowledge is.

That reframing is what makes the problem tractable. Wall-clock time across a distributed system is unreliable and expensive to reason about, but “did this operation know about that one” is a question the database can answer exactly, by tracking what each client had read before it wrote.

Three Possibilities

For any two operations on the same key, there are exactly three cases:

  • A happens before B. B is the later state. It should win.
  • B happens before A. A is the later state. It should win.
  • A and B are concurrent. Neither supersedes the other, and the system has a genuine conflict.

The first two need no cleverness — one value overwrites the other, and nothing is lost. Only the third case requires a decision, and that decision is where LWW quietly throws data away.

So the whole problem reduces to a single mechanical question the database must answer for every pair of writes it holds: did either of these know about the other? Answer that, and overwrites resolve themselves while true conflicts are surfaced instead of silently discarded.

Answering it is what version vectors do, and they are where this goes next.

Key Takeaways

  • A quorum guarantees that read and write sets overlap, not that writes have an order. Concurrent writes conflict even when w + r > n holds.
  • Read repair and hinted handoff are themselves writes, so the repair machinery can create the conflicts it exists to resolve.
  • Last write wins converges by imposing an arbitrary order via timestamps, discarding every concurrent write but one — after telling each client it succeeded.
  • LWW is only genuinely safe when a key is written once and never updated, which is why the Cassandra pattern is a fresh UUID per write.
  • A happens before B when B knew about A. When neither knew about the other, they are concurrent — regardless of how far apart in time they occurred.
  • Concurrency is about knowledge, not clocks. Writes ten minutes apart are concurrent if the network kept each client unaware of the other.
  • Every pair of writes is either an overwrite or a conflict. Only conflicts need resolving, so the database’s real job is telling the two apart.

References

Subscribe via RSS