The Runtime Theory
mediumApplicationDSA#distributed-systems#clocks#ordering#time

Why can't you compare timestamps from different machines?

They want the clock model: wall clocks drift and get stepped by NTP, so timestamp order across machines isn't causal order — and what systems do about it.

The Runtime Theory Team2 min readasked at google · databricks · microsoft · amazon

The answer is the clock model, stated precisely: wall clocks on different machines are not a single time source, so comparing their readings tells you nothing about causality. This is the question that separates people who think "timestamp = order" from people who know order must be constructed.

What the machine actually does: every server's wall clock is an oscillator counted since some epoch, corrected by NTP against a time source. Between corrections, two machines' clocks drift — quartz oscillators skew by tens of parts per million, so two uncorrected clocks diverge by milliseconds per minute. NTP keeps them within a few milliseconds in a well-run datacenter, but it does so by stepping the clock when the error is large: a clock can jump forward or backward. A machine's wall clock is not monotonic. Now the failure: process A on machine 1 reads the time as t1 and writes a record; process B on machine 2 reads the time as t2 — where t2 < t1 because machine 2's clock is 200ms behind. Timestamp comparison says the write happened after the read that actually saw it. Every distributed system built on timestamp ordering has this hole.

The concrete consequences: last-write-wins conflict resolution with client timestamps silently loses writes when clocks skew — which is why production systems use version vectors or server-assigned clocks instead; replica-lag measurement and debug correlation by timestamp are wrong by the skew between the two machines' clocks; and "the log line said this happened first" is a claim about clocks, not events.

What to use instead: causality needs logical clocks — Lamport clocks order causally-related events; vector clocks track concurrent updates; hybrid logical clocks attach a physical timestamp plus a counter so they're comparable and skew-tolerant. For real-time ordering, Spanner's TrueTime returns an interval [earliest, latest] with bounded uncertainty — roughly 7ms — and the commit protocol waits out that uncertainty before committing, buying total order at the price of latency. On a single machine, monotonic clocks are fine for measuring duration (GC pauses, timeouts) because they only move forward — but they are explicitly per-machine and meaningless across machines.

The one-liner to close: "a timestamp from one machine is a fact about that machine's clock; a comparison across machines is a guess about the universe — if you need order, build it with logical clocks or bounded physical uncertainty."

This answer walks

Follow-ups they'll push on

  1. 01What's the difference between a monotonic clock and a wall clock?
  2. 02How does Spanner order transactions across datacenters using TrueTime?
  3. 03When is last-write-wins with client timestamps actually safe?

More interviews in this topic

One dispatch a week

The trace behind each question, the tradeoff that explains it, and one technical dispatch per week — no noise.

One technical dispatch per week. No noise.