The canonical definition of a distributed system — Leslie Lamport's — is a one-liner with more engineering in it than most textbooks:
A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.
Read it again, because the point is not about multi-machine complexity. It is about compounding independence: every machine gets its own clock, its own opinion about ordering, its own failure modes — and none of them owes you a heads-up. The phrase "distributed" is not a deployment choice; it's a physics that enters the moment a second clock exists.
The fallacies that pre-wire the failures
The "Eight Fallacies of Distributed Computing" were coined in the 1990s at Sun and have aged exactly as well as physics predictions: they're all still true.
| Fallacy | The reality |
|---|---|
| the network is reliable | packets are lost, delayed, duplicated, reordered — constantly |
| latency is zero | every hop costs real, variable, queueable time |
| bandwidth is infinite | a saturated link is a queue with a queue |
| the network is secure | every node is a trust boundary with its own enemies |
| topology doesn't change | releases, scale-ups, and failovers reshape the graph live |
| there is one administrator | there are N owners, N on-call rotations, N change windows |
| transport cost is zero | serialization + crypto + TCP machinery are real CPU |
| the network is homogeneous | every hop can speak a slightly different dialect (proxies, TLS, timeouts) |
Every architectural debate that "has no answer" is usually here: the fallacies are the hidden specs of your system. Retry policies, timeouts, and queueing are all responses to "the network is reliable?", dressed in configuration.
The two clocks that break everything
Two physics problems appear the instant distance does:
1. Ordering. Two machines — no shared clock — must agree whether a happened before
b. Lamport's clocks (and vector clocks after them) proved that ordering can only be
partial: you can only know the order of events that causally affect each other.
The entire industry's "eventual consistency" exists because total ordering is
unachievable without a single authority — and a single authority is a single point of
failure.
2. Timeouts. A machine cannot distinguish "the other side is slow" from "the other side is dead except not" — the indistinguishable pair that every timeout-based protocol must design around. This is exactly why consensus algorithms (Paxos, Raft) don't ask "is the leader alive?" but "who can still talk to a majority?" — the only question a split brain can answer safely.
When exactly did you build one?
The most honest symptom list, in little code:
you have a database
├─ replication? → you designed for lag, conflicts, failover — briefly
└─ connection pooling? → you designed against transient unavailability
you have a queue
├─ at-least-once? → your consumer is built for duplicates
└─ exactly-once? → congratulations; you've built consensus
you have caches
└─ invalidation → you've built a *consistency protocol* without a proofIf any of those lines is "we just assumed it works", you have a distributed system whose specs were written by hope. The honest engineering response is not "add Kubernetes" — it is to label the protocol you already run: which consistency model, which delivery guarantee, what happens to a message processed twice.
The two theorems that set the price
Two results set the boundaries of everything distributed, and they're worth quoting because they're limits, not opinions:
CAP. You can't have all three of consistency, availability, and partition-tolerance — and since partitions are physical, you choose between C and A during partitions. The real engineering question isn't "which of CAP do I pick?" (everyone picks "available when possible, consistent when possible") — it's what happens during the partition: do reads return stale data (eventual), or fail (strict)?
FLP. In an asynchronous system, no consensus protocol is guaranteed to terminate — for real systems, that's why we use timeouts and why those timeouts occasionally make the "wrong" decisions. It is also why "exactly once delivery" is a marketing term in every distributed story: you can get at-least-once plus deduplication, and the map between them is the hard part.
The mindset that fixes most of it
Good distributed engineers write their systems' failure budget inventories: a one-page table of every component, its delivery/consistency guarantee, what happens under timeout, and who owns the retry. Teams that can't fill in the "timeout" column are not distributed engineers — they're people with a deployed distributed system.
That inventory is also the platform's whole philosophy in miniature: not "how do I use the tool" but "what actually happens when it fails — and what did we choose to happen instead?"