Why Distributed Systems Are Hard

Distributed systems are not "normal systems with network calls." They are a fundamentally different discipline where every assumption about computation — that memory is consistent, that operations complete, that time moves forward — breaks down. The hardness is not in any one failure; it is in the combination of failures that no single test can reproduce and no single engineer can hold in their head simultaneously. This article explains why.

The eight fallacies of distributed computing

Peter Deutsch's eight fallacies (1994) are still the most accurate diagnosis of why distributed systems fail:

1. The network is reliable. It is not. Packets are dropped, reordered, duplicated, and delayed. A TCP connection can hang indefinitely if the remote process crashes without sending a FIN.

2. Latency is zero. It is not. A cross-datacenter call costs 50–150ms. A call within a datacenter costs 0.5–5ms. A call across the ocean costs 100–300ms. Every one of these numbers is a minimum — the actual latency includes queue time, GC pauses, and OS scheduling.

3. Bandwidth is infinite. It is not. A 1 Gbps link between datacenters can saturate during bulk data transfers. Backpressure is real and must be handled.

4. The network is secure. It is not. Man-in-the-middle attacks, DNS hijacking, and compromised certificates are all real threats. TLS protects against passive observation but not active interception.

5. Topology doesn't change. It does. Load balancers fail over, DNS records change, cloud providers migrate instances, and network paths reroute during outages.

6. There is one administrator. There isn't. Different teams control different services, different cloud providers control different zones, and different organizations control different networks.

7. Transport cost is zero. It isn't. Serialization, compression, encryption, and deserialization all consume CPU and memory. A 1 KB protobuf message might cost 10µs to serialize and 10µs to deserialize — more than the network transit in a same-datacenter call.

8. The network is homogeneous. It isn't. IPv4 vs. IPv6, MTU differences, TCP vs. UDP, HTTP/1.1 vs. HTTP/2 vs. HTTP/3 — the network layer is a patchwork of protocols and behaviors.

The fallacies matter because each one represents an assumption that, when violated, causes a class of bugs that are nearly impossible to reproduce in testing.

Partial failure: the concept that changes everything

In a single-process system, a function either succeeds or fails. The failure is total — you know immediately that it happened. In a distributed system, failures are partial:

Scenario: service A calls service B, which calls service C
 
Possible outcomes:
- A succeeds, B succeeds, C succeeds       → normal
- A succeeds, B succeeds, C fails           → partial failure
- A succeeds, B fails, C succeeds           → partial failure
- A fails, B succeeds, C succeeds           → partial failure
- A succeeds, B times out, C unknown        → partial failure (most dangerous)

The last case is the killer: B timed out. Did B receive the request? Did B process it and the response was lost? Did B crash before processing? The caller cannot distinguish between "the request is still being processed" and "the request was lost." This is the two generals problem in practice — and it is the reason distributed systems require fundamentally different error handling than single-process code.

Partial failure means:

• A operation can appear to succeed but actually fail. (The response was lost; the operation completed on the remote side.)
• A operation can appear to fail but actually succeed. (The timeout fired; the operation completed on the remote side.)
• The system can be partially alive. (Some nodes are up, some are down, and the network between them is unreliable.)

Every distributed system must handle all three cases. Most systems don't, and the bugs that result are the most insidious kind — they appear under load, disappear under observation, and reproduce only in production.

The CAP theorem: what it actually says

Brewer's CAP theorem (2000), proven by Gilbert and Lynch (2002):

A distributed system can provide at most two of three guarantees:

• Consistency (linearizability): every read sees the most recent write.
• Availability: every request receives a (non-error) response.
• Partition tolerance: the system continues operating despite network partitions.

The theorem is about network partitions — not about latency, not about crashes, not about disk failures. A partition is when two groups of nodes can communicate within their group but not between groups. During a partition, you must choose:

• CP: reject requests to the minority partition (sacrifice availability).
• AP: serve stale data to the minority partition (sacrifice consistency).

The theorem does not say you must choose at system startup. It says you must choose during every partition. A system that is normally CA becomes either CP or AP when the network splits.

Normal operation:     A ↔ B ↔ C    (all consistent, all available)
 
Network partition:    A ↔ B  ✕  C  (C is isolated)
 
CP: C rejects writes          AP: C accepts writes (stale data)
    A and B continue              A and B continue
    System is unavailable         System is inconsistent
    to C's clients                for C's clients

The practical consequence: you cannot have both strong consistency and high availability during a network partition. Every system makes a choice, and the choice defines its failure mode. DynamoDB is AP (eventual consistency with conflict resolution). Spanner is CP (rejects writes during partitions). Both are correct — they serve different requirements.

The FLP impossibility result

The FLP theorem (1987) is even more fundamental than CAP:

In an asynchronous system with even one faulty process, no deterministic consensus protocol can guarantee termination.

In plain language: if the network can delay messages (which it always can) and even one process can crash (which it always can), there is no algorithm that guarantees consensus will complete. The protocol can be likely to complete (and Raft, Paxos, and others do), but it cannot be guaranteed to complete.

This is why consensus protocols use timeouts — they don't guarantee termination; they make termination probable by using timeouts to detect (suspected) failures and trigger leader changes. The timeout is a heuristic, not a guarantee.

The real difficulty: reasoning about concurrency

The fundamental challenge is not any single failure — it is the combinatorial explosion of possible interleavings:

• 3 nodes, 2 operations each: 90 possible interleavings.
• 5 nodes, 3 operations each: millions of possible interleavings.
• 10 nodes, 5 operations each: more interleavings than atoms in the observable universe.

No engineer can reason about all of them. No test suite can cover all of them. The only defense is invariants — properties that hold across all possible interleavings — and formal verification — mathematical proof that the invariants hold.

This is why consensus protocols are described as state machines with formal safety properties. The protocol is not "correct" because it passes tests. It is "correct" because a mathematical proof shows that no possible interleaving of messages can violate its invariants.

What this means for your code

1. Every network call is a failure waiting to happen. Design for timeouts, retries, and idempotency from the start. Do not add them later — you will miss cases.

2. Partial failure is the norm, not the exception. Your system is always partially degraded. The question is whether the degradation is visible to users.

3. Choose your consistency model deliberately. Do not default to "strong consistency everywhere" — it is expensive and availability-destroying. Do not default to "eventual consistency everywhere" — it is correctness-destroying. Match the consistency model to the data's requirements.

4. Test with network failures, not just process crashes. Kill a process and you test crash recovery. Partition the network and you test distributed correctness. Use tools like toxiproxy or chaos mesh to simulate network failures.

5. Document your failure modes. Every distributed system has a failure mode — a partition, a leader election, a degraded state — that is architecturally significant. If you can't describe yours, you haven't designed your system.