The Runtime Theory
mediumApplicationDSA#storage#s3#consistency#distributed-systems

Explain object storage and its consistency model

Tests whether you understand object storage as a distributed key-value HTTP API — strong read-after-write since 2020, no multi-object atomicity, and durability as a repair-velocity property.

The Runtime Theory Team2 min readasked at cloudflare · amazon · google · netflix

This question is testing whether you understand object storage as a distributed system with a specific contract, not as "a disk in the cloud." A strong answer names the contract, explains exactly what consistency you get, and where it breaks.

The mental model: object storage is a key-value HTTP API, not a disk. A PUT is an HTTP request to a control plane that places the object across a minimum of three AZs and returns 200 when internal replication accepts it. There is no mount, no file descriptor, no filesystem — there is a request and a response. The provider owns the filesystem, the replication, and the durability math; you exchange a request for a result. The durability number follows from the design: 99.999999999% means an expected loss of 0.0001 objects per 10 million per year — statistically one object every 10,000 years — and it only holds because durability is a repair-velocity property: corrupted or lost replicas are re-replicated faster than replicas fail.

The consistency model, precisely. Since December 2020, S3 gives strong read-after-write consistency in all regions: a GET immediately returns the written value, lists include new objects, and overwrites are immediately visible. That killed an entire category of "S3 races" people designed around. But strong consistency is not transactionality: there is no multi-object atomicity. Two objects updated together — a config file and its checksum — can be read in a torn state, one new and one old, because each PUT is its own distributed transaction. If the application needs all-or-nothing, the manifest pattern is the answer: write the payloads first, then write a manifest object last, and treat the manifest as the commit point. Readers that see the old manifest see old data; readers that see the new manifest see the new set — never a mix.

Where it breaks, and what it's for. Object storage is the wrong tool when you need random-access, sub-millisecond I/O or filesystem semantics — POSIX locking, rename atomicity, mmap. That is block storage's job: the device gives strict ordering and read-after-write within its replication domain, and consistency is whatever your filesystem and application do with it. Object storage wins on write-once-read-many, immutable artifacts, and backups. The real gotchas: versioning is what turns durability into protection against your own mistakes — a DELETE or a botched overwrite without it is permanent by design — and egress charges make "download everything to iterate" the most expensive design decision you can make, ~$0.09/GB out of S3.

Edge cases. Durability does not equal availability: an 11-nines object is still unreachable through a degraded API path — the object survives, your access path doesn't. And latency is the tell: a well-behaved S3 GET is 10-50 ms plus network RTT, where a gp3 EBS volume serves random 4 KB reads in sub-millisecond — but a byte-range GET pulls a 5 MB window from a multi-GB object in one request, which is why S3 backs video streaming anyway.

This answer walks

Follow-ups they'll push on

  1. 01If S3 is strongly consistent, what races still exist?
  2. 02What is the manifest pattern, and what problem does it solve?
  3. 03Where does object storage lose to block storage, and why?

More interviews in this topic

One dispatch a week

The trace behind each question, the tradeoff that explains it, and one technical dispatch per week — no noise.

One technical dispatch per week. No noise.