Container Images Are Layered Filesystems

A container image is not a disk image, a tarball, or an archive of your application. It is a list of filesystem diffs — ordered, immutable, content-addressed layers that the runtime stacks into a single virtual filesystem using overlayfs. Everything people optimize about containers — build cache hits, pull times, image size, layer reuse — is a consequence of this one design decision.

An image is a manifest of diffs

The OCI image format is a tree of JSON documents. The top-level manifest lists the layers by their content digest (SHA-256 of the layer blob), and the config carries metadata — environment variables, entrypoint, exposed ports, user:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "layers": [
    { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:7e0f5e8a8f6b...", "size": 28148261 },
    { "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:4b8e9d2c3a1f...", "size": 1043 }
  ]
}

Each layer is a tar archive of the changes made since the previous layer — deleted files are recorded as whiteouts, not removed from earlier layers. The image is the sum of those diffs, in order. That ordering is what makes everything downstream work, and break.

overlayfs: stacking diffs into a filesystem

When a container starts, the container runtime (runc, crun, containerd) creates a stacked mount:

merged view (what the container sees)
  ├── upperdir   — the writable container layer (scratch space, tmpfs-backed by default)
  └── lowerdirs  — the image layers, ordered top to bottom, all read-only
       └── base image layer

The runtime's mechanics come straight from the kernel:

• Copy-on-write. When your process writes a file that exists in a lower layer, the kernel copies it into the upper layer first (overlayfs does this on open-for-write). Reads stay in the lower layers; nothing shared is ever modified in place.
• Whiteouts. A file deleted in a later layer appears as a character device marker (.wh.<name>) in that layer's tar, hiding the lower-layer file from the merged view.
• Merged view is virtual. df -h inside a container reports the size of the merged overlay, which is not the size on the host — the same base layer is shared by every container using it, mapped once into the page cache.

This is why "image size" and "disk usage" diverge: ten containers from the same image cost one copy of the base layers on disk. Layer sharing is the memory, not the files.

Why layer order determines your cache

The build cache is keyed on layer inputs: a layer whose content and parent digest are unchanged is never rebuilt — Docker and BuildKit return the existing layer blob instantly. The corollary is the rule everyone learns by burning a CI cycle:

# Bad: dependency layer invalidated on every code change
FROM node:20-alpine
COPY . /app
RUN npm ci
 
# Good: dependencies cached; only the app layer rebuilds
FROM node:20-alpine
COPY package.json package-lock.json /app/
RUN npm ci
COPY src/ /app/src

When COPY . precedes npm ci, any change to any file invalidates the layer that npm ci produces, so the dependency layer is rebuilt and re-pushed on every commit. Reorder the lines and the dependency layer's digest stays identical — the runtime reuses it locally and the registry deduplicates it remotely. Layer order is your cache TTL, encoded in a Dockerfile.

The same rule governs pull time. Registries fetch layers in parallel, so a 50-layer image pulls faster wall-clock than a 10-layer image of the same total size — but each layer has a per-layer overhead (HTTP request, decompression, extraction), so very many tiny layers waste time on ceremony. The practical sweet spot: 10–25 layers, largest first (a huge base layer downloads while small app layers race behind it).

What actually makes images big

• Base image selection. node:20 is ~360 MB compressed; node:20-alpine ~50 MB. Alpine's musl libc also changes ABI behavior — native modules compiled for glibc won't run on it, so the saving is not free.
• Build leftovers. npm ci followed by npm cache clean still leaves the cache inside the layer unless the cache is written and cleaned in the same RUN, or the cache dir is a separate layer that a later layer deletes — which works only because layers are diffs.
• Multi-stage builds fix this at the structural level:

FROM golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /bin/app .
 
FROM gcr.io/distroless/static-debian12
COPY --from=build /bin/app /app
ENTRYPOINT ["/app"]

The first stage carries the compiler and module cache; the final image contains only the statically linked binary — typically 90% smaller than the naive single-stage image. The compiler was never in the shipped image; it lived in a discarded intermediate.

Reading a real image

docker history myapp:latest --no-trunc --format "{{.CreatedBy}}"
 
docker image inspect myapp:latest --format '{{json .RootFS.Layers}}'

The history output is the Dockerfile translated into layers; the inspect output is the ordered digest list the runtime will actually merge. If a layer's digest is large and its Dockerfile line is COPY src/, you are paying to ship source code you could have excluded with .dockerignore — the diff semantics mean anything in the build context that reaches the layer ends up in the image forever.

The runtime view

• Images are ordered, immutable diffs; the runtime stacks them with overlayfs and copy-on-write, so writes never touch shared layers.
• Layer order is your build cache and your pull pipeline — put stable, large, rarely changing content at the bottom.
• Deleted files don't shrink the image; they hide in the diff. Multi-stage builds and .dockerignore are the real size controls.
• Image size is a proxy for pull time, disk usage, and cold-start cost — every megabyte you don't ship is a latency you don't pay.