Batch jobs process a million rows where a request handler processes one. The trace follows one job — "recompute prices for every product" — from scheduler dispatch to completion, and shows why checkpointing turns a job that dies at 99% into a job that resumes, not restarts.
1. The scheduler dispatches the job
A cron trigger (see the cron trace for the scheduling mechanics) fires. The scheduler creates a job instance with a unique run ID and hands it to the worker. The job's first act is to define its working set: SELECT id FROM products WHERE status = 'active' — but it does not load a million rows into memory. It fetches a count or a max ID to establish bounds, then proceeds in chunks.
2. The job defines bounds and chunk size
A typical chunk: 500-1000 items, bounded by a key range or a cursor. The point of chunking is that a job that processes a million rows with 500-row chunks has 2000 discrete checkpoints — a restart loses at most one chunk, not the whole run. The chunk size is a transaction-size decision: each chunk becomes one transaction, so a large chunk means a long transaction, longer held locks, and more rows to redo on failure.
3. A chunk is fetched and processed
The worker queries WHERE id > $cursor ORDER BY id LIMIT 1000, loads the 1000 rows, and processes each item in the application: fetch the latest cost components, recompute price, write the new price to the staging table. Per-item cost is maybe 2-5ms of work plus a handful of queries. Per chunk: ~2-5s. The chunk's writes are staged, not yet committed.
4. The checkpoint commits progress
At the end of the chunk, the worker commits. In the same transaction — or a closely coupled one — it writes its checkpoint: the run ID, the last processed cursor (last_processed_id = 501000), and a count. This is the moment of truth for resume: the checkpoint row is durable state, the only record of "how far did we get." Committing every chunk is the classic trade: 2000 commits per job vs 1 — but each commit is cheap (single-row insert), and the failure story becomes trivial.
5. An item fails inside a chunk
Item 500 of a chunk throws — a malformed row, a race, an upstream API 500. The policy decision: fail the chunk (rollback all 1000, reschedule) or skip and record (write the failure to an error table, continue). The skip-and-record policy is what most production jobs do: the job finishes, and a repair run or an alert handles the error rows. The key rule is that the failure must be recorded — a job that silently skips has the same failure signature as a job that silently lies.
6. The job dies
The worker process is killed at 47% complete — a deploy, an OOM, a node eviction. The job's in-memory state vanishes; nothing was committed since the last checkpoint. From the scheduler's view, the job instance is gone mid-run: stale.
7. Resume reads the checkpoint and continues
The scheduler or operator restarts the job. Before processing, the new run reads the checkpoint: run_id = 487, last_processed_id = 501000, processed = 501000. It starts at WHERE id > 501000. The rework is bounded by one chunk's worth of work — a few seconds — instead of re-processing 470,000 rows. The restart is correct as long as item processing is idempotent, which is why batch jobs carry the same idempotency discipline as retries: a processed item recomputed twice must converge to the same result.
What checkpointing buys
The numbers tell the story. A job processing 1,000,000 rows at 300 rows/sec takes ~55 minutes. Without checkpoints, a crash at 50 minutes costs 55 minutes of rework. With 1000-row chunks and per-chunk checkpoints, a crash costs at most 3 seconds of rework plus a resume. The only way that math breaks is if item processing is not idempotent — then resuming is replaying side effects, and the job needs a dedupe table to make replay safe.