This is a system design question in miniature. The interviewer wants the full pipeline — enqueue, persist, dispatch, execute, retry, monitor — and evidence that you know where each piece can fail.
The mental model: a job is a message with a lifecycle; the queue is the source of truth, and the worker is a disposable executor.
Start at the producer: jobs go to a broker — Redis Streams, RabbitMQ, SQS, or a database-backed queue — rather than running inline, because inline work makes request latency hostage to slow jobs and a crash mid-job loses the work. The enqueue call should be synchronous and acknowledged. The payload should be small: an ID and parameters, not the data itself, because the worker re-reads current state at execution time — a payload with stale data is how jobs process against a world that no longer exists.
Workers pull jobs. The critical mechanism is the visibility timeout: a worker claims a job, processes it, and must delete it within the timeout, or the broker considers it failed and redelivers. That is how the system gets at-least-once delivery — and the cost is duplicates. A worker that crashes after doing the work but before deleting causes the job to run again, so jobs must be idempotent, exactly like webhook handlers.
Retries happen at two levels. The worker retries transient failures with backoff; the broker redelivers when visibility timeouts expire. After N attempts, the job goes to a dead-letter queue — a holding area that keeps poisoned messages (malformed payloads, permanent business failures) from blocking the main queue and where an alert fires. Without a DLQ, one bad job retries forever and stalls everything behind it.
Priorities and ordering matter. One FIFO queue means one slow batch job blocks a thousand fast ones — head-of-line blocking. Options: separate queues per priority with weighted polling, or a fair-share scheduler. Cron jobs are the same machinery with a scheduler emitting messages on a timer — scheduled, but with no guarantee of exact execution time, so cron-triggered work must tolerate drift.
Tradeoffs and edge cases: at-least-once is the default; exactly-once needs a dedupe store keyed by job ID, which is expensive and rarely worth it. Backpressure: if producers outpace workers, queue depth grows — you need monitoring on queue depth, worker lag, and visibility-timeout expiry rate. The classic silent failure: workers blocking on external calls with no timeout, inflating visibility windows and stalling the queue without any alert. Close with observability — queue depth and message age are the two metrics that show degradation before users do.