The scheduler must accept jobs (one-off or recurring), run them at a deadline or after a delay, retry failures, and survive worker crashes. At 10M jobs/day the machine sustains ~116 jobs/sec on average and up to 10K/sec at batch-ingest peaks, with up to 50K pending jobs in the queue at any moment.
The core is the queue tier. Ready jobs live in a priority queue (Redis sorted set keyed by priority, or RabbitMQ/Kafka with priority support); delayed jobs live in a second sorted set scored by their run timestamp. A dispatcher loop polls the delayed set for jobs whose score has passed, promotes them to the ready queue, and hands ready jobs to workers. Promotion and lease must be atomic — a Lua script or a transactional broker operation — or two dispatchers hand out the same job.
Workers lease a job: the machine marks it claimed with a lease TTL (say 60s). If the worker finishes, it acks and the job is done. If it crashes, the lease expires and the job is re-enqueued — at-least-once delivery, so the job handler must be idempotent. Failures retry with exponential backoff (1s, 2s, 4s, 8s) up to 5 attempts, then land in a dead-letter queue for inspection.
Recurring jobs (cron-like) live in schedules(id, cron, handler); a scheduler process materializes each due occurrence into the delayed queue, catching up missed occurrences at the next tick bounded by a max-lag policy.
Data model: jobs(id, payload, priority, status, run_at, lease_until, attempts, max_attempts), delayed and ready as Redis sorted sets. The DB is the source of truth for status; the queues are the working set.
Bottlenecks: dispatcher single-thread contention (shard queues by job-type hash); a slow worker pool backing up ready-queue depth (scale workers on queue-depth metrics); a low-priority flood starving high-priority jobs (fair queuing by priority class).
API → enqueue → delayed set (score=run_at) → dispatcher (promote when due)
→ ready set (priority) → workers (lease w/ TTL)
worker ack → done | crash → lease expiry → re-enqueue → backoff → dead-letter