Fault-tolerant workers with visibility leases, exponential backoff, idempotency keys, dead-letter routing, heartbeat reclaim for crashed workers and queue-depth metrics.
Background job systems tend to fail in the same handful of ways: a worker dies mid-job and the job is lost, a retry runs the side effect twice, one poison message blocks a partition, and nobody notices the backlog until a user does. I built the platform to handle each of those explicitly rather than hope.
Producing services
Hand off work and know it will run exactly once, or land somewhere visible if it cannot.
Operators
See queue depth, retry rate and dead letters, and replay a fixed batch without a deploy.
A worker claims a job by taking a lease with a TTL rather than holding a lock. If the worker dies the lease simply expires and another worker picks the job up — no lock to release, no manual cleanup, and the failure mode is a delay rather than a lost job.
Every job carries a key. The handler checks it before performing the side effect and records it after, so an at-least-once delivery becomes an effectively-once outcome. Retries and duplicate submissions are both safe by construction.
Retries back off exponentially with jitter, so a downstream outage does not turn into a synchronised retry storm the moment it recovers. Attempts are bounded; what exhausts them is dead-lettered rather than retried forever.
Running jobs heartbeat to extend their lease. This separates 'slow' from 'dead': a long job keeps its claim, and a crashed worker's jobs return to the queue within one lease window instead of hanging until someone notices.
Unprocessable messages go to a dead-letter topic with their failure reason attached, keeping one bad payload from blocking a partition. A replay path pushes a corrected batch back through the normal flow after the fix.
Queue depth, lease expiry rate, retry rate, dead-letter rate and handler latency are exported. Depth trending up while throughput is flat is the signal that matters, and it is visible before users feel it.