Background Jobs & Message Queues Visualized: Workers, Retries & Dead Letters

Why did this job run twice? That question is more useful than asking whether the queue is reliable. A production queue is designed to survive crashes and lost acknowledgements, and redelivery is often how it keeps that promise.

Move waiting out of the request path

An API that generates a report, resizes images, sends email, and calls a slow partner before responding inherits every dependency's latency and failure rate. A producer can instead validate the request, durably enqueue intent, and return a job identifier.

const job = await queue.send({
  id: crypto.randomUUID(),
  type: "receipt.email",
  orderId,
  userId: session.userId,
});
return Response.json({ jobId: job.id }, { status: 202 });

Asynchronous does not mean unimportant. The queue becomes part of the product contract: users need status, failures need ownership, and urgent work may need a separate priority lane.

Producer, broker, consumer

The producer creates a message. The broker or queue stores and delivers it according to product-specific semantics. A consumer or worker reserves a delivery, performs the side effect, then acknowledges only after success.

If the worker crashes after sending an email but before acknowledging, the broker cannot know whether the side effect happened. It delivers again. This is why at-least-once systems require idempotent consumers.

INSERT INTO processed_jobs(job_id, result)
VALUES ($1, $2)
ON CONFLICT (job_id) DO NOTHING;

The deduplication record and business write should share a transaction when possible. An in-memory set is lost on restart and cannot coordinate multiple workers. The idempotency tutorial covers stored results and concurrent duplicates in depth.

Acknowledgements and visibility

Different systems use visibility timeouts, leases, explicit acknowledgement deadlines, or push-based delivery. The common idea is temporary ownership: if completion is not confirmed in time, another delivery becomes possible.

Choose the interval longer than normal processing but not so long that a crash stalls recovery. Long-running jobs often extend a lease or checkpoint work. A lease extension can also fail, so handlers still cannot assume exclusive execution forever.

Retry only transient failures

Network timeouts, 503 responses, and exhausted connection pools may recover. Invalid payloads, missing required data, and forbidden operations usually will not. Retrying a permanent failure creates a poison message and wastes capacity.

const delayMs = Math.min(60_000, 1_000 * 2 ** attempt);
const jittered = Math.random() * delayMs;
await retryLater(job, jittered);

Exponential backoff with jitter spreads retries instead of creating synchronized spikes. Bound attempts and total age. After the policy is exhausted, move the job to a dead-letter queue with enough metadata to diagnose it, but without secrets or unnecessary personal data.

Exactly once is a system-wide claim

A broker may provide exactly-once processing within a narrow protocol, yet an external email provider, payment API, or database can still observe a duplicate across a crash boundary. End-to-end exactly-once effects require coordination or idempotency at every relevant boundary. Treat broad exactly-once marketing claims with precise scope.

Concurrency and ordering

Adding workers raises throughput only until a downstream limit is reached. Ten workers can turn a mild database bottleneck into an outage. Apply concurrency limits per dependency and key when necessary.

Global order is expensive and rarely preserved by a pool of workers. If events for one account must remain ordered, partition by account ID and document what happens after a poison event. Do not assume FIFO enqueue order equals completion order.

Backpressure is information

Queue depth alone is incomplete. Track the age of the oldest ready job, arrival rate, completion rate, retry rate, failure reasons, and saturation. A stable depth of 10 may be healthy; 10 jobs waiting two hours may not be.

When producers exceed safe capacity, options include rejecting low-priority work, slowing producers, batching, coalescing duplicate jobs, scaling consumers, or shedding optional work. Unlimited buffering only postpones the failure while making recovery larger.

Cloudflare-style architectures

An edge handler can enqueue a task and return quickly, while queue consumers process batches elsewhere. Cloudflare Queues, managed cloud brokers, Kafka-like logs, and database-backed job tables do not share identical ordering, retry, retention, or acknowledgement behavior. Use the product's documented guarantees rather than this visual model.

Production checklist

• Give every logical job a stable identifier and make effects idempotent.
• Acknowledge only after durable success.
• Classify retryable and permanent errors.
• Bound retries, add jitter, and alert on dead-letter age.
• Measure queue age, depth, attempts, throughput, and worker saturation.
• Test crashes before, during, and after each side effect.
• Propagate a request or correlation ID into job telemetry; observability makes that path visible.

Key takeaways

• Queues trade synchronous latency for asynchronous state and operational responsibility.
• At-least-once delivery means duplicates are normal.
• Exactly-once effects are difficult across independent systems.
• Backoff protects dependencies; dead-letter queues isolate work that needs attention.
• The originating HTTP request and its security boundary still matter after work becomes asynchronous.