Asynchronous processing is an intriguing way of thinking about processes that can be delayed. A basic background worker can handle heavy work while your users enjoy the fast processing of their critical transactions. But what if a request sent to the background worker fails? Ignoring it is not an option - the user's gotta have their receipt generated, their report processed, or their image compressed.

You might think a retry will solve the issue! But wait - what if the PDF or report-generation service is down? Even a thousand retries won't solve the issue. And what if the service came back to life only to find every waiting client retrying at once? The retries themselves become the flood - overwhelming the service and potentially bringing it down again.

This post is about solving those issues, with three cheap mechanisms that turn retries from a liability back into the safety net they were supposed to be: backoff, jitter, and the dead letter queue.

Realizing the Problem

Picture an image-compression service, with a pool of clients sending it work. The service has a fixed capacity - say 50 concurrent requests - and a published availability of 95%, meaning the service goes down roughly 5% of the time.

Those brief outages are the source of retry storms: the service comes back up, and every client that was waiting hammers it at once with the requests that failed while it was down. The worker gets bottlenecked right at the moment it's trying to recover.

See It Happen

Choose one of the presets or tinker with the settings as you like, then press Start to kick off the simulation. More on what each preset does below.

Presets
Service configuration
Retry strategy
Retries
Backoff
Jitter
Dead letter queue
Fixed model parameters
Tick100msSend window30sPoison rate5%Base retry delay200ms
Client pool1,000users
PDF serviceIdle
Load: 0/50
Completed0
Dead letter queue0
Lost0
Speed
Success rate—
Avg response time—
Retry storms0
Service load0/50
DLQ size0
Total processed0
Time elapsed0.0s
See in full page
95%Typical availability target
0.5-2sA single outage burst
50Concurrent capacity
  1. No protection - no retries, no safety net. When the worker goes down, every request in flight is lost forever.
  2. Backoff only - adding retries with exponential backoff meaningfully reduces the number of lost requests.
  3. Full resilience - jitter spreads the retries out, improving recovery even further. With the DLQ enabled, nothing is lost - failed requests are held for investigation later.
  4. Overwhelmed - shows how when the load is overwhelming, even retries won't help that much. Read below to understand why.

Fix One: Backoff

The first instinct is to retry failed requests so they're not lost - but a retry with just a fixed delay (say 200ms) can make things worse. When the service goes down, lots of requests fail together, and with a fixed delay they all retry together - hammering the freshly recovered service at the exact moment it's trying to stand back up.

Backoff spaces successive attempts apart. The two common ways are:

Linear

Delay grows by a fixed step: 200ms, 400ms, 600ms, 800ms.

Exponential

Delay doubles each attempt: 200ms, 400ms, 800ms, 1600ms - giving a struggling service more room to recover.

Fix Two: Jitter

Jitter is the missing half of backoff. Even with exponential delays, every client computes the same schedule - so their retries still land at the same moments, just further apart. Jitter adds a random offset to each delay, breaking that synchronization.

For example, instead of waiting exactly delay, a client waits delay + random(0, delay/2).

Fix Three: Know When to Stop

Backoff and jitter handle the transient failures - but some requests can never succeed. A malformed payload, a reference to a deleted record, a bug in the data - these will fail on attempt one and attempt one hundred the same way. The simulator models this as poison messages: a small fraction of requests that fail regardless of service health. If your retry policy is "try forever," every poison message becomes a permanent source of load.

This is what maxRetries and the dead letter queue are for:

  • After a bounded number of attempts, stop retrying.
  • Route the exhausted message to a DLQ - a holding area where it stops consuming capacity and waits for a human or a separate process to inspect it.

Putting It Together

Try the Overwhelmed preset last: it throws a much larger pool of users against the same small service - with backoff and jitter enabled. This is a different failure. The service is simply undersized for the load, and even a solid retry policy can't save it. The retries themselves become part of the problem, and the storm is near-constant. The lesson from this is that resilience patterns protect a correctly sized service through transient failures. They are not a substitute for capacity.

❖

The full recipe is:

BackoffReduce the retry rate
JitterSpread retries across time
Max retriesStop after a few tries
DLQHold failed messages for review

None of this is complex to implement. Most queue clients and HTTP libraries already ship with all four knobs; the failure is almost always that the defaults were left naive.