A downstream service started timing out.
So we did what seemed like the responsible thing.
We added retries.
The thinking was straightforward.
If a request fails because of a temporary network problem or a momentary downstream slowdown, try again.
The first retry succeeded often enough to make the approach look correct.
But then the system became slower.
As the downstream service became more overloaded, the number of retries increased.
Those retries generated additional traffic.
The additional traffic increased the load on the already struggling service.
That caused more timeouts.
Which caused more retries.
We had created a feedback loop.
The problem wasn’t that retries were inherently wrong.
The problem was that we were using retries without considering what happened to the system when the dependency was already under pressure.
That distinction changed how we thought about resilience.
1. The Failure Looked Temporary
The incident initially looked like a normal downstream reliability problem.
One of our services depended on another service to complete part of a request.
Under normal conditions, the interaction was straightforward:
Request
│
▼
Service A
│
│ API call
▼
Service B
│
▼
Response
Most requests completed within the expected latency range.
Occasionally, however, Service B became slow.
Some requests exceeded the configured timeout.
From Service A’s perspective, the request had failed.
The obvious response was to assume that the failure might be temporary.
So we introduced retries.
Conceptually:
Service A
│
├── Request ──► Service B
│
├── Timeout
│
├── Retry ────► Service B
│
└── Success
This worked in some cases.
A temporary network interruption could recover.
A short-lived dependency slowdown could clear.
A transient error didn’t necessarily need to become a user-visible failure.
The problem was that we were looking at the success rate of individual retries.
We weren’t looking at the effect of retries on the entire system.
2. The Numbers Started Moving in the Wrong Direction
As the downstream service experienced more load, its response times increased.
That caused more requests to reach the timeout threshold.
More timeouts triggered more retries.
The number of requests entering the downstream service therefore became larger than the number of original business requests.
This was the first important realization.
A retry isn’t free.
If 1,000 business requests arrive and some percentage are retried, the downstream system may actually receive significantly more than 1,000 requests.
The traffic pattern becomes:
Original Requests
│
▼
Service A
│
├──────────────► Service B
│
├── Retry ─────► Service B
│
└── Retry ─────► Service B
The dependency doesn’t know that these additional requests are attempts to recover from failure.
As far as it is concerned, they are more work.
And if the dependency is already struggling, adding more work is usually the opposite of what it needs.
3. The Retry Storm
The real problem became visible when we looked at the relationship between latency, timeouts, and retries.
The system was effectively doing this:
Dependency slows down
↓
Requests exceed timeout
↓
More retries
↓
More downstream traffic
↓
More dependency load
↓
Dependency slows down further
↓
More timeouts
↓
Even more retries
This is a classic failure-amplification pattern.
The original problem may have been relatively small.
The resilience mechanism turned it into a much larger problem.
This is why I now think about retries differently.
A retry isn’t simply an error-handling mechanism.
It is also a traffic-generation mechanism.
Every retry consumes resources somewhere.
It uses network capacity.
It consumes a thread.
It occupies a connection.
It may execute another database query.
It may acquire another lock.
It may trigger another downstream call.
And it keeps the original request alive for longer.
That last point is especially important.
4. Retries Also Increase Latency
Suppose a request normally takes 200 milliseconds.
Now the downstream dependency takes too long to respond.
The first attempt waits until the timeout.
Then the application retries.
The second attempt waits again.
If the retry succeeds, the overall request may still have taken several seconds.
From the user’s perspective, the request did not become resilient.
It became slow.
This creates another important relationship:
Retry
↓
More waiting
↓
Longer request lifetime
↓
More resources held
↓
Less available capacity
↓
More contention
For a highly concurrent system, request lifetime matters.
A thread waiting on a downstream call is still a thread being consumed.
A connection waiting on a slow operation is still a connection being occupied.
A request that stays alive for several seconds consumes resources for much longer than one that completes in a few hundred milliseconds.
So retries can increase both load and resource occupancy.
5. The First Approach: Add More Retries
The first option was the obvious one.
If transient failures were causing requests to fail, perhaps we needed more attempts.
For example:
Attempt 1
↓
Failure
↓
Attempt 2
↓
Failure
↓
Attempt 3
↓
Success
The problem was that this approach optimized for the success of an individual request.
It did not optimize for the health of the system.
If the dependency was genuinely unavailable, three attempts didn’t make it more available.
They simply created three times the potential work.
This led to a principle that became important in the design:
A retry policy should be designed around the failure mode of the dependency, not simply around the desire to make a request succeed.
If the dependency is experiencing a short transient failure, a retry can help.
If the dependency is overloaded, immediate retries can make the situation worse.
Those are two very different failure modes.
6. The Second Approach: Remove Retries
The opposite extreme was also tempting.
If retries could amplify failures, perhaps we should remove them completely.
That would certainly reduce the additional traffic.
But it would also turn transient failures into permanent failures.
A short network interruption could cause an otherwise recoverable business operation to fail.
That wasn’t acceptable for every interaction.
There are situations where retrying is exactly the right thing to do.
For example, if a downstream operation fails because of a brief network interruption, retrying after a short delay may allow the operation to succeed without involving the user.
So the lesson wasn’t:
Retries are bad.
It was:
Retries need boundaries.
7. The Third Approach: Controlled Retries
The better approach was to treat retries as a carefully controlled part of the architecture.
Instead of:
Failure → Retry immediately
we needed something closer to:
Failure
│
▼
Is this error retryable?
│
├── No ──► Fail
│
└── Yes
│
▼
Retry budget available?
│
├── No ──► Fail / fallback
│
└── Yes
│
▼
Backoff + jitter
│
▼
Retry
This changed the purpose of the retry mechanism.
We weren’t trying to make every request eventually succeed.
We were trying to recover from transient failures without overwhelming the dependency.
8. Not Every Error Should Be Retried
This sounds obvious, but it is one of the easiest mistakes to make.
A retry only makes sense if there is a reasonable expectation that another attempt could produce a different result.
A temporary network failure may be retryable.
A timeout may be retryable depending on the operation.
A service temporarily returning an overload response may require a carefully controlled retry.
But retrying a request because the input was invalid doesn’t make sense.
Neither does retrying a business rule failure.
For example:
Invalid Account
Insufficient Funds
Invalid Request
Unauthorized Request
Trying the same request again doesn’t change the underlying condition.
The retry policy therefore needs to understand more than HTTP status codes.
It needs to understand the semantics of the operation.
9. Backoff Changes the Behavior
One of the simplest improvements is to avoid retrying immediately.
Instead of:
Failure
↓
Retry immediately
↓
Failure
↓
Retry immediately
we introduce a delay:
Failure
↓
Wait
↓
Retry
↓
Wait longer
↓
Retry
This is the basic idea behind exponential backoff.
The delay between attempts increases.
For example:
Attempt 1 → failure
↓
100 ms
↓
Attempt 2 → failure
↓
200 ms
↓
Attempt 3 → failure
The exact values depend on the system.
The important idea is that retries should not all arrive at the dependency at the same time.
That leads to another important mechanism:
jitter.
Without jitter, thousands of clients can follow the same retry schedule.
They fail at approximately the same time.
They wait approximately the same amount of time.
Then they retry at approximately the same time.
That can create another traffic spike.
Jitter introduces controlled randomness into the delay.
Instead of:
1000 clients
│
▼
Retry at exactly 1 second
│
▼
Traffic spike
we want something closer to:
1000 clients
│
├── retry at 0.8s
├── retry at 1.1s
├── retry at 1.3s
├── retry at 0.9s
└── ...
The objective is to spread the recovery traffic.
10. Retry Budgets Matter
One of the most important changes in thinking was to stop treating retries as unlimited.
A service should have a defined tolerance for additional retry traffic.
Otherwise, every caller can independently decide:
“I’ll just try again.”
Imagine several upstream services calling the same dependency.
Without a coordinated policy:
Service A ──┐
Service B ──┼──► Dependency
Service C ──┤
Service D ──┘
If the dependency slows down, every service starts retrying.
The dependency then receives additional traffic from all of them.
This is how a relatively small failure can propagate through an entire architecture.
A retry budget provides a boundary.
Once the budget is exhausted, the system stops trying to force success.
It fails fast, falls back, queues the work, or returns an appropriate degraded response depending on the business operation.
Sometimes the correct resilient behavior is to stop calling the failing dependency.
11. Circuit Breakers Solve a Different Problem
Retries and circuit breakers are often discussed together, but they solve different problems.
A retry says:
“This failure may be temporary. Try again.”
A circuit breaker says:
“This dependency is failing badly enough that continuing to call it is no longer useful.”
Conceptually:
┌───────────────┐
│ Dependency │
└───────┬───────┘
│
Increasing
failures
│
▼
┌───────────────┐
│Circuit Breaker│
└───────┬───────┘
│
Stop sending
traffic
│
▼
Fallback / Fail
This is particularly important during dependency outages.
Without a circuit breaker, upstream services can continue generating traffic toward something that is already unavailable.
With a circuit breaker, the system can recognize the failure pattern and temporarily stop making calls.
That protects both the dependency and the callers.
12. Idempotency Changes the Retry Equation
In financial systems, there is another dimension that makes retries particularly interesting:
What happens if the first request actually succeeded, but the response was lost?
Consider a payment operation.
Client
│
│ Create Payment
▼
Payment Service
│
│ Payment succeeds
▼
Client
X
│
Response lost
From the client’s perspective, the request failed.
So it retries.
But the original payment may already have been processed.
Now the problem isn’t simply performance.
It is correctness.
This is where idempotency becomes critical.
A retryable financial operation often needs an idempotency mechanism so that repeated requests representing the same business operation do not create duplicate state changes.
Conceptually:
Request
Idempotency-Key: ABC123
│
▼
Payment Service
│
├── First request → Execute
│
└── Retry → Return existing result
The retry policy and the business semantics therefore cannot be designed independently.
A technically reliable retry mechanism can still create a business failure if the operation isn’t safe to repeat.
13. The Real Fix Was Not “Better Retries”
The most important lesson from the incident was that retries were not the root problem.
The root problem was that we had treated a distributed failure as though it were a local application error.
We had asked:
“How many times should we retry?”
We should have asked:
“What happens to the entire system when this dependency is slow?”
That changed the investigation.
We started looking at:
- Dependency latency
- Timeout rates
- Retry volume
- Connection usage
- Thread utilization
- Queue depth
- Request lifetime
- Downstream saturation
- Error rates
- Circuit-breaker state
The important metric wasn’t just:
How many requests succeeded?
It was:
How much additional work did our resilience mechanisms generate while the dependency was unhealthy?
That was the architectural question we had initially missed.
14. Resilience Can Amplify Failure
This is probably the biggest lesson I took from the problem.
We usually think about resilience mechanisms as protection:
Failure
↓
Retry
↓
Recovery
But in a distributed system, the real behavior can be:
Failure
↓
Retry
↓
More Load
↓
More Failure
↓
More Retry
↓
More Load
The mechanism intended to protect the system becomes part of the failure.
This pattern isn’t unique to retries.
The same principle applies to:
- Aggressive polling
- Automatic failover
- Cache refreshes
- Message redelivery
- Health checks
- Connection establishment
- Batch reprocessing
- Autoscaling
A resilience mechanism creates additional work.
That work has to be accounted for.
15. What I Would Do Differently
There are a few principles I would apply from the beginning.
1. Retry only operations that can reasonably recover
Don’t retry every error.
Understand what failed and whether another attempt can change the outcome.
2. Put a hard limit on retries
An operation should not be allowed to consume unlimited resources trying to succeed.
3. Use backoff
Immediate retries can turn one failure into a traffic spike.
4. Add jitter
Avoid synchronized retry waves from large numbers of clients.
5. Protect the dependency
A dependency that is already unhealthy doesn’t need more traffic from its callers.
6. Use circuit breakers where appropriate
Sometimes the most resilient action is to stop making calls.
7. Design for idempotency
Especially when retrying operations that change financial or business state.
8. Monitor retry traffic separately
If the system receives 100,000 business requests but generates 180,000 downstream attempts, that difference matters.
Retry traffic is part of your system load.
16. When Should You Suspect Retry Amplification?
There are several signals I now look for.
Latency increases together with retry volume.
If p95 or p99 latency rises and retries rise at the same time, investigate the relationship.
A downstream service becomes slower as upstream traffic increases.
The dependency may already be saturated.
The number of downstream requests is significantly higher than business requests.
This can indicate repeated attempts.
Timeouts trigger additional traffic.
This is a particularly dangerous feedback loop.
Multiple services retry the same dependency.
Independent retry policies can combine into a much larger load spike.
The dependency recovers when callers stop sending traffic.
That is a strong indication that the callers themselves are contributing to the overload.
Failures cascade across otherwise healthy services.
A single struggling dependency should not automatically cause the entire system to become unstable.
17. The Architectural Lesson
The lesson isn’t:
Don’t use retries.
Retries are an important part of resilient distributed systems.
The lesson is more subtle:
Every resilience mechanism changes system behavior under failure.
A retry changes traffic.
A timeout changes resource occupancy.
A circuit breaker changes availability.
A queue changes latency and delivery semantics.
A fallback changes correctness or user experience.
An autoscaler changes concurrency.
These mechanisms cannot be evaluated only by asking whether they work during normal operation.
They need to be evaluated under failure.
That is where architectural assumptions become visible.
18. Final Thought
Distributed systems don’t usually fail because one component stops working.
They fail because the rest of the system reacts to that failure in unexpected ways.
A slow dependency triggers retries.
Retries generate more traffic.
More traffic creates more contention.
Contention creates more latency.
More latency creates more timeouts.
More timeouts create more retries.
And suddenly the system is spending most of its resources trying to recover from the mechanism that was supposed to make it resilient.
That was the real lesson.
Resilience isn’t about making every request succeed.
It is about keeping the system healthy when some requests cannot succeed.
Sometimes that means retrying.
Sometimes it means waiting.
Sometimes it means falling back.
Sometimes it means putting the work on a queue.
And sometimes the most resilient thing a system can do is simply stop trying.
Good distributed-system design isn’t about eliminating failure.
It is about making sure that one failure doesn’t become everybody’s failure.