The application was healthy.
CPU was normal.
Memory was stable.
The database was performing well.
Internal services were responding within their expected latency.
And yet one part of the system was getting slower.
Payment requests.
At first, we looked at our own infrastructure.
Nothing obvious was wrong.
Then we followed the request path further.
One of our critical workflows depended on a third-party API.
That API was getting slower.
Our system wasn’t failing because our code was broken.
It was failing because we had allowed an external dependency to become part of our critical execution path without designing enough isolation around it.
That was the lesson:
Every external dependency becomes part of your architecture whether you intended it or not.
1. The Symptom We Couldn’t Explain
The incident started with increasing payment latency.
Normally:
Payment request
↓
Internal processing
↓
Third-party verification
↓
Payment completed
~300ms
During the incident:
Payment request
↓
Internal processing
↓
Third-party verification
↓
...
↓
Payment completed
3–8 seconds
Our application servers weren’t overloaded.
The database wasn’t saturated.
Internal services looked healthy.
The third-party API was responding.
But it was responding slowly.
That distinction mattered.
The dependency wasn’t completely unavailable.
It was degraded.
And partial degradation can be more dangerous than a clean outage.
2. The Architecture Looked Reasonable
The workflow looked roughly like this:
┌───────────────┐
│ Client │
└───────┬───────┘
│
▼
┌───────────────┐
│ Payment API │
└───────┬───────┘
│
▼
┌───────────────┐
│ Payment │
│ Service │
└───────┬───────┘
│
▼
┌────────────────┐
│ Third-Party │
│ API │
└────────────────┘
The external service performed an important part of the workflow.
That seemed reasonable.
The mistake wasn’t using the third-party service.
The mistake was treating it like an internal service.
Internal services were under our control.
The external provider wasn’t.
We controlled:
- Our timeout configuration
- Our retry behavior
- Our connection pools
- Our concurrency
- Our fallback behavior
- Our traffic patterns
We did not control:
- Their latency
- Their availability
- Their capacity
- Their rate limits
- Their deployments
- Their incidents
- Their network path
That difference should have been reflected in the architecture.
3. The First Mistake: Assuming Availability
The original design implicitly assumed:
Third-party API
↓
Available
↓
Responds quickly
↓
Workflow continues
But external dependencies don’t work that way.
The real model is:
Third-party API
│
├── Fast
├── Slow
├── Rate limited
├── Partially unavailable
├── Timeout
├── Incorrect response
└── Completely unavailable
Our architecture had been designed around the happy path.
Production forced us to design around the failure modes.
4. Slow Is a Failure Mode
One of the most important lessons was that a timeout isn’t the only problem.
Suppose our normal dependency latency is:
100ms
Then it becomes:
500ms
Then:
2 seconds
The service is still technically available.
But every request now occupies application resources for much longer.
Consider:
100 requests/second
At 100ms:
~10 concurrent requests
At 2 seconds:
~200 concurrent requests
The external service didn’t necessarily cause an application crash.
It increased the amount of concurrency our application needed to handle.
That can eventually exhaust:
- Threads
- Connections
- Worker pools
- Memory
- Queues
- File descriptors
- HTTP client resources
The bottleneck can propagate backward through the system.
5. The Failure Amplification Loop
The architecture started behaving like this:
Third-party latency increases
↓
Requests take longer
↓
More requests remain in flight
↓
Connection pools fill
↓
Application queues grow
↓
Application latency increases
↓
Clients retry
↓
More requests reach the system
↓
Third-party traffic increases
The system had created a feedback loop.
The original problem was:
Third-party API is slow
But the resulting problem became:
Our entire application is slow
This is one of the most dangerous characteristics of external dependencies.
A small degradation downstream can become a large failure upstream.
6. The Three Approaches We Considered
Once we identified the dependency, we considered several options.
Option 1: Increase the Timeout
The simplest response was to give the third-party API more time.
For example:
Current timeout: 2 seconds
New timeout: 10 seconds
That would reduce some timeout errors.
But it would make resource exhaustion worse.
A request waiting for 10 seconds holds resources for 10 seconds.
We would be turning:
Slow dependency
into:
More long-lived requests
That wasn’t resilience.
It was waiting longer.
Option 2: Add More Application Capacity
We could add more application instances.
That would increase the number of requests our system could process concurrently.
But the external dependency remained unchanged.
We would simply generate more traffic toward the same bottleneck.
We had seen this mistake before.
Scaling the caller doesn’t necessarily scale the dependency.
Option 3: Isolate the Dependency
This was the better approach.
We needed to treat the external API as an unreliable boundary.
That meant introducing:
- Timeouts
- Circuit breakers
- Rate limiting
- Bulkheads
- Carefully controlled retries
- Fallback behavior
- Dependency-specific monitoring
The objective wasn’t to make the external service reliable.
We couldn’t.
The objective was to prevent its failure from becoming our failure.
7. Timeouts Became an Architectural Boundary
The first change was establishing explicit timeouts.
Before:
HTTP request
↓
Wait...
↓
Wait...
↓
Wait...
There wasn’t a strong upper bound on how long the request could occupy resources.
After:
Request
│
▼
Third-Party API
│
├── Success → Continue
│
└── Timeout → Controlled failure
A timeout is not simply an HTTP configuration.
It defines how long our architecture is willing to wait for another system.
That makes it an architectural decision.
8. But Timeouts Alone Were Not Enough
Suppose the dependency fails.
Every request hits it.
Every request waits for the timeout.
Then another request arrives.
It waits.
And another.
Eventually:
Dependency unavailable
↓
Every request waits
↓
Resources exhausted
↓
Application unavailable
We needed a way to stop repeatedly calling a dependency that was clearly failing.
That led to the circuit breaker.
9. The Circuit Breaker
The circuit breaker introduced three conceptual states:
┌─────────┐
│ CLOSED │
└────┬────┘
│ failures increase
▼
┌─────────┐
│ OPEN │
└────┬────┘
│ after recovery period
▼
┌─────────┐
│HALF-OPEN│
└────┬────┘
│
┌──────┴──────┐
▼ ▼
Success Failure
│ │
▼ ▼
CLOSED OPEN
When the dependency was healthy:
Application → Third Party
When it became unhealthy:
Application
│
X
│
Circuit Breaker
│
└── Controlled fallback
Instead of continuously sending requests into a failing dependency, we stopped the traffic temporarily.
This protected both systems.
10. Rate Limits Changed the Problem
The third-party provider also had rate limits.
That introduced another failure mode:
Our traffic
↓
Provider rate limit
↓
429 responses
The dangerous reaction would be:
429
↓
Retry
↓
429
↓
Retry
↓
429
That can turn a rate limit into a retry storm.
We therefore treated rate limits as part of the dependency contract.
We needed to know:
- Allowed request rate
- Burst limits
- Quotas
- Per-customer limits
- Per-account limits
- Provider response semantics
- Retry guidance
Then we enforced appropriate limits on our side.
11. Bulkheads Prevented One Dependency From Taking Down Everything
Another problem was shared resource pools.
Suppose the payment service handles:
Payments
Fraud checks
Customer verification
Reporting
If all external calls share the same worker pool, a slow provider can consume most of the available capacity.
Then:
Third-party slowdown
↓
Workers occupied
↓
Fraud requests wait
↓
Payment requests wait
↓
Other workflows wait
The external dependency has effectively become a system-wide bottleneck.
Bulkheads change that.
We can isolate capacity:
Payment
│
└── Payment dependency pool
Fraud
│
└── Fraud dependency pool
Verification
│
└── Verification dependency pool
If one dependency becomes unhealthy, it consumes only its allocated capacity.
The rest of the system can continue operating.
12. Fallbacks Required Business Decisions
Fallbacks sound simple.
But in financial systems, they’re not.
Suppose the external provider is responsible for a fraud decision.
What should happen if it is unavailable?
Possible answers include:
Approve
Reject
Queue for later
Require manual review
Return "processing"
There is no universal technical answer.
The correct decision depends on the business risk.
For example:
Low-risk operation
↓
Fallback may be acceptable
But:
High-risk financial operation
↓
Fallback may need to fail safely
The important point is that fallback behavior cannot be invented during an incident.
It needs to be designed beforehand.
13. Not Everything Should Be Retried
We also reviewed retries carefully.
A retry can be useful when:
Temporary network failure
But dangerous when:
Provider is overloaded
or:
Request may already have succeeded
For a financial operation, this is especially important.
Imagine:
Our request
↓
Provider processes payment
↓
Provider succeeds
↓
Network response is lost
Our application sees:
Timeout
If we blindly retry:
Retry
↓
Provider processes another request
we may create a duplicate financial operation.
This is why retry behavior must be designed together with idempotency.
14. Idempotency Became Part of the Dependency Contract
For operations that can change financial state, we needed idempotency.
For example:
idempotency-key: payment-847291
If the same request is submitted again:
Request #1 → payment-847291
Request #2 → payment-847291
Request #3 → payment-847291
the system should recognize that these represent the same logical operation.
The goal is:
3 attempts
↓
1 financial effect
rather than:
3 attempts
↓
3 financial effects
This becomes particularly important when external dependencies are unreliable.
15. Vendor Outages Became a Normal Failure Mode
Eventually, we stopped thinking about vendor outages as exceptional events.
They were possible states of the system.
The dependency could be:
Healthy
Degraded
Rate-limited
Unavailable
Recovering
Our architecture needed to behave sensibly in each state.
That led to dependency-specific runbooks.
For example:
Dependency degraded
↓
Observe latency
↓
Reduce concurrency
↓
Circuit breaker if thresholds exceeded
↓
Activate fallback
↓
Monitor recovery
↓
Gradually restore traffic
We didn’t want the incident response to begin with:
“What do we do?”
We wanted it to begin with:
“Follow the known failure mode.”
16. We Started Monitoring the Dependency Separately
Previously, we monitored our own service:
CPU
Memory
Latency
Errors
We added dependency-specific metrics:
Dependency latency
Dependency error rate
Timeout rate
429 rate
Circuit breaker state
In-flight requests
Fallback rate
Retry rate
Provider availability
This made the dependency visible as a first-class architectural component.
Even though it wasn’t running in our infrastructure.
17. Dependency Health Became Part of Our Health
We eventually changed the mental model from:
Our application
to:
Our application
+
Dependencies
+
Network
+
External providers
The customer experiences the entire chain.
They don’t care whether the latency came from:
Our code
or:
Someone else's API
The request is simply slow.
Therefore, dependency health is part of our effective system health.
18. The Result Wasn’t “The Vendor Became Reliable”
The external provider didn’t suddenly become perfect.
Instead, our system became more resilient to its behavior.
Before:
Vendor slowdown
↓
Application slowdown
↓
Connection exhaustion
↓
Customer impact
After:
Vendor slowdown
↓
Timeout / circuit breaker
↓
Bulkhead isolation
↓
Controlled fallback
↓
Reduced blast radius
That’s an important distinction.
Resilience doesn’t require every dependency to be reliable.
It requires the system to remain within acceptable behavior when dependencies aren’t.
19. What We Would Not Do
We wouldn’t assume an external API is always available.
We wouldn’t set extremely long timeouts simply to reduce error rates.
We wouldn’t retry every failure automatically.
We wouldn’t let retries ignore rate limits.
We wouldn’t share unlimited concurrency across unrelated dependencies.
We wouldn’t allow one vendor’s outage to consume the entire application’s worker pool.
We wouldn’t design fallbacks without understanding their business consequences.
And we wouldn’t make a third-party dependency critical without defining what happens when it isn’t available.
20. The Questions I Ask Now
Whenever a system introduces an external dependency, I ask:
What happens when it becomes slow?
Then:
What happens when it times out?
Then:
What happens when it returns errors?
Then:
What happens when it rate-limits us?
Then:
What happens when it succeeds but our response is lost?
Then:
Can one dependency consume all of our resources?
And finally:
Can the rest of the system continue operating if this dependency disappears?
If the answer is no, that dependency isn’t just an integration.
It’s an architectural dependency.
21. The Bigger Architectural Pattern
Every external dependency creates another boundary:
Our System
│
┌────────────┼────────────┐
│ │ │
▼ ▼ ▼
Database Message Bus External API
│
▼
Vendor System
Each boundary introduces uncertainty.
The dependency may have:
- Different latency
- Different availability
- Different scaling characteristics
- Different rate limits
- Different failure semantics
- Different deployment schedules
- Different operational practices
We cannot eliminate those differences.
We can design around them.
22. The Architectural Lesson
One of the most important lessons was this:
An external dependency is still part of your system from the customer’s perspective.
You may not own the server.
You may not control the network.
You may not control the deployment.
You may not control the provider’s capacity.
But if your application depends on it to complete a customer transaction, its behavior becomes part of your architecture.
That means it deserves:
- Capacity analysis
- Failure analysis
- Monitoring
- Timeouts
- Isolation
- Recovery procedures
- Clear ownership
- Explicit business fallbacks
23. Final Thought
The original mistake wasn’t choosing a third-party provider.
Using external services can be an excellent architectural decision.
It can reduce development time.
It can provide capabilities that would be expensive to build internally.
It can let engineering teams focus on their core business.
The mistake was assuming that because the provider was external, its failures were external to our architecture.
They weren’t.
When that API became slow, our customers experienced the slowdown as our problem.
That changed how we thought about dependencies.
A third-party API isn’t just an integration sitting at the edge of your architecture.
If a critical workflow depends on it, it is part of your architecture.
So design the boundary accordingly.
Set the timeout.
Control the retries.
Protect the resources.
Define the fallback.
Isolate the dependency.
Monitor its behavior.
And know what happens when it disappears.
Every external dependency becomes part of your architecture whether you intended it or not.