We had a performance problem.
So we did what seemed like the obvious thing.
We added more application capacity.
The system got slower.
At first, the numbers didn’t make sense.
The system was serving traffic across multiple regions, and workload had been steadily increasing. At peak periods, we were handling roughly 4,000 requests per second, with traffic distributed across multiple application instances.
As the workload increased, p95 latency started approaching 280 ms, with p99 occasionally reaching around 700 ms.
The first instinct was straightforward: scale horizontally.
We increased the application fleet from 12 instances to 24.
We expected the additional capacity to give us more headroom.
Instead, something unexpected happened.
Throughput increased by only around 20–25%, while p95 latency climbed from roughly 280 ms to more than 600 ms. Under heavier peaks, p99 latency approached 1.8 seconds.
We had doubled the application capacity.
The system had become slower.
That was the interesting part.
We were trying to increase throughput by adding application capacity.
Instead, we had significantly degraded response time without getting a proportional increase in throughput.
The problem wasn’t simply insufficient application capacity.
Our scaling strategy was increasing concurrency against a shared downstream resource faster than that resource could handle it.
1. The Symptom We Couldn’t Explain
The system was handling an increasing workload, and response times were starting to rise.
The first instinct was straightforward: scale horizontally.
We increased the number of application instances so incoming traffic could be distributed across more nodes.
On paper, the reasoning was sound.
More instances should mean:
- More concurrent requests
- More available processing capacity
- Better distribution of workload
- Higher throughput
But after the additional capacity was introduced, the behavior wasn’t what we expected.
Latency increased instead of decreasing.
Throughput improved only marginally.
We had doubled the application fleet, but throughput increased by only around 20–25%.
At the same time, p95 latency went from approximately 280 ms to more than 600 ms, while p99 approached 1.8 seconds during heavier peaks.
Adding capacity had not removed the bottleneck.
It had exposed it.
This was the point where the question changed from:
“How do we add more capacity?”
to:
“What is actually limiting the useful throughput of the system?”
That turned out to be the more important question.
2. What We Investigated First
The first step was to make sure we weren’t solving the wrong problem.
We looked at application CPU utilization.
It wasn’t showing the kind of saturation we would normally expect if the application tier itself were the bottleneck.
There was more compute available after scaling out.
We looked at memory and garbage collection.
Nothing immediately explained the increase in latency.
We looked at database performance.
The database wasn’t simply running out of CPU or storage capacity.
Database CPU had increased from roughly 55% to around 78%, but that alone didn’t explain why application latency had increased so significantly.
We also looked at network latency and other downstream dependencies.
There wasn’t one obvious infrastructure metric screaming:
“This is the problem.”
That was an important lesson by itself.
A system can be unhealthy even when the usual infrastructure dashboards look relatively healthy.
The key is understanding where requests are waiting.
CPU utilization tells you how much work is being performed.
Latency tells you that something is taking too long.
The gap between those two signals is often where the interesting problem is hiding.
3. The Bottleneck Was in the Connection Pool
The clue came from looking at the application more closely.
Each application instance maintained its own database connection pool.
When we increased the number of instances, we didn’t just increase application processing capacity.
We also increased the number of database connections that could be opened concurrently.
That changed the behavior of the entire system.
Before the scale-out, the 12 application instances collectively had the potential to establish roughly 300 database connections.
After scaling to 24 instances, that potential increased to roughly 600 connections.
The database, however, had not doubled its ability to process concurrent transactional work.
We had increased the number of clients competing for the same shared resource.
That distinction was critical.
The application tier had more capacity.
The database had not.
We started seeing connection-pool wait time increase during peak periods.
What had previously been negligible became noticeable, reaching roughly 150–250 ms for requests during the busiest periods.
The more important signal was that transactions were taking longer under increased concurrency.
Longer database operations meant connections were held for longer.
Connections being held longer meant fewer connections were available to other requests.
Requests then waited for connections.
Latency increased.
The architecture looked roughly like this:
┌───────────────┐
│ Load Balancer │
└───────┬───────┘
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
Application Application Application
Instance Instance Instance
│ │ │
Pool 1 Pool 2 Pool 3
│ │ │
└──────────────┼──────────────┘
│
▼
┌────────────┐
│ Database │
└────────────┘The important point was that the application instances weren’t independent.
They were all competing for a shared downstream resource.
Adding more application instances therefore increased concurrency against that resource.
More concurrency did not produce proportional throughput.
4. Why More Capacity Made Things Worse
This was the most important part of the investigation.
The system had created a feedback loop.
More instances meant more connection pools.
More connection pools meant more potential concurrent database work.
More concurrent work increased contention.
Higher contention increased transaction time.
Longer transactions held connections for longer.
Connections remained occupied.
Other requests waited for available connections.
Latency increased.
The simplified relationship looked like this:
More Application Instances
↓
More Connection Pools
↓
More Database Connections
↓
More Concurrent Database Work
↓
More Contention
↓
Longer Transactions
↓
Connections Held Longer
↓
More Requests Waiting
↓
Higher LatencyThis explained the numbers.
We had doubled application capacity, but throughput had increased by only around 20–25%.
The additional capacity wasn’t being converted into useful throughput.
Instead, part of it was being converted into additional contention.
That was the turning point in the investigation.
The important distinction was between capacity and useful capacity.
We had added more capacity at the application layer.
But the system as a whole had not gained the same amount of useful processing capacity because the database remained a shared constraint.
5. The Three Approaches We Considered
Once the bottleneck became clear, there were several possible responses.
Option 1: Add Even More Application Instances
This was the easiest option to propose.
If additional instances had helped before, perhaps adding even more would help again.
But the investigation had already shown why this was dangerous.
Every additional instance brought another connection pool.
Adding instances without controlling the total database concurrency would simply increase pressure on the bottleneck.
We would be increasing the ability to generate work while making the shared dependency work harder.
So we ruled this out.
Option 2: Reduce the Number of Instances
We could simply roll back to the previous number of instances.
That reduced the pressure on the database and improved the immediate situation.
But it wasn’t a real solution.
We still needed to handle increasing workload.
Reducing application capacity was useful as a stabilization measure, but it didn’t address the underlying scaling model.
Option 3: Control Concurrency at the Database Boundary
This was the direction that mattered.
Instead of asking:
“How many application instances can we run?”
we started asking:
“How much concurrent database work can the system safely generate?”
That changed the architecture discussion.
The connection pool was no longer treated as an instance-level tuning parameter in isolation.
It became part of the overall capacity model.
We had to consider:
- How many application instances could run?
- How large should each connection pool be?
- How much database concurrency could the workload sustain?
- How long were connections being held?
- Which operations genuinely required a database connection?
- What happened when the pool was exhausted?
The goal wasn’t to eliminate concurrency.
It was to keep concurrency within the capacity of the shared dependency.
6. What We Actually Changed
The solution wasn’t simply to reduce the number of application instances.
That would have given us immediate relief, but it wouldn’t have solved the underlying scalability problem.
We needed to control concurrency at the database boundary.
We adjusted the connection-pool configuration and application capacity together, rather than treating them as independent settings.
The important constraint became:
How much database concurrency can the system safely generate?
We reduced the amount of database concurrency each application instance could introduce and established a more deliberate relationship between application fleet size, connection-pool capacity, and database capacity.
We also examined how long connections were being held.
That led us to identify a few transaction paths where database connections were being retained longer than necessary.
Those paths were tightened so that connections were acquired closer to the point of database use and released as soon as the database work was complete.
The result was not simply a smaller connection pool.
It was a better capacity model.
Application instances could still scale horizontally, but they could no longer amplify database concurrency without limit.
After the change, peak p95 latency returned to roughly 250–300 ms, while p99 was brought back below approximately 800 ms under comparable workload.
More importantly, additional application capacity started producing useful throughput again instead of primarily increasing contention.
7. The Lesson: Capacity Is a Chain
One of the biggest lessons from this incident was that application capacity cannot be considered in isolation.
A request might pass through:
Client
↓
Load Balancer
↓
Application Instances
↓
Connection Pool
↓
Database
↓
External ServicesEvery layer has a capacity limit.
If the application tier can process 10,000 concurrent requests but the database can safely handle only a fraction of the resulting database workload, the application doesn’t really have 10,000 requests worth of useful capacity.
The real capacity of the system is constrained by the relevant shared dependencies.
This is why horizontal scaling can sometimes produce the opposite of the intended result.
You add capacity at one layer and increase pressure on another.
The system becomes faster at generating work than the downstream dependency is at consuming it.
That isn’t effective scaling.
It’s amplifying contention.
8. When Should You Suspect This Pattern?
There are several signals I now look for when adding instances doesn’t improve performance.
1. Latency increases after scaling out
If adding instances makes p95 or p99 latency worse, don’t immediately add more instances.
Ask what new concurrency those instances introduced.
2. Application CPU is relatively low
Low CPU doesn’t necessarily mean the system has spare capacity.
Requests may be waiting on database connections, locks, queues, or downstream services.
3. Connection-pool wait time increases
This is a particularly useful signal.
If requests are spending more time waiting to obtain a connection, the problem may be concurrency at the database boundary rather than application compute.
4. Database activity increases disproportionately
If adding 50% more application capacity results in a much larger increase in database contention, the architecture may be amplifying workload rather than scaling efficiently.
5. Database query or transaction time increases under higher concurrency
A query that performs well at moderate concurrency may behave very differently when many application instances execute it simultaneously.
6. Scaling out produces diminishing returns
If each additional group of instances provides less throughput improvement than the previous one, you may be approaching a shared-resource limit.
And if throughput eventually starts decreasing, you may already be beyond the useful concurrency level.
9. The Anti-Pattern: Scaling Each Layer Independently
A common mistake in distributed systems is to think about capacity one component at a time.
For example:
“The application is slow, so let’s add application instances.”
But application instances don’t operate in isolation.
Each instance may create:
- Database connections
- Threads
- Network connections
- Cache traffic
- Messages
- Requests to downstream services
Scaling one component can therefore multiply load somewhere else.
This is particularly important in financial systems, where transaction processing often involves several shared resources and where increasing concurrency without understanding downstream limits can turn a capacity problem into a stability problem.
The architecture has to be considered as a system.
Not as a collection of independently scalable boxes.
10. The Bigger Architecture Lesson
The most important lesson wasn’t about connection pools.
It was about how we think about scalability.
We often describe horizontal scaling as:
“Add more instances.”
That’s only part of the story.
Scalability is about how effectively a system can handle increasing workload.
Throughput is one important measure of that behavior.
Latency is another.
A system that processes more requests per second but pushes p99 latency from hundreds of milliseconds to several seconds may have increased raw throughput while delivering a significantly worse user experience.
The real goal is not simply:
More throughput.
It is:
More useful throughput while keeping latency, reliability, and resource consumption within acceptable limits.
Before scaling out, ask:
What additional work will each new instance generate?
Does it create more database connections?
More cache traffic?
More messages?
More calls to downstream services?
More lock contention?
More coordination?
More network traffic?
If every application instance introduces additional pressure on a shared resource, then the application isn’t scaling independently.
It is increasing contention on the next bottleneck.
That’s why the right question isn’t always:
“How many instances do we need?”
Sometimes the better question is:
“What is the maximum useful concurrency this system can sustain?”
Once you know that, scaling becomes a capacity-engineering problem rather than a simple infrastructure exercise.
Final Thought
The instinct to add capacity is usually a good one.
But capacity is not just the number of application instances.
A system is a chain of dependencies, and increasing capacity at one layer can increase pressure at another.
In this case, adding application instances increased database concurrency through independent connection pools.
The application had more capacity to generate work.
The database had not gained the same capacity to process it.
And that was enough to make the entire system slower.
The lesson I took away was simple:
Don’t just scale the component that looks busy. Understand how scaling that component changes the load on everything behind it.
Sometimes the fastest way to make a system faster is not to add more capacity.
It is to stop adding the wrong kind of capacity.