We had a performance problem.
One of the most frequently accessed pieces of data was being read far more often than it changed.
The database was handling a large number of repeated reads.
The obvious solution was caching.
We introduced a cache.
The performance improvement was immediate.
Database load dropped.
Response times improved.
The system could handle significantly more read traffic.
It looked like a successful optimization.
Then we discovered a different problem.
The system was fast.
But sometimes, it was wrong.
That changed the question from:
“How do we make this faster?”
to:
“Which data are we actually allowed to make stale?”
That turned out to be the more important architectural question.
1. The Performance Problem
The original request path was relatively simple:
Request
│
▼
Application
│
▼
Database
│
▼
Response
The problem was that the same data was being requested repeatedly.
A large number of requests were asking for information that had already been read moments earlier.
The database was doing work that wasn’t necessarily producing new information.
This created unnecessary pressure on the database:
- More queries
- More connection usage
- More CPU
- More I/O
- Higher latency during traffic spikes
The data was a good candidate for caching.
So we added a cache between the application and the database.
The new architecture looked like:
Request
│
▼
Application
│
▼
Cache
│
├──── Hit ────► Response
│
└──── Miss
│
▼
Database
│
▼
Cache
│
▼
Response
For reads that could tolerate some staleness, this was a straightforward improvement.
But not all data has the same correctness requirements.
That distinction became critical.
2. The Performance Improvement Was Real
The cache was doing exactly what we expected.
Requests that previously required database access could now be served directly from memory or a low-latency cache layer.
The database saw fewer repeated reads.
Application response times improved.
The database had more headroom for operations that actually required authoritative state.
From a performance perspective, the change was successful.
That is what made the eventual problem more interesting.
Nothing was obviously broken in the infrastructure.
The cache was healthy.
The database was healthy.
The application was healthy.
Requests were fast.
But some responses were based on information that was no longer current.
The system wasn’t failing loudly.
It was producing stale answers.
3. The First Sign of Trouble
The problem appeared when data changed.
Imagine a simplified flow:
Database
Balance = $1,000The application reads the value.
The cache stores:
Balance = $1,000Later, a transaction changes the authoritative state:
Database
Balance = $600But the cache still contains:
Cache
Balance = $1,000The next request hits the cache.
The application returns:
$1,000The system has just returned a value that is no longer authoritative.
Nothing crashed.
No exception was thrown.
The cache was functioning correctly.
The database was functioning correctly.
The application was functioning correctly.
The architecture was simply allowing stale data to be treated as current data.
That was the real problem.
4. Not All Stale Data Is Equally Dangerous
This was one of the most important lessons.
Caching isn’t inherently dangerous because cached data can become stale.
Staleness is sometimes acceptable.
Consider something like:
Product description
A few seconds of staleness may not matter.
The same could be true for:
Dashboard statistics
Reporting data
Exchange-rate display
Search results
Reference information
But now consider:
Available balance
Credit limit
Account status
Transaction state
Authorization decision
Risk decision
The consequences of stale data can be very different.
The architecture therefore needed to distinguish between:
Data that can be eventually consistent
and:
Data that must be authoritative at decision time
That distinction is more important than simply asking:
“Can we cache this?”
5. The Wrong Question About TTL
The first instinct when discussing stale data is often:
“What should the TTL be?”
Five seconds?
Thirty seconds?
One minute?
Ten minutes?
But TTL doesn’t answer the fundamental correctness question.
Suppose a cache has a TTL of five seconds.
That means the value can be stale for up to five seconds under the cache’s expiration model.
If five seconds of staleness is acceptable, that’s fine.
If the value is used to make a decision where even a brief stale period is unacceptable, then a five-second TTL isn’t a correctness guarantee.
It is simply a limit on one type of staleness window.
This distinction matters.
TTL is a performance and freshness mechanism. It is not automatically a consistency strategy.
6. The Dangerous Architecture
The problematic design looked conceptually like this:
┌─────────────┐
│ Application │
└──────┬──────┘
│
▼
┌────────┐
│ Cache │
└────┬───┘
│
▼
┌────────┐
│Database│
└────────┘
The application treated the cache as though it were authoritative.
That created a hidden assumption:
If the cache has a value, the value is safe to use.
That assumption wasn’t always true.
A cache is generally a copy of state.
The database, ledger, or authoritative service may own the actual state.
Once we recognized that distinction, the architecture became easier to reason about.
7. The Principle We Adopted
The key principle became:
Cache data for performance. Do not accidentally turn the cache into the source of truth.
That sounds obvious.
In practice, it changes how individual operations are designed.
For example, there is a significant difference between:
Read cached balance
and:
Determine whether this transaction is allowed
The first may tolerate cached data depending on the business requirement.
The second may require an authoritative check.
The application therefore needs to know which path it is taking.
8. Read Paths and Decision Paths
We started thinking about cached data in two categories.
Read path
The user wants to display information.
For example:
Account overview
Dashboard
Transaction history
Profile information
A small amount of staleness may be acceptable depending on the requirement.
The cache provides significant performance benefits.
Decision path
The system is about to make a state-changing or authorization decision.
For example:
Should this transaction proceed?
or:
Is this operation allowed under the current account state?
That is fundamentally different.
The system may need authoritative state before making the decision.
The architecture therefore becomes:
Request
│
┌─────────┴─────────┐
│ │
Read path Decision path
│ │
Cache Authoritative State
│ │
▼ ▼
Response Decision
This is much safer than treating every read as equivalent.
9. Cache Invalidation Became a First-Class Design Problem
Once data could change, we needed to think about what happened to the cached copy.
There are several common strategies.
Expiration
Allow entries to expire after a defined period.
Write
│
▼
Database
│
▼
Cache expires later
Simple, but it allows a freshness window.
Explicit invalidation
When authoritative data changes, remove the corresponding cache entry.
Write
│
▼
Database
│
▼
Invalidate Cache
This can provide better freshness, but creates another distributed interaction that can fail.
Cache update
Update the cache when the underlying state changes.
Write
│
├────► Database
│
└────► Cache
This can work well, but now both paths need to remain consistent.
Read-through / write-through patterns
The cache layer can participate more directly in reads and writes.
These approaches can simplify some flows but don’t eliminate the underlying consistency questions.
There is no universal answer.
The correct choice depends on what the data means and how stale it is allowed to become.
10. The Invalidation Race
One of the more subtle problems appeared when updates and reads happened concurrently.
Imagine:
T1: Read database
T2: Update database
T3: Write old value into cache
Now the database contains the new value.
But the cache contains the old value.
The system has created a stale cache entry even though the update path attempted to maintain the cache.
This is why cache invalidation is famously difficult.
It isn’t simply:
Update database
Delete cache
The timing between readers, writers, cache population, and invalidation matters.
Distributed systems introduce races that don’t always appear in a single-process implementation.
11. The Cache Failure Problem
There was another architectural question.
What happens when the cache itself fails?
A cache should normally improve performance.
It shouldn’t become a single point of failure for the entire application.
A common fallback is:
Application
│
▼
Cache
│
X
│
▼
Database
If the cache is unavailable, the application falls back to the authoritative data source.
That improves resilience.
But it creates another capacity problem.
Imagine the cache normally handles 90% of reads.
The database handles only the remaining 10%.
Now the cache fails.
Suddenly, the database may receive close to 100% of the read traffic.
That can create a secondary failure.
The system moves from:
Cache failure
to:
Database overload
to:
Application latency
to:
Request failures
Caching therefore needs a failure strategy, not just a hit-rate strategy.
12. Cache Stampedes
There was another pattern worth watching.
Suppose a frequently accessed cache entry expires.
Thousands of requests arrive at roughly the same time.
They all discover:
Cache miss
They all go to the database.
Instead of reducing database load, the cache expiration creates a sudden burst of database traffic.
Conceptually:
Cache Miss
│
┌────────────┼────────────┐
▼ ▼ ▼
Request Request Request
│ │ │
└────────────┼────────────┘
▼
Database
This is often called a cache stampede or thundering herd.
Possible mitigations include:
- Request coalescing
- Jittered expiration
- Background refresh
- Controlled refresh ownership
- Rate limiting
- Local caching
- Stale-while-revalidate strategies
Again, the important point isn’t memorizing a pattern.
It’s recognizing that a performance optimization can create a new load pattern when it fails or expires.
13. The Cache Became Part of the Architecture
Before introducing caching, the data flow was relatively easy to reason about:
Application → Database
After introducing caching:
Application
│
├──── Cache
│
└──── Database
That added another state-bearing component.
Now we had to answer:
- Who owns the data?
- How fresh must the cache be?
- When is the cache populated?
- When is it invalidated?
- What happens if invalidation fails?
- What happens if the cache is unavailable?
- What happens if the database changes outside the application?
- Can stale data be used for decisions?
- How do we observe cache inconsistencies?
The cache was no longer just a performance optimization.
It had become part of the system’s consistency model.
That was the architectural shift.
14. What We Changed
The solution wasn’t to remove caching.
The performance improvement was valuable.
Instead, we changed how we used it.
We established a few rules.
1. Define the source of truth
Every important piece of state needed a clearly identified authoritative owner.
The cache was not automatically that owner.
2. Classify data by freshness requirements
We explicitly asked:
Can this data be stale?
If yes:
How stale can it safely be?
And:
What happens if the cache is unavailable?
3. Keep authoritative decisions authoritative
Where correctness depended on the latest state, the application used the authoritative source rather than blindly trusting cached data.
4. Design invalidation deliberately
Cache invalidation was treated as an architectural concern rather than an implementation detail.
5. Monitor the cache
We cared about more than cache hit rate.
We also needed visibility into:
- Cache misses
- Evictions
- Latency
- Error rates
- Fallback traffic
- Refresh failures
- Database load during cache degradation
A cache can have an excellent hit rate and still be causing correctness problems.
15. The Trade-Off
Caching creates a very attractive trade-off:
Less database work
↓
Lower latency
↓
Higher throughput
But potentially:
Cached copy
↓
Stale state
↓
Incorrect decision
The goal isn’t to eliminate that trade-off.
The goal is to make it explicit.
For some data:
Performance > Immediate freshness
For other data:
Correctness > Cache latency
And for many systems, the answer is somewhere in between.
The mistake is assuming that the same consistency strategy applies to every piece of data.
16. When I Would Use a Cache
I would strongly consider caching when:
- The same data is read frequently
- The underlying data changes relatively infrequently
- Database reads are expensive
- The data has a defined freshness tolerance
- The system benefits from absorbing read traffic
- A fallback exists if the cache becomes unavailable
I would be much more cautious when caching:
- Authoritative balances
- Security or authorization state
- Transaction state
- Risk decisions
- Account status
- Any value where stale information can create an incorrect business outcome
That doesn’t mean these values can never be cached.
It means the correctness model needs to be explicit.
17. The Question I Ask Now
When someone proposes:
“Let’s put this in Redis.”
I don’t start with:
“How large should the cache be?”
I start with:
“What happens if this value is stale?”
Then:
“Who owns the authoritative state?”
Then:
“How quickly does the cached value need to converge?”
And:
“What happens if invalidation fails?”
Finally:
“Can a stale value cause a business decision that we cannot undo?”
Those questions tell us much more than cache hit rate ever will.
18. The Architectural Lesson
Caching didn’t break the system because caching is bad.
Caching exposed a distinction that was previously hidden:
Performance and correctness are different dimensions of system design.
A faster answer isn’t necessarily a correct answer.
A database query taking 20 milliseconds may be more valuable than a 1-millisecond cached response if the cached response can cause the wrong business decision.
The right architecture therefore isn’t:
“Cache everything possible.”
And it isn’t:
“Never cache important data.”
It is:
Cache aggressively where staleness is acceptable, and preserve authoritative reads where correctness requires current state.
Caching is a powerful performance tool.
But once cached data participates in a business decision, the cache is no longer just about performance.
It has become part of the correctness model.
Final Thought
The easiest architectural optimizations are often the ones that change only performance.
Caching is different.
It can change what the system knows at the moment it makes a decision.
That makes cache design much more than a question of latency and throughput.
Before adding a cache, ask:
What is the source of truth?
How stale can this data safely be?
How is invalidation handled?
What happens when the cache fails?
What happens during a cache stampede?
Can stale data create an irreversible business outcome?
If those answers are clear, caching can dramatically improve a system.
If they aren’t, you may successfully optimize the system into producing the wrong answer faster.
Performance is valuable.
But in systems where correctness matters, a fast wrong answer is still wrong.