The payment was successful.
The ledger had been updated.
The transaction had completed.
Everything looked correct from the backend.
And yet the customer opened the app and saw the old balance.
They refreshed.
Still the old balance.
They checked again.
Nothing changed.
From the system’s perspective, the payment was complete.
From the customer’s perspective:
The money had disappeared.
That was the moment we learned an important lesson about distributed systems:
A system can be technically correct and still be wrong from the customer’s perspective.
The problem wasn’t that eventual consistency was inherently broken.
The problem was that we had allowed a consistency boundary to become a customer experience problem.
1. The Symptom We Couldn’t Explain
The incident looked simple.
A customer initiated a payment.
Customer
│
▼
Payment Service
│
▼
Ledger
The payment succeeded.
The ledger showed the transaction.
But the customer-facing balance showed the previous value.
For example:
Before payment:
Balance = $1,000
Customer makes a:
$200 payment
The ledger correctly records:
Balance = $800
But the application displays:
Balance = $1,000
for several seconds.
Technically, the system was functioning as designed.
Operationally, everything was healthy.
But the customer didn’t care about the architecture.
They saw:
“I made a payment, but my balance didn’t change.”
And that felt like a broken financial system.
2. The Architecture Looked Reasonable
The system had evolved toward an event-driven architecture.
The payment service handled the transaction.
The ledger remained the authoritative source of financial state.
Other services consumed ledger events and built their own read models.
Conceptually:
┌───────────────┐
│ Payment │
│ Service │
└───────┬───────┘
│
▼
┌───────────────┐
│ Ledger │
│ Source of │
│ Truth │
└───────┬───────┘
│
Event
│
▼
┌───────────────┐
│ Balance │
│ Read Model │
└───────────────┘
│
▼
Customer
The design had an intentional separation.
The ledger was authoritative.
The balance displayed by the application was a derived representation.
That gave us scalability and decoupling.
It also created a delay between:
Financial state changed
and:
Customer-visible state changed
That delay was the problem.
3. The First Mistake: Calling It “Just Eventual Consistency”
The first reaction was:
“That’s expected. The system is eventually consistent.”
Technically, that statement was correct.
But it wasn’t a sufficient answer.
The customer doesn’t experience:
eventual consistency
They experience:
My balance is wrong.
This distinction matters.
An architecture can tolerate eventual consistency internally while still requiring strong consistency at specific user-facing boundaries.
We had treated the consistency model as a technical implementation detail.
It was actually part of the product behavior.
4. What Eventual Consistency Actually Meant
The system effectively had two states:
Authoritative State
│
▼
Ledger
│
│ event
▼
Read Model
│
▼
Customer UI
The ledger might update at:
T+0ms
The balance read model might update at:
T+200ms
or:
T+2s
or, under load:
T+10s
The architecture remained eventually consistent as long as the read model eventually converged.
But the customer’s expectation was different.
They expected:
Payment completed
↓
Balance reflects payment
almost immediately.
That expectation became part of the consistency requirement.
5. The Real Question Wasn’t “Is It Consistent?”
We changed the question.
Instead of asking:
“Is the system eventually consistent?”
we asked:
“Where does the business require consistency?”
That was much more useful.
Not every piece of information needs the same consistency guarantee.
For example:
Fraud analytics
→ seconds may be acceptable
Marketing dashboard
→ minutes may be acceptable
Search index
→ seconds may be acceptable
Customer balance
→ potentially unacceptable
The architecture needed different consistency boundaries for different workflows.
6. The Balance Was Different From Other Read Models
This was one of the most important discoveries.
We had treated the customer balance like another derived read model.
But a balance is not merely informational.
It can influence:
- Whether a customer can spend money
- Whether a transfer is allowed
- Whether a withdrawal is permitted
- Whether a payment is accepted
- Whether a financial decision can be made
That makes balance a much more sensitive piece of state.
The architecture needed to distinguish:
Convenience data
from:
Financially authoritative state
Those aren’t always the same thing.
7. We Defined the Source of Truth
We explicitly established:
The ledger is authoritative for financial state.
The balance read model could be derived from it.
But it could not become an independent source of truth.
This distinction prevented a dangerous architectural mistake.
We didn’t want:
Ledger says:
$800
Balance service says:
$1,000
and then have the system decide that both values were equally authoritative.
There had to be one canonical financial state.
The read model was a representation.
Not the source of truth.
8. Then We Had to Decide What the Customer Should See
There were several possible approaches.
Option 1: Wait for the Read Model
The simplest approach was to make the UI wait.
Payment
↓
Ledger
↓
Event
↓
Balance Read Model
↓
UI updates
This preserved the architecture.
But it could make the user experience feel slow.
Option 2: Read the Ledger Directly
Another option was to retrieve the balance directly from the authoritative system.
This could improve correctness at the boundary.
But putting more synchronous traffic on the ledger could increase load.
Option 3: Return the Updated Balance From the Transaction
The payment operation could return enough authoritative information for the customer experience to immediately reflect the new state.
For example:
{
"paymentId": "pay-123",
"status": "COMPLETED",
"availableBalance": 800
}
The UI could immediately display the value returned from the authoritative transaction.
Option 4: Show State Explicitly
Another option was to make the temporary state visible:
Payment completed
Balance updating...
This is often better than showing a confidently incorrect number.
9. The Important Principle: Don’t Lie to the User
This became one of our strongest principles.
If the system knows:
Payment completed
Balance projection pending
then showing:
Balance: $1,000
without context can be misleading.
A better experience might be:
Available balance: $800
Balance updated
or, during a short propagation window:
Payment completed
Your balance is being updated.
The user doesn’t need to know about Kafka, projections, consumer offsets, or replication.
But the system needs to communicate the state honestly.
10. Read Models Are Powerful
The reason we used a read model in the first place was scalability.
Instead of making every screen query the transactional ledger directly, we could create optimized representations.
For example:
Ledger
│
├──► Balance View
├──► Transaction History
├──► Reporting View
├──► Analytics View
└──► Search View
Each representation could be optimized for a different access pattern.
This is one of the strengths of event-driven architecture.
But every derived representation creates a consistency boundary.
The more read models we create, the more carefully we need to decide:
How stale is acceptable?
11. We Started Measuring Staleness
Previously, we monitored:
API latency
Error rate
CPU
Memory
Queue depth
We added:
Read-model lag
For example:
Ledger event
│
│ 150ms
▼
Balance projection
We could now measure:
p50 projection lag
p95 projection lag
p99 projection lag
maximum lag
number of stale accounts
This transformed eventual consistency from an abstract architectural concept into an observable system property.
12. The Problem Was Worse Under Load
At low traffic, the system looked perfect.
Ledger update
↓
Event
↓
Read model
~100ms
Nobody noticed.
Under higher traffic:
Ledger update
↓
Event
↓
Queue backlog
↓
Consumer delay
↓
Read model
The delay became:
5 seconds
Then:
20 seconds
The architecture hadn’t changed.
The workload had.
This taught us something important:
A consistency guarantee that works at low load may become a customer-visible problem when the system is under pressure.
13. Reconciliation Became Essential
Because read models are derived, we needed a way to detect divergence.
We periodically compared:
Authoritative Ledger
vs
Derived Read Model
For example:
Account 123
Ledger:
$800
Read Model:
$800
Status:
MATCH
But if:
Ledger:
$800
Read Model:
$1,000
Status:
MISMATCH
we needed to know immediately.
Reconciliation wasn’t a sign that the architecture had failed.
It was part of operating a distributed system responsibly.
14. Reconciliation Is Especially Important in Financial Systems
Financial systems cannot simply assume:
“It will eventually become correct.”
Eventually is not a control mechanism.
We need to know:
- What changed?
- What was processed?
- What was missed?
- What diverged?
- How long did the divergence last?
- Was customer-visible state affected?
- Can the state be repaired automatically?
A reconciliation process gives us confidence that derived state remains aligned with authoritative state.
15. Consistency Boundaries Became Explicit
We eventually started documenting consistency requirements per workflow.
For example:
| Workflow | Consistency Requirement |
|---|---|
| Ledger update | Strong / transactional |
| Payment authorization | Strong |
| Available balance used for authorization | Strong or authoritative read |
| Customer balance display | Near-real-time |
| Transaction history | Near-real-time |
| Fraud analytics | Eventual |
| Reporting | Eventual |
| Marketing analytics | Eventual |
The exact requirements depend on the system.
The important thing is that they are explicit.
16. The Dangerous Architecture
The dangerous design looked like this:
Ledger
│
▼
Event Stream
│
▼
Read Model
│
▼
Customer
with an unstated assumption:
“The read model will always be fresh enough.”
That’s not a consistency strategy.
It’s a hope.
A better design asks:
What happens if the read model is:
100ms behind?
5 seconds behind?
5 minutes behind?
And:
At what point does staleness become a business incident?
17. We Added Business-Level Guarantees
Technical guarantees were not enough.
We needed statements such as:
After a successful payment, the customer-visible balance should reflect the transaction within X seconds.
Now we had something measurable.
We could create an alert when:
Projection lag > threshold
And a stronger alert when:
Customer-visible financial state
diverges from authoritative state
This connected architecture to customer experience.
18. We Also Had to Handle Unknown States
Another important case was:
Payment request
↓
Network timeout
Did the payment succeed?
We might not know immediately.
The UI cannot safely assume:
FAILED
just because the request timed out.
Nor should it assume:
SUCCESS
without confirmation.
Sometimes the correct state is:
PROCESSING
or:
STATUS UNKNOWN
followed by reconciliation or status lookup.
This is particularly important for financial operations.
19. Eventual Consistency Requires Product Decisions
This was one of the biggest lessons.
Engineers can say:
“The system is eventually consistent.”
But product requirements determine:
How eventual is acceptable?
For some workflows:
500ms
is fine.
For others:
5 seconds
is unacceptable.
For financial authorization:
Incorrect state
may be unacceptable regardless of latency.
Consistency is therefore not only a database decision.
It’s a product and business decision.
20. What We Actually Changed
We didn’t eliminate eventual consistency.
That would have been the wrong conclusion.
Instead, we made it explicit.
We:
- Defined authoritative sources
- Documented consistency boundaries
- Measured projection lag
- Added reconciliation
- Improved consumer monitoring
- Designed customer-facing states carefully
- Used authoritative reads where necessary
- Returned authoritative transaction results where appropriate
- Added alerts for dangerous divergence
- Distinguished financial state from derived views
The architecture remained distributed.
But the boundaries became intentional.
21. What We Would Not Do
We wouldn’t assume every read model needs strong consistency.
That can make the architecture unnecessarily expensive and tightly coupled.
We wouldn’t assume eventual consistency is acceptable simply because the technology supports it.
We wouldn’t make the customer stare at stale financial information without explanation.
We wouldn’t let a derived read model become an unofficial source of truth.
We wouldn’t rely on “eventually” without measuring how long eventual actually takes.
And we wouldn’t remove reconciliation just because the system is usually correct.
22. The Questions I Ask Now
When evaluating a distributed workflow, I ask:
What is the authoritative source of truth?
Then:
Which parts of the system are derived?
Then:
How stale can each representation safely become?
Then:
What happens when propagation is delayed?
Then:
What does the customer see during that delay?
Then:
How do we detect divergence?
And finally:
If the system is technically correct but the customer believes it is wrong, which one are we going to call an incident?
The answer should never be determined accidentally.
23. The Bigger Architectural Lesson
Distributed systems force us to accept that there may not be one universally synchronized view of reality.
Instead, we may have:
Authoritative State
│
▼
Events
│
┌────┼────┐
▼ ▼ ▼
Read Fraud Reporting
Model Model Model
Each representation has a purpose.
Each can have a different consistency model.
The architecture becomes much easier to reason about when those boundaries are explicit.
The problem begins when the system has one consistency model but the customer has another expectation.
24. Final Thought
The payment wasn’t wrong.
The ledger wasn’t wrong.
The read model wasn’t necessarily broken.
The architecture was behaving exactly as designed.
But the customer didn’t experience our architecture.
They experienced the balance on their screen.
That was the real lesson.
Technical consistency and perceived correctness aren’t always the same thing.
Eventual consistency is a powerful architectural tool.
It lets systems scale.
It reduces coupling.
It enables asynchronous processing.
It allows specialized read models.
But every time we introduce a consistency boundary, we are making a decision about what users may temporarily see.
In financial systems, that decision deserves particular care.
Because a few seconds of stale analytics is usually harmless.
A few seconds of stale financial state can destroy confidence.
The goal isn’t to eliminate eventual consistency. It’s to know exactly where you can tolerate it, how long you can tolerate it, and what the customer should experience while the system converges.