For a long time, one region was enough.
The application ran there.
The database ran there.
The message infrastructure ran there.
Backups existed.
Monitoring was in place.
We had redundancy inside the region.
From an infrastructure perspective, the system looked highly available.
Then the region became the failure domain.
Not one server.
Not one database node.
Not one availability zone.
The region itself.
That changed the question completely.
We weren’t asking:
“What happens when a server fails?”
We were asking:
“What happens when everything in this region becomes unavailable?”
The answer wasn’t simply:
“Start the same infrastructure somewhere else.”
Because the infrastructure was only one part of the problem.
The harder problems were:
data, consistency, traffic, failover, recovery, and business state.
That was when we learned:
Multi-region isn’t simply duplicating infrastructure. It’s designing the system to operate across failure domains.
1. The Architecture We Started With
The original architecture was straightforward:
Internet
│
▼
┌───────────┐
│ Region │
│ A │
└─────┬─────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
Application Database Message Bus
│
▼
Customers
Everything lived in Region A.
We had redundancy inside the region.
Multiple application instances.
Database replicas.
Load balancing.
Failover between availability zones.
Backups.
Monitoring.
It was resilient to many failures.
But not to:
Region A
↓
Completely unavailable
That was the architectural boundary we hadn’t fully addressed.
2. The First Mistake: Thinking Multi-Region Means Two Copies
The obvious architecture was:
Global Traffic
│
┌────────┴────────┐
▼ ▼
Region A Region B
│ │
Database Database
At first glance, this looks like high availability.
But immediately, difficult questions appear.
What data exists in Region B?
How quickly does it replicate?
Which region is authoritative?
What happens if both regions accept writes?
What happens when the network between regions fails?
How does traffic move?
How do we know Region B is actually ready?
What happens to in-flight transactions?
Where are financial records legally allowed to live?
A second region doesn’t automatically answer any of those questions.
3. Availability Is More Than Infrastructure
We initially thought about availability as:
Servers
Databases
Networks
Load balancers
But a customer’s ability to complete a transaction depends on much more:
Customer
↓
DNS / Global Routing
↓
Region
↓
Application
↓
Database
↓
Message Infrastructure
↓
External Dependencies
↓
Financial State
Any one of those can prevent the system from operating correctly.
A region can be running while the application is functionally unavailable.
For example:
Region B:
Application = healthy
Database = healthy
Network = healthy
But:
Financial data = stale
From the customer’s perspective, Region B isn’t ready.
This introduced an important distinction:
Infrastructure availability is not the same as application readiness.
4. The Real Question Was: What Does Failover Mean?
We had to define what failover actually meant.
There are several possibilities.
Active-Passive
Region A
│
├── Active
│
▼
Customers
Region B
│
└── Standby
Region B remains mostly passive until Region A fails.
Active-Active
Customers
/ \
▼ ▼
Region A Region B
│ │
└──────┬──────┘
▼
Data Layer
Both regions actively serve traffic.
This provides better utilization and potentially faster failover.
But it also introduces significantly more consistency and coordination complexity.
Neither model is automatically better.
The correct choice depends on the workload.
5. Replication Became the Hard Part
Application servers are relatively easy to duplicate.
Data is not.
Suppose Region A processes:
Payment $500
The ledger becomes:
Balance = $500
That state must somehow become available in Region B.
The architecture now has a replication path:
Region A
│
▼
Primary Data
│
│ replication
▼
Region B
│
▼
Replica Data
The question becomes:
How much replication lag can the business tolerate?
If replication takes:
10ms
that’s one problem.
If it takes:
10 seconds
that’s another.
If Region A fails before replication completes, Region B may not have the latest state.
That becomes a recovery problem.
6. RPO Became a Business Decision
This led us to Recovery Point Objective, or RPO.
Conceptually:
How much data can we afford to lose if a region fails?
For some systems:
RPO = minutes
might be acceptable.
For financial transaction records:
RPO = "some transactions disappeared"
may be completely unacceptable.
That means the replication strategy needs to reflect the business requirement.
The question isn’t simply:
“Can we replicate this database?”
It’s:
“What consistency and recovery guarantees does the business require?”
7. RTO Was Just as Important
Then there was Recovery Time Objective, or RTO.
If Region A fails:
Region A
↓
Unavailable
How long until customers can operate again?
Maybe:
30 seconds
Maybe:
5 minutes
Maybe:
1 hour
Again, the architecture changes based on the requirement.
A system designed for:
RTO = 1 hour
can look very different from one designed for:
RTO = 30 seconds
Multi-region architecture therefore starts with recovery objectives, not infrastructure diagrams.
8. Traffic Routing Became Part of the Architecture
Once Region B existed, traffic needed to know where to go.
Conceptually:
Users
│
▼
Global Routing
/ \
▼ ▼
Region A Region B
Routing decisions could depend on:
- Region health
- Latency
- Capacity
- Availability
- Deployment state
- Data residency
- Customer location
But routing traffic isn’t enough.
We also need to know:
Is the target region actually capable of processing the request?
A region might respond to health checks while its database is stale or its critical dependencies are unavailable.
So health checks need to reflect application readiness, not just infrastructure reachability.
9. The Dangerous Failover
One of the most dangerous scenarios was:
Region A
↓
Looks unhealthy
The system automatically routes traffic to:
Region B
But Region A wasn’t actually dead.
It was partitioned from the global routing layer.
Now both regions believe they should process traffic.
We can end up with:
Region A ── accepts writes
↕
X
↕
Region B ── accepts writes
The regions cannot communicate properly.
Now we have a split-brain scenario.
For ordinary applications, that is already difficult.
For financial state, it can be disastrous.
10. Active-Active Isn’t Free
Active-active sounds attractive:
Region A ←→ Region B
Both regions process traffic.
Both regions provide capacity.
If one fails, the other continues.
But now we need to answer:
Who owns the state?
Suppose the same account is modified simultaneously in both regions.
Region A:
Debit $100
Region B:
Debit $50
What is the correct final state?
If both operations are accepted independently, reconciliation becomes complicated.
This is why active-active architectures often require carefully designed ownership boundaries.
11. We Started Thinking in Terms of Data Ownership
Instead of allowing arbitrary writes from every region, we asked:
Which region is authoritative for which piece of state?
For example:
Customer Account A
│
▼
Primary Region A
while another account might be owned by:
Customer Account B
│
▼
Primary Region B
Or the entire ledger could have a single authoritative write region while other regions provide read capacity.
There are many possible models.
The important thing is that ownership must be explicit.
12. Financial State Changed the Design
In a financial system, not all data has equal importance.
Consider:
Product catalog
versus:
Ledger transaction
A stale product description may be harmless.
A missing financial transaction isn’t.
That means multi-region architecture should classify data by business criticality.
For example:
Consistency
│
┌────────────────┼────────────────┐
│ │ │
Eventually Near-real-time Strong
consistent consistent authority
│ │ │
Analytics Read models Ledger
Reporting Search Payments
Not every dataset needs the same replication strategy.
13. Data Residency Added Another Constraint
Financial systems often operate across jurisdictions.
That introduces another question:
Can this data legally be stored or processed in another region?
A multi-region architecture might look technically perfect:
Region A ↔ Region B
but business or regulatory requirements may restrict where certain data can reside.
We may need:
Customer data
↓
Specific jurisdiction
while allowing:
Aggregated analytics
↓
Other regions
This means geography can become an architectural constraint.
Multi-region design isn’t purely a technical availability problem.
It can also be a regulatory and data-governance problem.
14. Failover Doesn’t Mean “Everything Moves”
Another misconception we had to eliminate was:
“If Region A fails, move everything to Region B.”
That’s rarely that simple.
Some state may already be in transit.
Some messages may be duplicated.
Some transactions may be incomplete.
Some external providers may still think Region A is processing.
Some scheduled jobs may have already started.
Some caches may contain stale data.
Some customers may have requests in flight.
A real failover looks more like:
Region A failure
↓
Stop accepting new work
↓
Determine authoritative state
↓
Assess replication lag
↓
Recover pending workflows
↓
Promote Region B
↓
Route traffic
↓
Validate critical flows
↓
Resume normal processing
Failover is a workflow.
Not a DNS change.
15. We Needed a Clear Recovery Sequence
We eventually documented recovery as an explicit sequence.
For example:
Step 1: Detect
Regional health degradation
Step 2: Assess
Is this a partial outage
or a complete regional failure?
Step 3: Protect State
Stop unsafe writes
Step 4: Establish Authority
Determine which state is authoritative
Step 5: Promote
Activate Region B
Step 6: Route
Move customer traffic
Step 7: Validate
Test critical business workflows
Step 8: Reconcile
Compare recovered state
with authoritative records
The sequence was more important than the infrastructure diagram.
16. We Learned That Recovery Can Create Duplicate Work
Consider a transaction:
Customer
↓
Region A
↓
External Payment Provider
The provider successfully processes the transaction.
But Region A fails before recording the response.
Region B receives a retry.
Without idempotency:
Attempt 1 → Payment succeeds
Attempt 2 → Payment succeeds again
We have a duplicate financial effect.
This is why multi-region architecture connects directly to lessons from retries and payment resilience.
Failover increases the number of situations where the same logical operation may be retried.
Idempotency becomes even more important.
17. Queues Made Failover More Complicated
Suppose Region A has:
100,000 pending events
when the region fails.
Where are those messages?
Can Region B consume them?
Were some already processed?
Were some acknowledged but not persisted?
Are consumers allowed to resume from the last known offset?
Could messages be delivered twice?
A multi-region architecture therefore needs to consider message ownership and recovery as carefully as database replication.
The data plane isn’t only the database.
It can include:
Database
Message broker
Object storage
Caches
Search indexes
Workflow state
Each has its own recovery characteristics.
18. We Tested Regional Failure Before We Needed It
The architecture looked convincing on paper.
That wasn’t enough.
We needed to test:
Region A unavailable
Then:
Traffic → Region B
Then:
Critical workflow
Then:
Database state
Then:
Message processing
Then:
External dependencies
We discovered several assumptions that weren’t actually true.
Some services had hard-coded regional endpoints.
Some background jobs ran only in Region A.
Some caches were treated as if they were authoritative.
Some monitoring existed only in the primary region.
Some recovery procedures had never been tested.
The exercise exposed more than any architecture review could.
19. Multi-Region Increased Operational Complexity
Before:
One Region
After:
Region A
↕
Region B
But the operational model also changed:
Deployments
Monitoring
Incident response
Data replication
Failover
Testing
Security
Access control
Configuration
Secrets
Backups
Disaster recovery
Everything became more complex.
That led to another important realization:
Multi-region should only be introduced when the business value justifies the additional operational complexity.
If the requirement is:
"Five minutes of downtime is unacceptable"
then multi-region may be justified.
If the requirement is:
"We can recover within an hour"
a simpler disaster recovery architecture might be better.
20. What We Actually Changed
We didn’t simply duplicate Region A.
We redesigned the system around explicit failure assumptions.
We:
- Defined RPO and RTO requirements
- Classified data by consistency needs
- Defined regional ownership
- Designed replication strategies
- Added global traffic routing
- Introduced application-aware health checks
- Designed failover workflows
- Protected against split-brain scenarios
- Ensured critical operations were idempotent
- Tested message recovery
- Added reconciliation procedures
- Documented regional recovery
- Tested actual regional failure
The result wasn’t merely:
Two regions
It was:
Two failure domains
+
Explicit data ownership
+
Recovery procedures
+
Tested operational behavior
21. What We Would Not Do
We wouldn’t deploy two regions and assume we automatically have high availability.
We wouldn’t make both regions writable without carefully defining consistency and ownership.
We wouldn’t treat DNS failover as a complete disaster recovery strategy.
We wouldn’t ignore data residency requirements.
We wouldn’t assume replicated data is automatically current.
We wouldn’t promote a region without verifying its data and dependencies.
We wouldn’t forget background jobs and message processing during failover.
And we wouldn’t claim multi-region resilience without actually testing regional failure.
22. The Questions I Ask Now
When evaluating a multi-region architecture, I ask:
What failure are we actually trying to survive?
Then:
What is our RPO?
What is our RTO?
Which data must be strongly authoritative?
Who owns writes?
What happens during a network partition?
Can the regions operate independently?
How do we prevent split-brain?
Where does customer traffic go during failover?
What happens to messages already in flight?
What happens to external transactions already submitted?
What data can legally cross regions?
And finally:
Have we actually tested the failure we’re claiming to survive?
That last question is usually the most revealing.
23. The Bigger Architectural Lesson
Multi-region architecture isn’t about copying infrastructure.
It’s about designing for independent failure domains.
The system needs to answer:
Region A fails
↓
What happens to traffic?
↓
What happens to data?
↓
What happens to messages?
↓
What happens to transactions?
↓
What happens to external calls?
↓
What does the customer see?
If those answers aren’t explicit, the system isn’t truly multi-region.
It simply has infrastructure in two places.
24. Final Thought
The biggest mistake we made was thinking:
“If Region A goes down, Region B will take over.”
That sentence sounds simple.
The reality is much harder.
Region B needs:
- The right data
- The right configuration
- The right dependencies
- The right traffic
- The right permissions
- The right message state
- The right business ownership
- The right recovery procedures
And most importantly:
The system needs to know what state it is allowed to trust.
That’s what makes multi-region architecture fundamentally different from infrastructure duplication.
A second region doesn’t automatically give you resilience.
It gives you another place to run the system.
The architecture determines whether that second region can actually save you when the first one fails.
Multi-region isn’t simply duplicating infrastructure. It’s designing the system so that a regional failure becomes a controlled transition instead of an uncontrolled incident.