The transaction used to be simple.
At least, it looked simple from the application’s perspective.
A customer placed an order.
The system processed the payment.
The order was created.
Inventory was reserved.
The ledger was updated.
The customer received a notification.
Inside the original application, much of this could happen within a relatively straightforward transactional workflow.
Then we decomposed the system.
The architecture became:
Payment
↓
Order
↓
Inventory
↓
Ledger
↓
Notification
Each capability became its own service.
Each service had its own responsibility.
Some had their own database.
Deployments became more independent.
Teams had clearer ownership.
Everything looked more modular.
Then one service failed halfway through the transaction.
And we discovered something fundamental:
A transaction that is simple inside one system can become a distributed business workflow when its responsibilities cross service boundaries.
The database was no longer the only thing we needed to think about.
We needed to reason about:
- Distributed transactions
- Partial failure
- Compensation
- Consistency boundaries
- Recovery
- Idempotency
- Business state
The problem wasn’t simply:
“How do we commit five database transactions?”
The real question was:
“What should the business state be when one part of the workflow fails?”
1. The Transaction Before Decomposition
The original architecture was relatively straightforward.
Conceptually:
BEGIN TRANSACTION
Create Order
Process Payment
Reserve Inventory
Create Ledger Entry
Commit
If something failed:
ROLLBACK
The database transaction gave us a powerful guarantee.
Either the operation succeeded or the changes could be rolled back.
The application didn’t have to reason about every intermediate state.
The database handled much of that complexity.
This worked because the important operations lived within a common transactional boundary.
Then the system changed.
2. The Architecture Became Distributed
After decomposition:
Order Request
│
▼
Payment Service
│
▼
Order Service
│
▼
Inventory Service
│
▼
Ledger Service
│
▼
Notification Service
Each service could have its own database.
For example:
Payment Service
│
▼
Payment DB
Order Service
│
▼
Order DB
Inventory Service
│
▼
Inventory DB
Ledger Service
│
▼
Ledger DB
Now imagine trying to put the entire workflow inside one database transaction.
There is no single local transaction anymore.
The work crosses:
- Processes
- Networks
- Databases
- Service boundaries
The transaction has become distributed.
3. The Failure That Changed Our Thinking
The first few steps succeeded.
Payment ✓
Order ✓
Inventory ✓
Then:
Ledger ✗
What should happen now?
The system had already performed several operations.
We couldn’t simply execute:
ROLLBACK
across all services.
The payment had already been processed.
The order had already been created.
Inventory had already been reserved.
The ledger operation had failed.
Notification hadn’t happened yet.
We were now sitting in a partially completed business transaction.
This was the moment we realized:
Distributed failure creates intermediate business states that a local database transaction cannot automatically clean up.
4. The First Question: Should We Use a Distributed Transaction?
One theoretical solution is to coordinate multiple databases through a distributed transaction protocol.
The idea is roughly:
Coordinator
│
├──► Payment DB
├──► Order DB
├──► Inventory DB
└──► Ledger DB
The coordinator tries to ensure that all participants agree on the outcome.
This can provide strong transactional semantics in some environments.
But it also introduces significant complexity.
The system now depends on:
- Coordination
- Participant availability
- Network reliability
- Transaction time
- Failure handling
- Operational complexity
For many service architectures, this wasn’t the direction we wanted.
More importantly, we realized something deeper.
The business workflow itself needed to define what should happen when things failed.
That led us toward the Saga pattern.
5. The Saga Changed the Question
Instead of trying to create one giant distributed database transaction, we modeled the business operation as a sequence of local transactions.
Conceptually:
Payment
↓
Order
↓
Inventory
↓
Ledger
Each step commits locally.
If a later step fails, the system performs a compensating action for the earlier steps where necessary.
For example:
Payment
↓
Order
↓
Inventory
↓
Ledger ✗
The system might respond:
Ledger failed
↓
Release Inventory
↓
Cancel Order
↓
Refund Payment
The important difference is that we’re not literally rolling back the databases.
We’re executing new business operations that compensate for previously completed operations.
6. Compensation Is Not Rollback
This distinction is extremely important.
A database rollback says:
“Pretend this transaction never happened.”
A compensation says:
“The transaction happened, but we need another business action to reverse its effect.”
For example:
Charge Payment
might be compensated by:
Refund Payment
These aren’t technically the same operation.
The original payment may have:
- Created a transaction record
- Triggered fraud checks
- Created ledger entries
- Sent notifications
- Generated external provider activity
A refund doesn’t erase history.
It creates a new business event representing the reversal.
That matters enormously in financial systems.
7. The Saga We Designed
The workflow became something like:
Start Order
│
▼
Process Payment
│
▼
Create Order
│
▼
Reserve Inventory
│
▼
Create Ledger Entry
│
▼
Complete
If everything succeeds:
Payment ✓
Order ✓
Inventory ✓
Ledger ✓
But if Ledger fails:
Payment ✓
Order ✓
Inventory ✓
Ledger ✗
the Saga executes compensation:
Release Inventory
↓
Cancel Order
↓
Refund Payment
The system eventually reaches a valid business state.
That word matters:
Eventually.
8. Consistency Became a Business Concept
Inside one database, consistency often means satisfying database constraints.
Across services, consistency becomes more nuanced.
Suppose:
Payment = Successful
Order = Created
Inventory = Reserved
Ledger = Pending
Is that an invalid state?
Not necessarily.
It may be a legitimate intermediate state.
The important question becomes:
Which states are acceptable, and for how long?
For example:
Payment: Successful
Order: Pending
Inventory: Reserved
Ledger: Processing
might be perfectly acceptable for a few seconds.
But:
Payment: Successful
Order: Cancelled
Inventory: Released
Ledger: Missing
may require immediate investigation or compensation.
We had to define these business states explicitly.
9. We Started Designing State Machines
This led us to model workflows more explicitly.
Instead of treating the transaction as:
Success / Failure
we introduced meaningful states.
For example:
Order
│
├── PENDING
│
├── PAYMENT_CONFIRMED
│
├── INVENTORY_RESERVED
│
├── COMPLETED
│
├── COMPENSATING
│
├── CANCELLED
│
└── FAILED
Now the system could explain what was happening.
A partially completed workflow wasn’t necessarily an unknown failure.
It was a known business state.
That made recovery much easier to reason about.
10. The Saga Needs an Owner
Another important decision was orchestration.
Who decides what happens next?
There are two common approaches.
Choreography
Services react to events.
PaymentCompleted
↓
Order Service
OrderCreated
↓
Inventory Service
InventoryReserved
↓
Ledger Service
Each service knows how to respond to events.
This can reduce central coordination.
But complicated workflows can become difficult to understand.
The behavior is spread across many services.
Orchestration
A Saga orchestrator coordinates the workflow.
Saga Orchestrator
/ | \
/ | \
▼ ▼ ▼
Payment Order Inventory
│
▼
Ledger
The orchestrator knows the workflow and determines what should happen next.
For complex financial workflows, explicit orchestration can make the business process easier to understand and operate.
The correct choice depends on the workflow.
11. Compensation Isn’t Always Possible
This was one of the most important lessons.
Some actions are easy to compensate.
Reserve Inventory
↓
Release Inventory
That’s relatively straightforward.
Others are more complicated.
Consider:
Send Notification
Once an email or SMS has been sent, you can’t truly “roll it back.”
You can send another message explaining the correction.
But the original message already existed.
Similarly, external payment operations may not be reversible immediately.
A payment might need to be refunded rather than rolled back.
This means:
Saga design must consider the reversibility of every business action.
12. The Order of Operations Matters
We also learned that workflow ordering matters.
Consider:
Charge Payment
↓
Reserve Inventory
What happens if inventory isn’t available?
We might need to refund the payment.
Alternatively, we could reserve inventory first:
Reserve Inventory
↓
Charge Payment
Now a payment failure requires releasing inventory.
Neither sequence is universally correct.
The business requirements determine the appropriate ordering.
The architecture should minimize expensive or difficult-to-compensate operations where practical.
13. Idempotency Became Mandatory
Distributed workflows retry.
Messages can be duplicated.
Network responses can be lost.
A service might process a request successfully but fail before reporting success.
That means this can happen:
Process Payment
│
▼
Success
│
▼
Response Lost
│
▼
Retry
│
▼
Process Payment Again?
For financial operations, this is dangerous.
We therefore needed idempotency.
A payment request might carry:
idempotency_key:
order-847291-payment
If the same request arrives again, the payment service can recognize that it has already been processed.
This protects the business operation from duplicate execution.
14. Compensation Also Needs Idempotency
It isn’t enough for the original operation to be idempotent.
The compensating action should be safe too.
For example:
Refund Payment
might be retried.
We must ensure that:
Refund
Refund
Refund
doesn’t produce multiple refunds.
The same principle applies to:
Release Inventory
Cancel Order
Reverse Ledger Entry
A Saga is only as reliable as its ability to safely recover from retries and partial failures.
15. We Needed a Durable Workflow State
Another lesson was that the Saga itself needs durable state.
Suppose the workflow reaches:
Payment ✓
Order ✓
Inventory ✓
Ledger ✗
Then the orchestration process crashes.
When it comes back, how does it know what already happened?
It can’t rely on in-memory state.
The workflow needs durable information such as:
Order ID
Payment Status
Inventory Status
Ledger Status
Current Step
Compensation Status
Retry Count
Failure Reason
Now recovery becomes possible.
The system can resume or compensate based on persisted state.
16. The Failure Path Became More Important Than the Happy Path
Before decomposition, most of our design discussions focused on:
What happens when everything succeeds?
After introducing distributed workflows, we started asking:
What happens if step 3 fails?
What if the response is lost?
What if the consumer retries?
What if the service is unavailable for ten minutes?
What if compensation fails?
What if the system crashes during compensation?
What if the external provider succeeds but our request times out?
These questions weren’t edge cases.
They were normal distributed-system scenarios.
The failure path had become part of the primary architecture.
17. The External Provider Problem
This became especially important when external systems were involved.
Consider:
Payment Service
│
▼
External Payment Provider
We send the payment request.
The provider processes it.
Then our network connection times out.
Our system sees:
Unknown
Did the payment succeed?
We cannot simply retry blindly.
The retry could result in a duplicate charge.
Instead, we need mechanisms such as:
- Idempotency keys
- Provider transaction identifiers
- Status queries
- Reconciliation
- Explicit “unknown” states
This is another reason distributed transaction problems are ultimately business workflow problems.
The database cannot tell us what happened inside an external system.
18. Reconciliation Became Part of Reliability
Eventually we accepted something uncomfortable:
Some distributed workflows can reach an unknown state.
For example:
Payment request sent
↓
Network timeout
↓
Unknown outcome
The correct response isn’t always:
FAIL
It may be:
PENDING_RECONCILIATION
A later process can verify the external state and resolve the workflow.
This introduced another important capability:
reconciliation.
In financial systems, reconciliation isn’t merely an operational afterthought.
It can be part of the architecture.
19. What We Actually Changed
The final design had several layers.
Local transactions
Each service maintained its own local transactional guarantees.
Saga coordination
The business workflow tracked progress across services.
Compensation
Failed workflows triggered appropriate business-level reversal operations.
Idempotency
Retries could safely repeat requests.
Durable state
Workflow progress survived service restarts.
Reconciliation
Unknown external outcomes could be resolved later.
Observability
Every step carried correlation information so the complete workflow could be traced.
The result wasn’t a magical distributed transaction.
It was a system designed to recover from partial completion.
20. What the Architecture Looked Like
The final conceptual flow looked more like:
Saga / Workflow
│
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
Payment Order Inventory
│ │ │
└────────────────┼────────────────┘
│
▼
Ledger
│
▼
Notification
With failure paths:
Ledger Failure
│
▼
Start Compensation
│
┌────────────┼────────────┐
▼ ▼ ▼
Release Cancel Refund
Inventory Order Payment
Every step had an explicit success and failure strategy.
That was the real improvement.
21. What We Would Not Do Now
We wouldn’t assume that every workflow needs a distributed transaction.
We wouldn’t make every service participate in one giant synchronous chain.
We wouldn’t rely on a single database transaction to protect a multi-service workflow.
We wouldn’t assume compensation is equivalent to rollback.
We wouldn’t make external operations retryable without considering idempotency.
We wouldn’t hide intermediate business states.
And we wouldn’t design only the happy path.
The failure path is part of the workflow.
22. The Questions I Ask Before Splitting a Transaction
Before turning one transaction into multiple services, I ask:
What is the actual business transaction?
Then:
Which parts truly need atomic consistency?
Then:
Where can consistency be eventual?
Then:
What happens if step three succeeds but step four fails?
Then:
Can every completed step be compensated?
And:
What happens if compensation itself fails?
Finally:
Can the system recover without requiring a human to understand every internal state?
If the answer is no, the workflow isn’t finished.
23. The Bigger Architectural Lesson
Distributed transactions are often discussed as a database problem.
But once a workflow crosses service boundaries, the deeper problem is usually business semantics.
The database can answer:
“Did this local transaction commit?”
It cannot answer:
“What should the business do because another service failed?”
That requires business rules.
For example:
Payment succeeded
Inventory failed
What should happen?
Refund the payment?
Keep the payment and wait?
Reserve inventory later?
Cancel the order?
That isn’t a database question.
It’s a business workflow question.
24. Final Thought
The original transaction looked like this:
BEGIN
↓
Do Everything
↓
COMMIT
After decomposition, reality looked more like:
Payment ✓
↓
Order ✓
↓
Inventory ✓
↓
Ledger ✗
↓
Compensate
↓
Recover
We initially saw this as a problem created by microservices.
Eventually, we understood it differently.
The architecture had forced us to make something explicit that the monolith had hidden:
the business workflow.
Once the workflow crossed service boundaries, we had to define:
- What success means
- What failure means
- Which states are valid
- Which actions can be compensated
- How retries work
- How recovery works
- How unknown outcomes are resolved
That made the architecture more complicated.
But it also made the business process more explicit.
The most important lesson was simple:
Distributed transactions are usually a business workflow problem, not merely a database problem.
When a transaction becomes five services, the hard part isn’t committing five databases.
The hard part is deciding what the business should do when only four of them succeed.