The system was becoming increasingly difficult to change.
Every time we added a new capability, another service needed to know about it.
A payment completed.
The ledger needed to update.
Notifications needed to be sent.
Fraud analytics needed the transaction.
Reporting needed the same information.
Customer activity needed to be recorded.
At first, the architecture handled this with synchronous service calls.
It was straightforward.
One service called another.
That service called another.
The request moved through the system until the work was complete.
But as the number of capabilities grew, something became obvious:
Too many services were waiting for each other.
A small business event could trigger a chain of synchronous dependencies.
We needed to reduce that coupling.
So we started moving some workflows from synchronous calls to events.
It worked.
But it also introduced an entirely different class of problems.
We had reduced spatial coupling.
We had increased temporal and consistency complexity.
That became one of the most important lessons of the architecture.
Events reduce coupling, but they introduce temporal complexity.
1. The Synchronous Architecture Was Becoming Tightly Coupled
The original flow looked something like this:
Customer
│
▼
Payment Service
│
├────► Ledger Service
│
├────► Notification Service
│
├────► Fraud Service
│
└────► Reporting Service
The payment service knew about all of these dependencies.
When a payment completed, it had to coordinate with several downstream systems.
That created several problems.
If Notification Service was slow, the payment workflow could be affected.
If Reporting Service was unavailable, the payment service had to decide what to do.
If Fraud Service changed its API, Payment Service might need to change.
The dependency graph kept growing.
Eventually:
Payment Service
│
├──► Service A
├──► Service B
├──► Service C
├──► Service D
└──► Service E
The payment service had become a coordinator for everything that happened after a payment.
That wasn’t sustainable.
2. The Idea Was Simple: Publish What Happened
Instead of directly calling every downstream service, we changed the model.
The Payment Service would publish an event:
PaymentCompleted
Other services could consume it.
The architecture became:
Payment Service
│
▼
PaymentCompleted
│
▼
Event Broker
/ | \
/ | \
▼ ▼ ▼
Ledger Notification Analytics
Payment Service no longer needed to know every consumer.
It only needed to publish the fact that something had happened.
That was a significant reduction in coupling.
3. Events and Commands Are Not the Same Thing
One of the first design decisions we had to make was understanding the difference between an event and a command.
A command says:
Do this.
An event says:
This happened.
For example:
Command:
ProcessPayment
This is an instruction.
The receiver is expected to perform an action.
An event would be:
PaymentProcessed
This describes something that has already happened.
The distinction matters.
A command has a target.
An event can have many consumers.
For example:
PaymentProcessed
│
├──► Ledger
├──► Notifications
├──► Analytics
└──► Customer Activity
The producer doesn’t need to know which consumers exist.
That was one of the major architectural advantages.
4. Producers and Consumers
The Payment Service became the producer.
The other services became consumers.
Producer
│
▼
Event Broker
│
├──► Consumer A
├──► Consumer B
└──► Consumer C
This gave us a much cleaner dependency model.
The producer knew the event contract.
It didn’t need to know the implementation details of every consumer.
And adding a new consumer became easier.
Suppose we later needed a fraud analytics system.
Previously:
Payment Service
│
└──► Fraud Analytics
The payment service needed a new integration.
With events:
PaymentProcessed
│
├──► Existing Consumers
│
└──► Fraud Analytics
The producer didn’t necessarily need to change.
That was powerful.
5. But We Lost Immediate Consistency
This was the first major trade-off.
In the synchronous architecture:
Payment
↓
Ledger
↓
Response
The system could potentially confirm that all required work had completed before returning.
With events:
Payment
↓
Event Published
↓
Response
...later...
Event Consumer
↓
Ledger Updated
The payment could be completed before the ledger consumer processed the event.
The system had become eventually consistent.
That wasn’t automatically bad.
But it needed to be intentional.
For some workflows, eventual consistency was perfectly acceptable.
For others, it wasn’t.
The important question became:
Which parts of the business process must be immediately consistent, and which can converge later?
6. Eventual Consistency Became a Business Decision
Consider notifications.
If a payment succeeds at:
10:00:00
and the notification is sent at:
10:00:02
that’s usually acceptable.
But consider a ledger balance.
If the customer expects a balance to reflect a completed transaction immediately, two seconds of inconsistency might matter.
The architecture therefore needed to distinguish between:
Latency-tolerant work
and:
Correctness-critical state
We learned not to make everything asynchronous simply because asynchronous systems scale well.
The business semantics had to come first.
7. Then We Saw Duplicate Events
The next problem appeared in production.
An event was delivered.
The consumer processed it.
Then the same event was delivered again.
At first, that surprised us.
Then we realized something important:
Duplicate delivery is normal in distributed systems.
A consumer might process an event successfully but fail before acknowledging it.
The broker may then deliver the event again.
For example:
Event
│
▼
Consumer
│
▼
Database Update ✓
│
▼
Acknowledgement ✗
From the broker’s perspective:
“The consumer didn’t successfully acknowledge the event.”
So it delivered it again.
Now:
Event
│
├──► Consumer
│
└──► Consumer again
The consumer needed to tolerate that.
8. Idempotency Became Essential
We introduced idempotent processing.
Each event received a unique identifier.
For example:
event_id:
evt_8f42a91
The consumer could record processed event IDs.
Conceptually:
Receive Event
│
▼
Have we processed this event?
│
┌─┴─┐
│ │
Yes No
│ │
Ignore Process
│
▼
Record
The exact implementation depends on the system.
But the principle is universal:
Consumers should assume an event may be delivered more than once.
This became especially important in financial workflows.
An event representing:
PaymentCompleted
must not accidentally result in multiple financial effects simply because the message was delivered more than once.
9. Event Ordering Was Another Problem
Duplicate delivery wasn’t the only issue.
Events can also arrive in an unexpected order.
Suppose we have:
PaymentCreated
PaymentCompleted
PaymentRefunded
The expected sequence is:
Created
↓
Completed
↓
Refunded
But distributed systems can encounter:
PaymentCreated
PaymentRefunded
PaymentCompleted
depending on the architecture, partitions, retries, consumers, and event-processing model.
Now the consumer has a question:
What does this event mean if I haven’t seen the previous event yet?
We couldn’t simply assume that events would always arrive in the order they were generated.
Ordering had to become an explicit design consideration.
10. We Stopped Treating the Event Stream Like a Database Transaction
This was another mental shift.
In a traditional transaction:
BEGIN
↓
Operation A
↓
Operation B
↓
Operation C
↓
COMMIT
The ordering and atomicity are controlled by the transaction.
An event-driven workflow is different.
Event A
↓
Broker
↓
Consumer A
Event B
↓
Broker
↓
Consumer B
Events are messages moving through an asynchronous system.
They are not automatically a distributed transaction.
That distinction sounds obvious.
But forgetting it can create serious architectural problems.
11. Replay Changed How We Thought About Events
One of the most useful capabilities of event-driven architecture was replay.
Suppose we had historical events:
PaymentCreated
PaymentCompleted
PaymentRefunded
PaymentCompleted
PaymentRefunded
...
A new consumer could potentially process historical events to build a new projection.
For example:
Historical Events
│
▼
New Consumer
│
▼
New Read Model
This meant events could become more than messages.
They could become an important source of historical information.
But replay introduced another requirement:
Consumers needed to be safe to run against historical events.
If processing an old event triggered an external side effect again, replay could become dangerous.
For example:
Replay PaymentCompleted
│
▼
Send Customer Notification
Should that notification actually be sent again?
Not necessarily.
Replay requires clear separation between:
- Rebuilding state
- Performing external side effects
That became an important architectural distinction.
12. Event Contracts Became APIs
We initially thought of events as internal implementation details.
That changed quickly.
Once multiple services depended on an event:
PaymentCompleted
the event schema effectively became a contract.
Changing it carelessly could break consumers.
We therefore had to think about:
- Event versioning
- Backward compatibility
- Required fields
- Optional fields
- Schema evolution
- Consumer expectations
The event was no longer simply a message.
It was part of the architecture.
13. We Had to Think About Failed Consumers
What happens when one consumer fails?
For example:
PaymentCompleted
│
├──► Ledger ✓
├──► Notification ✗
└──► Analytics ✓
The payment itself might still be successful.
Notification might retry later.
That was another advantage of asynchronous processing.
One consumer could fail without necessarily blocking every other consumer.
But it also created operational questions.
How many times should we retry?
When should an event move to a dead-letter queue?
How do we detect a consumer that has fallen behind?
How do we replay failed events?
How do we know whether the business effect eventually completed?
The architecture now required answers to all of these questions.
14. Queue Depth Became a Business Signal
Traditional monitoring might tell us:
CPU: 45%
Memory: 60%
But for an event-driven system, we also needed:
Queue Depth
Consumer Lag
Processing Rate
Failure Rate
Retry Rate
Dead-Letter Count
These weren’t just infrastructure metrics.
For example:
Consumer Lag ↑
↓
Settlement Processing Delayed
↓
Customer Impact
The event pipeline had become part of the customer-facing architecture.
Observability therefore had to extend into messaging.
15. We Learned Not to Make Everything an Event
Once events worked well for a few workflows, there was a temptation to use them everywhere.
That would have been another mistake.
Not every interaction benefits from asynchronous communication.
Some operations need an immediate response.
For example:
Validate Authentication
↓
Immediate Result Required
A synchronous request can be the right choice.
Other operations naturally fit events:
Payment Completed
↓
Notify Customer
↓
Update Analytics
↓
Update Reporting
Those consumers don’t necessarily need to block the original transaction.
The architectural question became:
Does the caller need the result now, or does the system only need the fact that something happened?
That distinction helped us choose between commands, queries, and events.
16. Events Reduced Spatial Coupling
This was the biggest benefit.
Previously:
Payment
│
├──► Ledger
├──► Notification
├──► Reporting
└──► Analytics
The payment service knew about each dependency.
After introducing events:
Payment
│
▼
PaymentCompleted
│
▼
Event Broker
│
├──► Ledger
├──► Notification
├──► Reporting
└──► Analytics
The producer no longer needed to know every consumer.
That reduced spatial coupling.
But we discovered that the system had become more dependent on time.
17. Temporal Coupling Became the New Challenge
The services were no longer required to complete work at exactly the same moment.
That was good.
But now the system had to deal with:
When will the event arrive?
When will the consumer process it?
What if processing is delayed?
What if processing fails?
What if the event is delivered twice?
What if events arrive out of order?
What if a new consumer needs historical events?
The architecture had become less coupled spatially.
But more complex temporally.
This was the trade-off we hadn’t appreciated at the beginning.
18. The Architecture Became More Resilient in Some Places
There was another important benefit.
Suppose Notification Service went down.
In the synchronous architecture:
Payment
│
▼
Notification Service ✗
│
▼
Payment Workflow Affected
With asynchronous processing:
Payment
│
▼
PaymentCompleted
│
▼
Broker
│
▼
Notification Service ✗
The event could remain available for later processing, depending on the messaging system and delivery design.
The payment workflow didn’t necessarily need to wait.
That gave us a stronger failure boundary.
But only if we designed the consumer and messaging infrastructure correctly.
19. We Introduced Explicit Event Ownership
As the event system grew, another problem appeared.
Who owns an event?
For example:
PaymentCompleted
The Payment domain should own the meaning of that event.
Consumers shouldn’t redefine it according to their own needs.
We became more deliberate about:
- Event naming
- Event ownership
- Event schemas
- Versioning
- Producers
- Consumers
- Compatibility
This reduced the chance that the event platform would become an unstructured integration layer.
20. What We Actually Changed
The final architecture wasn’t simply:
“Replace REST calls with Kafka.”
It was more deliberate.
We classified interactions.
Synchronous
Use when:
- The caller needs an immediate response
- Strong consistency is required
- The operation is request/response oriented
Asynchronous commands
Use when:
- A specific component needs to perform an action
- The caller doesn’t need to wait for completion
Events
Use when:
- Something has already happened
- Multiple consumers may be interested
- The producer shouldn’t depend on knowing every consumer
This distinction made the architecture much easier to reason about.
21. The Result
After moving appropriate workflows to events, we saw several improvements.
Services became less tightly coupled.
New consumers could be added without modifying the original producer in many cases.
Some background workloads no longer blocked customer-facing requests.
Failures in non-critical consumers became easier to isolate.
The architecture became more extensible.
But we also had to build new capabilities around it:
- Idempotent consumers
- Event tracing
- Schema management
- Retry handling
- Dead-letter processing
- Consumer monitoring
- Replay strategies
- Consistency rules
We hadn’t eliminated complexity.
We had changed the shape of the complexity.
And that was the real architectural trade-off.
22. The Questions I Ask Before Introducing an Event
Before replacing a synchronous call with an event, I ask:
Does the consumer need the result immediately?
If yes, synchronous communication may be better.
Can the operation tolerate eventual consistency?
If no, asynchronous processing may be the wrong choice.
Can the consumer safely process duplicates?
If no, we have an idempotency problem to solve.
What happens if events arrive late or out of order?
If the answer is unclear, the design isn’t finished.
Can we replay the event safely?
If not, operational recovery could become difficult.
Who owns the event contract?
If nobody owns it, it will eventually become unstable.
And finally:
What happens when the consumer is unavailable for an hour?
That question often reveals more than the architecture diagram.
23. The Bigger Architectural Lesson
Event-driven architecture is powerful because it changes how components interact.
Instead of:
"Do this for me."
we can communicate:
"This happened."
That can dramatically reduce coupling.
But asynchronous systems don’t remove complexity.
They introduce new dimensions:
Events
↓
Temporal Decoupling
↓
Eventual Consistency
↓
Duplicates
↓
Ordering
↓
Retries
↓
Replay
↓
Operational Complexity
The system becomes more flexible.
It also becomes harder to reason about in time.
That’s the trade-off.
24. Final Thought
When we started introducing events, we thought the primary benefit would be simple:
fewer synchronous dependencies.
That was true.
But the deeper change was much larger.
We moved from a system where components needed to coordinate immediately to one where components could react independently over time.
That gave us:
- Lower coupling
- Better failure isolation
- More extensibility
- More asynchronous processing
But it also gave us:
- Eventual consistency
- Duplicate delivery
- Ordering problems
- Replay complexity
- Consumer lag
- More difficult debugging
The lesson wasn’t:
“Everything should be event-driven.”
It was:
“Use events where temporal decoupling creates real architectural value.”
An event is not just another way to call a service.
It changes the consistency model, the failure model, and the way the system evolves.
Events reduce coupling, but they introduce temporal and consistency complexity.
That isn’t a reason to avoid them.
It’s the reason to design them deliberately.