We originally introduced Kafka for a simple reason.
We needed asynchronous communication.
A service would publish an event.
Another service would consume it.
The producer didn’t need to wait.
The consumer could process the event independently.
It looked straightforward.
Payment Service
│
▼
Kafka
│
├──────────► Fraud Service
│
├──────────► Notification Service
│
└──────────► Analytics
Kafka was infrastructure.
The database was where the important data lived.
At least, that was how we thought about it.
Then the system grew.
We increased event retention.
More consumers started depending on historical events.
Teams began replaying topics to rebuild state.
New services were created by consuming old events.
Operational teams started asking questions like:
“Can we replay everything from last Tuesday?”
And eventually we realized something important.
Kafka wasn’t just transporting messages anymore.
It had become durable business data for parts of the architecture.
That changed how we had to design, operate, and govern it.
1. Kafka Started as a Messaging Layer
The original mental model was simple:
Producer
│
▼
Kafka
│
▼
Consumer
The producer published an event.
The consumer processed it.
Once processing completed, we didn’t think much about the event itself.
It was essentially a message in transit.
For example:
{
"paymentId": "pay-123",
"status": "COMPLETED"
}
The Payment Service knew the payment.
Kafka delivered the notification.
The consuming service updated whatever state it needed.
Simple.
But then we started retaining those events for longer.
And that changed everything.
2. Retention Changed the Meaning of the Data
Initially, events might have been retained for a relatively short period.
Then someone asked:
“Why delete them?”
Keeping events longer gave us useful capabilities.
We could:
- Replay historical events
- Rebuild consumer state
- Recover from processing failures
- Bootstrap new services
- Investigate historical behavior
- Reprocess corrected logic
- Recover certain downstream datasets
Suddenly, the event stream wasn’t disposable.
It contained history.
That history had value.
We had effectively created another durable representation of business activity.
3. The First Replay Changed Our Thinking
A downstream service had a processing bug.
Some events had been consumed incorrectly.
The traditional approach would have been:
Restore database
Repair records
Run correction scripts
Instead, we were able to reset the consumer position and replay the events.
Kafka Topic
│
│ replay
▼
Consumer
│
▼
Rebuilt State
That was powerful.
But it introduced a new question:
What exactly does replay mean for the business?
If an event says:
PaymentCompleted
and we process it again, are we:
- Rebuilding state?
- Sending another notification?
- Triggering another settlement?
- Updating analytics?
- Executing another financial action?
Replay is not automatically safe.
It depends on what the consumer does with the event.
4. We Learned That Events Are Not Commands
This distinction became extremely important.
A command says:
Do something.
An event says:
Something happened.
For example:
Command:
CapturePayment
versus:
Event:
PaymentCaptured
If we replay:
PaymentCaptured
we should generally be rebuilding or reacting to the fact that it happened.
We shouldn’t accidentally interpret replay as:
Capture the payment again.
That sounds obvious.
But when event-driven systems become complex, this distinction can become blurred.
The semantic meaning of an event matters enormously.
5. Then We Had to Understand Ordering
Kafka provides ordering within a partition.
Not necessarily across an entire topic.
That distinction matters.
Imagine these events:
PaymentCreated
PaymentAuthorized
PaymentCaptured
PaymentSettled
If they belong to the same business entity and are placed in the same partition, consumers can observe them in order.
But if related events are distributed across different partitions, consumers may not receive a globally ordered sequence.
This means partitioning became an architectural decision.
Not simply a performance configuration.
6. Partitioning Became Part of the Business Design
Suppose we partition payments using:
paymentId
Then events for the same payment can remain ordered within a partition.
Partition 0
────────────
payment-A
payment-A
payment-A
Partition 1
────────────
payment-B
payment-B
Partition 2
────────────
payment-C
payment-C
This gives us useful ordering characteristics.
But now another question appears:
What should the partition key be?
Possible choices might include:
paymentId
customerId
accountId
merchantId
Each choice creates a different distribution of workload and ordering behavior.
A poor key can create hot partitions.
A different key can break the ordering assumptions required by a consumer.
Partitioning therefore became part of the architecture.
7. Consumer Offsets Became Important State
Another thing we initially underestimated was the consumer offset.
A consumer isn’t simply:
Read message
Process message
Done
It has a position in the event stream.
Conceptually:
Event 100
Event 101
Event 102
Event 103
Event 104
▲
│
Consumer offset
That offset determines where processing resumes.
If the consumer crashes, restarts, or is intentionally rewound, the offset determines what happens next.
That means offsets are operationally important.
In some workflows, they are effectively part of the processing state.
8. Then Duplicate Processing Happened
One of the most important lessons was that consumers must be prepared for duplicate delivery or duplicate processing.
Consider:
Kafka
│
▼
Consumer
│
├── Process event
│
├── Update database
│
└── Crash before safely recording progress
After restart, the consumer may process the event again.
That means:
Event A
↓
Process
↓
Crash
↓
Restart
↓
Process Event A Again
This isn’t necessarily a Kafka failure.
It’s a consequence of distributed processing.
The consumer needs to be designed accordingly.
9. Idempotency Became Essential
For many consumers, processing the same event twice should produce the same final state as processing it once.
For example:
PaymentCompleted
A consumer might update:
payment.status = COMPLETED
Processing that event twice can still produce:
COMPLETED
That’s relatively safe.
But imagine the consumer does:
account.balance += 100
Processing the same event twice could incorrectly add the money twice.
That is a completely different risk.
So we started designing consumers around idempotency.
Possible strategies include:
- Event IDs
- Processed-event tables
- Idempotency keys
- Unique constraints
- Transactional state updates
- Deduplication logic
The important principle was:
Never assume an event will be processed exactly once simply because the broker is reliable.
10. Event Schemas Became Contracts
As more consumers depended on Kafka, event schemas became just as important as API schemas.
An event like:
{
"paymentId": "pay-123",
"amount": 1000,
"currency": "USD"
}
was no longer owned only by the producer.
Multiple systems depended on its meaning.
Changing:
amount
from an integer representing cents to a decimal representing dollars could silently break consumers.
The JSON could remain perfectly valid.
The architecture could still be completely wrong.
That is why event schema evolution became a formal concern.
11. We Started Treating Events Like APIs
The same principles we applied to APIs started appearing with events.
We needed:
- Schema ownership
- Compatibility rules
- Versioning strategy
- Consumer awareness
- Contract testing
- Documentation
- Deprecation policies
- Observability
For example:
PaymentCompleted v1
│
▼
Consumers migrate
│
▼
PaymentCompleted v2
│
▼
Old consumers removed
│
▼
v1 retired
The event stream had become a contract between teams.
12. New Services Started From Historical Events
This was where the architectural implications became even clearer.
Imagine we already have years of:
PaymentCreated
PaymentAuthorized
PaymentCaptured
PaymentSettled
events.
A new analytics service needs historical payment data.
Instead of asking the transactional database for everything, we could potentially build its state by consuming historical events.
Historical Events
│
▼
New Consumer
│
▼
Analytics Database
This was powerful.
It meant the event stream could act as the source from which downstream state was reconstructed.
But that also meant we had to ask a much more serious question:
Can we still treat Kafka as merely a message broker?
Increasingly, the answer was no.
13. Kafka Became a Second Database
This didn’t mean Kafka replaced our primary database.
It meant something more subtle.
Different parts of the architecture now had different durable representations of business state.
For example:
Primary Database
│
│ business transaction
▼
Payment Service
│
│ publishes events
▼
Kafka
│
├──────────► Fraud State
│
├──────────► Analytics State
│
├──────────► Reporting State
│
└──────────► Notification State
Kafka retained the history of business events.
Consumers derived their own representations.
For those workflows, the event stream had become a durable architectural data source.
That is much closer to a data architecture problem than a simple messaging problem.
14. But Kafka Is Not a Relational Database
This distinction is critical.
Kafka provides:
- Durable ordered logs within partitions
- Retention
- Replay
- Consumer offsets
- High-throughput streaming
A relational database provides very different capabilities:
- Arbitrary queries
- Transactions
- Constraints
- Joins
- Updates
- Rich indexing
- Referential integrity
Kafka is not a replacement for a relational database.
The better mental model is:
Kafka can be a durable event log from which other state is derived.
That distinction prevents many architectural mistakes.
15. Reporting Exposed the Problem
Eventually someone asked:
“Can we run this report directly against Kafka?”
Technically, there are ways to process streams.
But Kafka isn’t designed to be your general-purpose reporting database.
Suppose someone wants:
All settled payments
for customers in region X
between January and March
grouped by merchant
with historical adjustments
That’s a very different access pattern.
We generally want a purpose-built read model or analytical store.
So the architecture became:
Kafka
│
▼
Stream Processing
│
▼
Reporting / Analytics Store
Kafka provided the durable event stream.
The downstream database provided queryability.
16. Replay Became a Powerful Recovery Mechanism
One of the biggest benefits was recovery.
Suppose a consumer had a bug.
Before event streaming, recovery might require:
Database backup
+
Manual correction
+
Data repair scripts
With a durable event stream:
Kafka
│
│ replay
▼
Corrected Consumer
│
▼
Rebuilt State
This can dramatically simplify recovery.
But only if the events contain enough information.
That led to another important design question:
Does the event contain the information required to reconstruct the state we care about?
If it doesn’t, replay cannot magically recover missing information.
17. Event Retention Became a Business Decision
At first, retention looked like an infrastructure setting.
For example:
Retain events for 7 days.
Then:
30 days.
Then:
1 year.
Eventually someone asked:
“How long do we need these events?”
That question isn’t purely technical.
It can involve:
- Recovery requirements
- Audit requirements
- Regulatory obligations
- Data privacy
- Storage cost
- Historical reconstruction
- Operational replay
- Business reporting
Retention became part of data architecture.
Especially in financial systems.
18. Operational Complexity Increased
The benefits were real.
But so was the complexity.
We now had to operate:
Topics
Partitions
Consumer groups
Offsets
Retention
Replication
Schema compatibility
Consumer lag
Rebalancing
Dead-letter handling
Replay procedures
Monitoring
Capacity planning
A database has operational complexity.
So does Kafka.
Once Kafka became a critical business data path, we could no longer treat it as “just infrastructure.”
19. Consumer Lag Became a Business Signal
Initially we monitored:
CPU
Memory
Disk
Network
Eventually we started paying much more attention to:
Consumer lag
Suppose:
Producer rate: 10,000 events/sec
Consumer rate: 8,000 events/sec
The consumer is falling behind.
That might eventually mean:
Payment completed
↓
Event published
↓
Consumer delayed
↓
Downstream state delayed
The infrastructure may look healthy while business state becomes stale.
This is another example of why technical metrics and business metrics must be connected.
20. The Dangerous Part: Assuming Events Are Immutable Truth
Events are durable.
But that doesn’t automatically mean they are perfect historical truth.
Suppose an event contains:
{
"customerName": "John Smith"
}
Years later, the customer name changes.
What does replay mean?
Should the old event still contain the old name?
Usually yes, if the event represents what was true at that point in time.
But if consumers need current customer information, they may need another source.
This is why event design requires clarity around:
What happened?
When did it happen?
What state existed at that moment?
What information is authoritative?
Events are facts about the past.
They aren’t necessarily the current state of every entity.
21. We Had to Separate Facts From State
This became one of the most useful mental models.
An event might say:
PaymentCaptured
The current payment state might be:
payment.status = CAPTURED
The event is historical.
The state is current.
Kafka can retain the history.
A database or materialized view can represent current state.
Event History
│
▼
Kafka
│
┌───────┼────────┐
▼ ▼ ▼
Payment Analytics Fraud
State State State
Different consumers can derive different views from the same event history.
That is one of the most powerful properties of event-driven architecture.
22. What We Actually Changed
Once we realized Kafka had become an architectural data layer, we changed our approach.
We defined clear ownership for topics.
We documented event schemas.
We established compatibility rules.
We monitored consumer lag.
We treated consumer offsets as operationally important.
We designed consumers for duplicate processing.
We documented replay procedures.
We separated transactional state from derived state.
We built appropriate read models for reporting.
And we became much more deliberate about retention.
Kafka was no longer something teams simply published to.
It became part of the system’s architecture.
23. What We Would Not Do
We would not assume:
“Kafka guarantees exactly-once business behavior.”
We would not assume:
“If the event is durable, the consumer is correct.”
We would not use Kafka as a general-purpose query database.
We would not allow teams to change event schemas without understanding consumers.
We would not assume replay is automatically safe.
We would not ignore partition-key design.
We would not treat consumer lag as merely an infrastructure metric.
And we would not let every team create topics without clear ownership.
24. The Questions I Ask Now
When introducing event streaming, I ask:
What is the event actually representing?
Then:
Who owns the event schema?
Then:
How long must it be retained?
Then:
Can consumers safely process duplicates?
Then:
What ordering guarantees does the business require?
Then:
What happens when consumers fall behind?
Then:
Can we replay the stream safely?
And finally:
If we had to rebuild a downstream system from these events six months from now, would we have enough information to do it correctly?
That last question changes how teams design events.
25. The Bigger Architectural Lesson
The biggest mistake is thinking:
Database = data
Kafka = messages
The architecture is often more nuanced.
Kafka can become a durable event history.
Consumers can create derived state.
Those derived stores can power:
- Search
- Reporting
- Fraud detection
- Notifications
- Analytics
- Operational workflows
The moment that happens, event streaming becomes part of the data architecture.
Not just the messaging architecture.
26. Final Thought
We introduced Kafka to decouple services.
It worked.
But the more the system grew, the more the event stream became something else.
It became:
History
+
Replay
+
Integration
+
Recovery
+
Derived State
At that point, Kafka wasn’t merely delivering messages.
It was carrying durable business facts that multiple parts of the architecture depended on.
That created enormous value.
It also created enormous responsibility.
The lesson wasn’t:
“Kafka is a database.”
That’s too simplistic.
The real lesson was:
Once events become durable business data, your message broker becomes an architectural system of record for some workflows.
And when that happens, you need to design it with the same seriousness you apply to any other critical data system.
A message is temporary only when nobody depends on its history. Once the system depends on replay, retention, ordering, and reconstruction, the event stream is no longer just transport. It is architecture.