When Asynchronous Processing Made the System Less Reliable

We had a performance problem.

One part of the application was doing too much work during a synchronous request.

The obvious solution was to move the work out of the request path.

Instead of:

Client
  │
  ▼
API
  │
  ▼
Process Everything
  │
  ▼
Response

we changed the flow to:

Client
  │
  ▼
API
  │
  ▼
Queue
  │
  ▼
Worker
  │
  ▼
Process

The API became faster.

The system could absorb bursts more easily.

Workers could scale independently.

On paper, the architecture was better.

Then we discovered something uncomfortable.

The system had become harder to make reliable.

The problem wasn’t the queue.

The problem was that asynchronous processing had changed the meaning of success.

1. The Synchronous Problem

The original workflow was straightforward.

A request arrived.

The application performed the required work.

The database was updated.

The application returned a response.

Conceptually:

Request
   │
   ▼
Validate
   │
   ▼
Process
   │
   ▼
Update State
   │
   ▼
Response

The advantage was simplicity.

When the API returned success, the operation had completed.

When something failed, the caller generally knew immediately.

But the approach had a limitation.

Some operations were expensive.

During periods of increased traffic, requests spent too much time waiting for downstream processing.

Threads remained occupied.

Queues formed inside the application.

Latency increased.

So we had a reasonable architectural question:

Did this work really need to happen before the API responded?

For some operations, the answer was no.

That led us toward asynchronous processing.

2. The Asynchronous Design

The new design separated accepting the request from performing the work.

The API would validate the request, record the necessary state, publish a message, and return.

The worker would process the message later.

Something like:

                ┌─────────────┐
                │    Client   │
                └──────┬──────┘
                       │
                       ▼
                ┌─────────────┐
                │     API     │
                └──────┬──────┘
                       │
                       ▼
                ┌─────────────┐
                │    Queue    │
                └──────┬──────┘
                       │
              ┌────────┴────────┐
              ▼                 ▼
        ┌───────────┐     ┌───────────┐
        │  Worker 1 │     │  Worker 2 │
        └─────┬─────┘     └─────┬─────┘
              │                 │
              └────────┬────────┘
                       ▼
                  ┌─────────┐
                  │Database │
                  └─────────┘

This gave us several useful properties.

The API no longer had to wait for the entire operation.

The queue could absorb temporary traffic spikes.

Workers could scale independently.

Slow downstream processing didn’t necessarily block incoming requests.

The architecture had become more elastic.

But it had also introduced a new question:

What exactly does it mean when the API says the request succeeded?

3. The Meaning of Success Had Changed

This was the first important realization.

In the synchronous model:

API success
     =
Operation completed

In the asynchronous model:

API success
     =
Request accepted

Those are not the same thing.

The API could successfully accept a request while the actual business operation was still waiting in a queue.

The worker could fail later.

The database operation could fail later.

The message could be retried later.

The consumer could crash after performing the operation but before acknowledging the message.

The system had moved part of the failure window outside the original request.

That wasn’t necessarily bad.

But it meant we had to design for it explicitly.

4. The Failure We Didn’t Expect

The most interesting failure involved message delivery.

Consider a worker processing a message:

1. Receive message
2. Process business operation
3. Commit database transaction
4. Acknowledge message

Now imagine something happens between steps 3 and 4.

The database transaction succeeds.

The business state has changed.

But the worker crashes before the message acknowledgment reaches the broker.

From the message broker’s perspective:

The message was never successfully processed.

So the broker delivers it again.

The worker receives the same message.

It processes it again.

Now we have:

Message
   │
   ▼
Worker
   │
   ▼
Database update ✓
   │
   X
   │
Acknowledgment lost
   │
   ▼
Message redelivered
   │
   ▼
Worker processes again

The system has a perfectly valid retry.

But the business operation has happened twice.

This is where asynchronous systems become fundamentally different from simple request-response systems.

5. The Problem Wasn’t the Retry

At first glance, the natural reaction is:

“Why did the message get delivered twice?”

But duplicate delivery is not necessarily a messaging failure.

In many distributed systems, consumers have to assume that a message may be delivered more than once.

The deeper problem is:

Could our business operation safely be executed twice?

If the answer is no, the consumer needs idempotency.

For example, imagine a worker receiving:

Process Payment
paymentId = 847291
amount = $500

The worker shouldn’t blindly assume that every delivery represents a new payment.

It needs to determine whether:

paymentId = 847291

has already been processed.

Conceptually:

Receive message
      │
      ▼
Check idempotency state
      │
   ┌──┴───┐
   │      │
 New    Already processed
   │      │
   ▼      ▼
Process   Ignore / return safely
   │
   ▼
Record completion

The important point is that message delivery and business execution are different concerns.

6. We Had Created a New Consistency Problem

The synchronous design had hidden much of this complexity because the operation happened within one request flow.

The asynchronous design separated the workflow across time and processes.

Now we had multiple states to reason about:

Request received
       ↓
Message published
       ↓
Message consumed
       ↓
Business operation started
       ↓
Business operation completed
       ↓
Message acknowledged

Those steps don’t necessarily happen atomically.

A failure can occur between almost any two of them.

That creates states such as:

Request accepted
Message published
Business operation not completed

or:

Business operation completed
Message not acknowledged

or:

Message consumed
Worker crashed
Operation status uncertain

The system can still be correct.

But correctness now depends on how these states are modeled and reconciled.

7. The Tempting Fix: Just Make Everything Synchronous Again

At this point, someone could reasonably ask:

Why not just go back to synchronous processing?

Sometimes that is exactly the right answer.

Asynchronous processing isn’t automatically better.

It introduces:

• Queues
• Consumers
• Retry policies
• Duplicate delivery
• Ordering concerns
• Dead-letter handling
• Eventual consistency
• Additional observability
• More operational components

If the work is small and must complete before the caller can continue, synchronous processing may be simpler and more reliable.

But in our case, asynchronous processing solved a real problem.

The API no longer needed to wait for expensive downstream work.

The queue provided buffering during traffic spikes.

Workers could scale independently.

So the answer wasn’t to remove asynchronous processing.

It was to make its failure semantics explicit.

8. The First Change: Idempotency

The most important change was making the consumer idempotent.

Every business operation needed a stable identifier.

For example:

operationId = 847291

The consumer could then determine whether that operation had already been successfully applied.

A simplified model:

┌──────────────────────────┐
│ Idempotency Record       │
├──────────────────────────┤
│ operation_id             │
│ status                   │
│ processed_at             │
└──────────────────────────┘

The exact implementation depends on the business operation and storage model.

The principle is more important:

Repeated delivery must not accidentally create repeated business effects.

This is especially important when the operation involves money, inventory, account state, notifications, or any other non-reversible side effect.

9. The Second Change: Make State Explicit

We also had to stop treating the operation as simply:

SUCCESS / FAILURE

Asynchronous workflows often need more states.

For example:

RECEIVED
   ↓
QUEUED
   ↓
PROCESSING
   ↓
COMPLETED

with possible paths to:

FAILED
RETRYING
DEAD_LETTERED

This makes the lifecycle visible.

It also makes recovery easier.

Instead of asking:

“Did the operation happen?”

we can ask:

“What state is the operation currently in?”

That is a much more useful question in a distributed system.

10. The Third Change: Control Retries

Retries are useful.

But retries without limits can turn a temporary failure into a sustained failure.

Suppose a downstream dependency is unavailable.

The worker fails.

The message is retried.

It fails again.

The message is retried immediately.

Again.

And again.

Now imagine hundreds or thousands of messages doing the same thing.

The retry mechanism itself becomes additional load on the failing dependency.

This creates a feedback loop:

Dependency slows down
        ↓
Requests fail
        ↓
Messages retry
        ↓
More requests arrive
        ↓
Dependency receives more load
        ↓
Dependency slows further

Retries need boundaries.

Depending on the system, that can include:

• Maximum retry attempts
• Exponential backoff
• Jitter
• Dead-letter queues
• Circuit breakers
• Rate limiting
• Explicit recovery workflows

The objective isn’t to retry forever.

It is to give a transient failure a reasonable opportunity to recover without turning the failure into a larger outage.

11. The Fourth Change: Observability

Asynchronous systems also change how you debug production problems.

With synchronous processing, you can often follow:

Request → Service → Database → Response

With asynchronous processing, the request might return before the business operation is completed.

The flow becomes:

Request
   │
   ▼
API
   │
   ▼
Message
   │
   ├─────────────── time passes ───────────────┐
   │                                           │
   ▼                                           ▼
Queue                                      Worker
                                               │
                                               ▼
                                           Database

A log line in the API is no longer enough.

We need to connect the entire workflow.

That means carrying identifiers such as:

requestId
operationId
messageId
correlationId

across the asynchronous boundary.

Then we can answer:

• When was the request received?
• When was the message published?
• When was it consumed?
• How many times was it retried?
• Which worker processed it?
• Did the database operation succeed?
• Was the message acknowledged?
• Did it eventually reach a dead-letter queue?

Without this information, asynchronous failures can become extremely difficult to reconstruct.

12. Ordering Became Another Architectural Question

There was another issue we couldn’t ignore.

Suppose two messages represent changes to the same business entity:

Message A: Increase balance
Message B: Decrease balance

If the system requires strict ordering, processing B before A may produce an incorrect result.

Asynchronous processing introduces questions that synchronous execution may have hidden:

Does ordering matter?

If it does:

Where is ordering guaranteed?

And:

What happens if messages are delayed or retried?

There isn’t one universal answer.

Some systems can process events independently.

Others require ordering per account, customer, transaction, or aggregate.

The important part is not choosing a particular messaging technology.

It is understanding the business semantics before designing the messaging model.

13. Eventual Consistency Wasn’t a Bug

Another misconception we encountered was expecting every part of the system to reflect the change immediately.

In an asynchronous architecture, that may not be possible or even necessary.

For example:

Request accepted
      ↓
Payment event published
      ↓
Payment processing
      ↓
Ledger update
      ↓
Notification

Different components may observe those state changes at slightly different times.

That is eventual consistency.

The problem isn’t eventual consistency itself.

The problem is introducing it without understanding where it is acceptable.

For some workflows:

A few seconds of delay

may be perfectly acceptable.

For others:

The balance must be authoritative before proceeding

may require synchronous confirmation.

The architecture has to follow the business invariant.

Not the other way around.

14. What We Learned About Asynchronous Architecture

The biggest lesson wasn’t:

“Asynchronous processing is dangerous.”

It wasn’t.

Asynchronous processing is extremely useful.

It can provide:

• Better request latency
• Independent scaling
• Traffic buffering
• Decoupling between components
• More resilient processing pipelines
• Better utilization of worker capacity

But those benefits come with a different failure model.

When you introduce a queue, you introduce time.

When you introduce time, you introduce intermediate states.

When you introduce intermediate states, you need to define how the system behaves when something fails between them.

That is the real architectural trade-off.

15. When Asynchronous Processing Makes Sense

I would consider asynchronous processing when:

• The caller doesn’t need the final result immediately
• Work can safely happen later
• Traffic arrives in bursts
• Processing can be separated from request handling
• Workers need to scale independently
• Temporary downstream failures should be absorbed
• The business process naturally consists of stages

But I would be cautious when:

• The caller requires an authoritative result immediately
• Ordering is critical and difficult to guarantee
• The operation has irreversible side effects
• The business cannot tolerate eventual consistency
• The team has limited observability
• Retry and recovery semantics haven’t been designed
• Nobody can clearly explain what happens when a worker crashes halfway through processing

That last point is particularly important.

16. The Question I Ask Now

When someone proposes:

“Let’s make this asynchronous.”

I don’t immediately ask which messaging platform we should use.

I ask:

“What happens if the consumer processes the message and crashes before acknowledging it?”

Then:

“What happens if the message is delivered twice?”

Then:

“What happens if the consumer succeeds but the downstream system times out?”

And:

“How do we know what actually happened?”

If those questions don’t have clear answers, the architecture isn’t ready.

The technology choice is secondary.

The failure model comes first.

17. The Architectural Lesson

Asynchronous processing didn’t make our system less reliable because queues are unreliable.

It made the system harder to reason about because we had changed the consistency and failure model.

We had traded:

Simple request-response

for:

Distributed workflow

That trade-off was worthwhile.

But only after we treated reliability as part of the design rather than something the messaging infrastructure would provide automatically.

The important lesson is:

Asynchronous processing doesn’t remove complexity. It moves complexity into retries, state transitions, ordering, consistency, and recovery.

A queue can absorb traffic.

It cannot decide what correctness means.

A message broker can deliver events.

It cannot make a non-idempotent operation safe.

A worker can retry.

It cannot determine whether the previous attempt already changed the business state.

Those decisions belong in the architecture.

Final Thought

The best asynchronous systems aren’t the ones with the most queues, events, or consumers.

They are the ones where the failure semantics are understood.

Before introducing asynchronous processing, ask:

What work can safely happen later?

What does success mean?

Can the operation happen twice?

Does ordering matter?

Where is consistency required?

What happens when the worker crashes halfway through?

How does the system recover?

If those answers are clear, asynchronous processing can be a powerful architectural tool.

If they aren’t, the queue may make the system faster while quietly making it harder to trust.

Performance is not the same thing as reliability.

And in distributed systems, making something asynchronous often means you need to design reliability more carefully, not less.