For years, we learned to think about distributed systems in a particular way.
A request enters a service. The service calls another service. That service may call a database, a message broker, a payment provider, or another internal system. Somewhere along the way, networks fail, requests time out, messages arrive twice, state becomes temporarily inconsistent, and dependencies become unavailable.
We built patterns around those problems.
Timeouts. Retries. Idempotency. Circuit breakers. Durable state. Event-driven workflows. Reconciliation. Observability.
After two decades of building Java systems, much of that experience in financial systems, these problems are familiar.
Then AI agents arrived.
At first, an agent can look like another application component:
User
|
v
AI Agent
|
+----> Tool
|
+----> Database
|
+----> External API
But once the agent can make decisions, invoke tools, maintain state, wait for asynchronous results, retry operations, and continue a workflow based on what it observes, something important has happened.
The agent is no longer simply an AI feature.
It has become a participant in a distributed system.
And in financial systems, that distinction matters enormously.
The Agent Looks Simple Until It Touches Reality
Imagine a user says:
“Pay this supplier’s invoice and let me know when the payment is complete.”
From the user’s perspective, this is one request.
Architecturally, it may involve:
User
|
v
Agent
|
+--> Retrieve Invoice
|
+--> Validate Supplier
|
+--> Check Policy
|
+--> Initiate Payment
|
+--> Wait for Result
|
+--> Reconcile Status
|
+--> Notify User
Those operations may cross several independent systems.
The invoice may live in one system.
The supplier information may come from another.
Authorization may belong to a policy service.
The payment may be processed by another service.
The final status may arrive asynchronously through an event.
The notification may be handled somewhere else.
The agent sits across all of these boundaries.
That means the familiar distributed-systems problems immediately return.
The difference is that one component deciding what happens next is now probabilistic.
A Tool Call Is a Distributed Boundary
One of the easiest mistakes to make with agentic systems is to think about a tool invocation like a local method call.
Conceptually, the agent may think:
transferMoney(account, amount)
But architecturally, that operation is much closer to:
Agent
|
| network
v
Tool Boundary
|
| network
v
Financial Service
|
| network
v
System of Record
The important question is therefore not:
“Did the agent call the tool?”
It is:
“What does the system know about the outcome of that call?”
Suppose the agent initiates a payment.
The financial system accepts it.
The transaction is committed.
But the response never reaches the agent because of a network timeout.
The agent sees:
TIMEOUT
The financial system knows:
PAYMENT COMPLETED
These are not the same thing.
The agent is now operating with incomplete information.
That is a classic distributed-systems problem.
The Lost Response Problem
Consider a payment request:
Agent
|
| Transfer $50,000
v
Payment Service
|
v
Ledger
|
| SUCCESS
v
Payment Service
|
| response lost
X
Agent
What does the agent know?
It knows that it did not receive a successful response.
It does not know that the payment failed.
That distinction is fundamental.
The outcome is now:
UNKNOWN
not:
FAILED
This is one of the first principles I would carry from traditional financial systems into agentic architecture:
Unknown is a state. It is not automatically failure.
If the agent responds to an unknown outcome by simply trying the operation again, the system may execute the same business operation twice.
For a payment, that is unacceptable.
Why Idempotency Becomes Even More Important
Retries are normal in distributed systems.
They are also normal in agentic workflows.
An agent may retry because:
- a tool timed out
- an external provider was unavailable
- the response was malformed
- the workflow was interrupted
- the agent interpreted the previous attempt as unsuccessful
But a retry does not necessarily mean a new business operation.
Suppose:
Attempt 1
Payment requested
Payment succeeds
Response lost
The agent sees:
No successful response
and makes:
Attempt 2
Payment requested
Without idempotency, the system could produce:
$50,000 transferred
+
$50,000 transferred again
The architecture therefore needs a durable identity for the business operation.
For example:
workflow_id
operation_id
payment_intent_id
idempotency_key
The important principle is:
A retry of an operation must not automatically become a new operation.
The agent may decide to retry.
The financial system must still recognize whether that retry represents something that has already happened.
This is not an AI-specific principle.
AI simply makes the importance of the principle much harder to ignore.
The Agent Must Not Own Financial Truth
There is another boundary that becomes critical when AI enters financial systems.
The model may know something.
The model may have retrieved something.
The model may have previously seen something.
None of those make the model the source of truth.
Imagine an agent sees:
Available balance: $100,000
A few seconds later, another transaction reduces the balance.
The real balance is now:
$40,000
The agent’s context may still contain:
$100,000
Which value is authoritative?
The ledger.
Not the conversation.
Not the model’s memory.
Not the previous tool response.
The system of record remains the system of record.
This leads to a simple architectural rule:
AI may reason about financial state, but it should not become the authoritative owner of financial state.
The agent can propose an action.
The deterministic financial system must decide whether that action is valid.
Context Is Not Durable State
This distinction becomes particularly important with agentic applications.
An agent may have a conversation containing:
Invoice validated.
Payment requested.
Payment timed out.
User asked for a status update.
That context is useful for reasoning.
But it should not automatically be treated as the authoritative workflow record.
A process can restart.
A model can be called again.
A context window can change.
A conversation can be summarized.
A workflow cannot simply forget what actually happened.
The durable state should exist independently:
PAYMENT_REQUESTED
|
v
PAYMENT_PROCESSING
|
+----> COMPLETED
|
+----> FAILED
|
+----> UNKNOWN
The model can use that state as input.
It should not be responsible for inventing the state.
This is the same separation we have always needed between ephemeral computation and durable business state.
The Agent Slowly Becomes an Orchestrator
As soon as an agent performs a multi-step operation, it starts looking less like a chatbot and more like a workflow orchestrator.
Consider:
1. Find invoice
2. Validate invoice
3. Check supplier
4. Check authorization
5. Create payment
6. Wait for completion
7. Verify result
8. Notify user
That workflow has:
- state
- transitions
- dependencies
- failures
- retries
- timeouts
- asynchronous events
- recovery requirements
The agent may decide what to do next, but the execution environment still needs durable control over the workflow.
Otherwise, imagine the runtime crashes after step 5.
When the system restarts, the critical question is not:
“What does the model think it was doing?”
The question is:
“What actually happened?”
That answer must come from durable system state.
AI Does Not Remove the Need for State Machines
Agentic systems can make workflows feel dynamic.
But dynamic decision-making does not mean the underlying business process should become undefined.
For a financial operation, explicit states are often safer:
PROPOSED
|
v
VALIDATED
|
v
AUTHORIZED
|
v
EXECUTION_PENDING
|
v
PROCESSING
|
+----> FAILED
|
+----> UNKNOWN
|
v
COMPLETED
The agent can help determine what action should be proposed.
The workflow state should remain explicit.
This becomes particularly important when humans, external providers, and asynchronous events participate in the process.
Asynchronous Processing Makes the Boundary Even More Interesting
Financial systems are rarely purely synchronous.
A payment request may return:
ACCEPTED
while the actual completion happens later.
The agent therefore cannot assume:
Request accepted = transaction completed
Instead:
Payment Requested
|
v
Processing
|
| event
v
Completed
The completion event may arrive:
- later
- twice
- out of order
- after a timeout
- after the agent has restarted
Again, these are familiar distributed-system concerns.
The system needs concepts such as:
- correlation IDs
- event IDs
- idempotent consumers
- durable workflow state
- replay handling
- reconciliation
The agent doesn’t eliminate these requirements.
It introduces another participant that has to operate correctly around them.
The Agent Can Amplify Failure
Traditional distributed systems already have a dangerous failure pattern:
slow dependency
|
v
timeout
|
v
retry
|
v
more load
|
v
more timeouts
Now introduce an agent.
A single user request might cause:
Agent
|
+--> Tool call
|
+--> Retry
|
+--> Another tool call
|
+--> Fallback
|
+--> Retry again
The agent can unintentionally amplify a dependency failure.
This means agentic systems need bounded execution.
For example:
Maximum tool calls
Maximum retries
Maximum workflow duration
Maximum parallel operations
Maximum financial exposure
The exact limits depend on the business.
The principle does not:
An agent should not have unlimited ability to continue acting simply because it has not yet achieved its goal.
Circuit Breakers Still Matter
AI does not make classical resilience patterns obsolete.
If anything, they become more important.
Suppose a third-party financial provider is experiencing an outage.
A poorly controlled agent might continue trying:
Call
|
Timeout
|
Retry
|
Timeout
|
Retry
|
Timeout
A circuit breaker creates a controlled boundary:
Agent
|
v
Circuit Breaker
|
+---- OPEN ----> Controlled Failure
|
+---- CLOSED --> Provider
The agent can then reason about:
PAYMENT_PROVIDER_UNAVAILABLE
rather than repeatedly hammering an unavailable dependency.
The model may determine how to communicate the situation.
The resilience layer should determine whether another call is permitted.
Again, intelligence and control belong at different boundaries.
Authorization Cannot Be Probabilistic
One of the most important boundaries in financial systems is authorization.
An agent may interpret:
“Move the money to the supplier.”
But interpreting the instruction does not establish that the operation is authorized.
The architecture should look more like:
User Intent
|
v
Agent
|
| proposed action
v
Authorization / Policy
|
+----> DENIED
|
+----> APPROVED
|
v
Financial Service
The agent can interpret intent.
The authorization system enforces authority.
Natural language is not an authorization mechanism.
This becomes particularly important as agents become capable of performing increasingly powerful operations.
The more capable the agent becomes, the more important the deterministic boundaries around it become.
Human Approval Is Another Distributed Boundary
Some financial actions may require human approval.
That introduces another asynchronous workflow:
Agent
|
v
Payment Proposal
|
v
Awaiting Approval
|
+----> Rejected
|
v
Approved
|
v
Execution
The human might approve immediately.
Or ten minutes later.
The underlying data may have changed.
The payment may already have been cancelled.
Another workflow may have acted on the same invoice.
Human approval therefore does not remove distributed-systems problems.
It adds another participant and another delay.
Approval itself needs durable state.
Observability Has to Follow the Decision
Traditional distributed tracing answers questions such as:
Which service failed?
Agentic systems introduce another question:
Why did the agent make that decision?
Consider:
User Request
|
v
Agent Run
|
+--> Model Decision
|
+--> Tool Call
| |
| +--> Payment Service
|
+--> Model Decision
|
+--> Tool Call
|
+--> External Provider
A production incident may require us to correlate all of those steps.
Useful identifiers might include:
request_id
agent_run_id
workflow_id
tool_call_id
operation_id
idempotency_key
The goal is not simply to log everything.
The goal is to reconstruct the relationship between:
what the system knew, what action was requested, what actually happened, and what the agent did next.
That is a much richer observability problem.
The Agent Is a New Failure Domain
In traditional architecture, we think about failures in:
- databases
- services
- queues
- networks
- external providers
With AI agents, we now also have to consider failures in decision-making.
The agent may:
- choose the wrong tool
- misunderstand the user’s intent
- use stale information
- make an inappropriate retry
- stop before completing the workflow
- continue when it should stop
This does not mean the agent is inherently unsafe.
It means the architecture should assume that the agent can be wrong.
That assumption is healthy.
It leads to a better design:
AI Agent
|
proposed decision
|
v
Deterministic Controls
|
+----------+----------+
| | |
v v v
Policy Validation Limits
| | |
+----------+----------+
|
v
Deterministic Service
|
v
System of Record
The agent is powerful.
But its power is bounded.
What Changes When AI Enters the System?
Interestingly, many of the fundamental problems do not change.
We still have:
- timeouts
- retries
- duplicate requests
- partial failures
- asynchronous processing
- stale state
- authorization
- concurrency
- reconciliation
- observability
- recovery
These are the same problems distributed-systems engineers have been solving for years.
What changes is the nature of one participant.
A traditional service generally follows deterministic application logic.
An agent can dynamically decide:
What should I call?
What should I do next?
Should I retry?
Should I ask for more information?
Should I stop?
That makes the system more dynamic.
It also makes boundaries more important.
The Most Important Architectural Boundary
For financial systems, I would think about the architecture as two worlds.
PROBABILISTIC WORLD
AI / Agent / Reasoning
|
|
Proposed Action
|
v
---------------------
DETERMINISTIC BOUNDARY
---------------------
|
Policy / Validation
|
Authorization
|
Workflow State
|
Financial Services
|
Ledger
The probabilistic side can interpret, reason, summarize, classify, retrieve, and propose.
The deterministic side protects the invariants.
That means:
AI can propose a payment.
The payment system decides whether the payment is valid.
AI can interpret a financial question.
The system of record provides the authoritative data.
AI can recommend an action.
Policy determines whether the action is permitted.
AI can decide to retry.
The receiving operation determines whether that retry is safe.
That distinction is the foundation of reliable agentic architecture.
The Real Shift Is Not “AI Replaces Software”
There is a tendency to describe AI agents as if they represent a complete replacement for traditional application architecture.
In financial systems, that is the wrong mental model.
AI doesn’t remove:
- APIs
- databases
- messaging
- transactions
- state machines
- authorization
- reconciliation
- observability
- resilience engineering
It changes how decisions are made around those components.
The agent becomes another participant in the architecture.
A powerful participant.
A flexible participant.
But still a participant.
The New Distributed System
The simplest mental model I have found is this:
AI Agent
=
Decision Making
+
Workflow
+
Tools
+
State
+
External Dependencies
+
Policies
+
Recovery
Once the agent can affect real-world state, especially financial state, the architecture must assume:
- calls can fail
- responses can disappear
- operations can be duplicated
- information can become stale
- providers can become unavailable
- workflows can crash
- events can arrive late
- decisions can be wrong
These aren’t reasons not to use agents.
They are reasons to engineer them properly.
Final Takeaway
After years of building distributed financial systems, the interesting thing about AI agents isn’t that they can call tools.
Software has been calling other software for decades.
The interesting change is that we have introduced a probabilistic decision-maker into the middle of a distributed system.
The resulting architecture has a new loop:
Reason
|
v
Act
|
v
Observe
|
v
Reason Again
Every action may cross a network boundary.
Every response may be delayed or lost.
Every retry may have consequences.
Every piece of retrieved information may be stale.
And when the system moves money, the consequences are no longer theoretical.
That is why the right mental model is not:
“We added an LLM to our application.”
It is:
“We introduced a probabilistic participant into a distributed system.”
The engineering response is not to make everything deterministic.
It is to put the right boundaries around the uncertainty.
Keep the model flexible.
Keep workflow state durable.
Keep authorization deterministic.
Keep financial truth in the system of record.
Make consequential operations idempotent.
Make failures explicit.
Make recovery possible.
And most importantly:
Let AI decide what might be done. Let deterministic systems decide what is actually allowed and what actually happened.
That is where reliable agentic architecture begins.