An experience-informed composite of patterns that emerge in complex engineering environments.
The strange thing about bad architecture is that it rarely begins with a stupid decision.
It usually begins with a reasonable decision.
The engineers are experienced.
The design looks clean.
The benchmarks look good.
The architecture review goes well.
Everyone leaves the room believing they made the right call.
And then, six months later, production tells a different story.
It Looked Like the Right Architecture
Imagine a team designing a high-volume financial transaction platform.
The system needs to process requests quickly, remain available during traffic spikes, and continue operating when downstream dependencies become slow or unavailable.
The team does what good engineers do.
They benchmark the database.
They test throughput.
They review failure scenarios.
They introduce asynchronous processing where appropriate.
They add retries.
They introduce caching.
They separate services around business capabilities.
The architecture looks modern.
It looks scalable.
It looks resilient.
The review concludes:
Approved.
Nothing about the decision looks obviously wrong.
And that’s the problem.
Six Months Later, the System Is Under Pressure
Traffic increases.
One downstream dependency becomes slower than usual.
A few requests start timing out.
The client retries.
The service retries.
The message queue begins accumulating work.
Consumers fall behind.
Database connections increase.
Latency rises.
More requests time out.
More retries arrive.
Suddenly, a component that was designed to improve resilience is participating in a failure-amplification loop.
Nobody intentionally designed the system to behave this way.
Every individual decision had a reasonable explanation.
The problem was the interaction between the decisions.
And that’s where architecture becomes much more interesting.
Nobody Made a Stupid Decision
Looking back, you can find explanations for almost everything.
The database team optimized query performance.
The application team optimized request latency.
The platform team optimized availability.
The product team optimized delivery time.
The operations team wanted automatic recovery.
The architecture team wanted service isolation.
Everyone was doing their job.
Everyone had evidence.
Everyone had good intentions.
And yet the overall system had become fragile.
The problem wasn’t a lack of intelligence.
The problem was that the relevant knowledge was distributed across different boundaries, while the decision process wasn’t designed to bring that knowledge together.
Each team could see its own problem.
Nobody had a strong enough view of the entire system.
Then the Uncomfortable Questions Started
Why did we choose this architecture?
Why didn’t someone challenge it?
Why did the architecture review approve it?
Why did the team believe the retry strategy was safe?
Why did we think asynchronous processing would reduce risk?
Why did we optimize individual services instead of the end-to-end transaction?
And eventually:
Why did smart people collectively make a decision that nobody would have chosen if they could see the final outcome?
That question is more important than:
“Who made the mistake?”
Because blaming one person doesn’t explain the system.
The Forces Behind the Decision
When you examine these situations closely, several patterns appear.
1. Local Optimization
The team responsible for the payment service optimized payment-service performance.
The team responsible for the database optimized database performance.
The team responsible for messaging optimized message throughput.
Each optimization made sense locally.
But the customer doesn’t experience services individually.
The customer experiences one transaction.
A locally successful decision can therefore become a globally expensive one.
2. Familiarity
The team had used the same architecture pattern successfully before.
So when the new problem appeared, the familiar solution felt safer.
That’s natural.
Experience gives us useful patterns.
But experience can also create a dangerous shortcut:
“We’ve done this before.”
The better question is:
“Are the constraints still the same?”
The architecture may look familiar while the business, traffic, failure modes, and risk profile are completely different.
3. Deadline Pressure
The business needed the system quickly.
The team had a delivery commitment.
There was no time to explore every architectural alternative.
So the team selected a solution that was good enough to move forward.
Again, perfectly reasonable.
The problem was that nobody explicitly identified:
Which compromises are temporary?
and
Which compromises are becoming architecture?
Temporary decisions have a way of becoming permanent when nobody owns the decision to revisit them.
4. Authority
The most senior person in the room supported the design.
Nobody strongly disagreed.
Not necessarily because everyone agreed.
Sometimes people simply didn’t have enough confidence to challenge the decision.
Sometimes they thought:
“Someone else must have considered that.”
Silence can look like consensus.
It isn’t.
A healthy architecture process doesn’t depend on the most junior engineer having enough courage to challenge the most senior person.
It creates a structure where challenge is expected.
5. Incentives
The team was measured on delivery.
The architecture team was measured on standards.
The platform team was measured on infrastructure reliability.
The business was measured on growth.
Each metric was reasonable.
But the system didn’t have one metric representing the end-to-end outcome.
So everyone optimized what they could see.
This is how organizations can produce locally successful teams and globally fragile systems.
6. Missing Information
The people who understood the downstream reconciliation consequences weren’t involved early enough.
The people who understood the operational failure modes joined the discussion late.
The people who understood customer impact weren’t part of the technical decision.
Nobody was deliberately hiding information.
The organization simply didn’t bring all the relevant context into the room.
That distinction matters.
Sometimes the problem isn’t that people have the wrong information.
The problem is that the right information never reaches the decision.
The Real Problem: The Decision System
The obvious lesson is:
“Be more careful when designing systems.”
That’s not enough.
Engineers are already trying to be careful.
The deeper lesson is:
Architecture quality depends partly on the quality of the environment in which architectural decisions are made.
You can have brilliant engineers inside a poor decision-making system.
And you can have very good engineers inside a system that consistently helps them make better decisions.
Those are very different organizations.
The question therefore changes from:
“How do we make engineers make better decisions?”
to:
“How do we build an environment that helps engineers make better decisions?”
That is a much more interesting leadership problem.
So What Should Change?
The answer isn’t another architecture framework.
It’s not another mandatory review.
And it’s certainly not:
“Let’s hire smarter engineers.”
The better approach is to improve the decision system itself.
Here are the changes that make the biggest difference.
1. Change the Question
Don’t ask only:
“Will this service scale?”
Ask:
“What happens to the entire business transaction when this service is under pressure?”
The second question forces system-level thinking.
It moves the conversation from component performance to business outcomes.
2. Make Trade-Offs Explicit
Every architecture decision optimizes something.
Performance.
Cost.
Availability.
Consistency.
Simplicity.
Speed of delivery.
Flexibility.
Security.
You cannot maximize everything.
So write down:
What are we optimizing for?
And:
What are we consciously sacrificing?
That simple discipline makes hidden risk visible.
A trade-off that is visible can be managed.
A trade-off that remains hidden becomes a future surprise.
3. Bring the Right People Into the Decision
Not every person needs to attend every architecture meeting.
But decisions that cross boundaries should include the people who understand those boundaries.
For a financial transaction, that might mean involving people who understand:
- Transaction processing
- Reconciliation
- Fraud
- Operations
- Customer impact
- Data consistency
- Regulatory requirements
The goal isn’t more meetings.
The goal is better context.
4. Make Disagreement Part of the Process
Don’t depend on someone being brave enough to challenge the architecture.
Make challenge expected.
Ask:
“What would make this architecture fail?”
Then ask:
“What assumption are we least certain about?”
And:
“If this decision turns out to be wrong, how will we know?”
These questions create constructive friction.
Good architecture needs friction.
Not political friction.
Intellectual friction.
The strongest architecture discussions aren’t the ones where everyone agrees quickly.
They are the ones where assumptions are exposed before production exposes them for us.
5. Distinguish Reversible From Irreversible Decisions
Some decisions can be changed next quarter.
Others create years of dependency.
A good organization shouldn’t spend equal energy on both.
The more expensive a decision is to reverse, the more carefully its assumptions should be examined.
This is particularly important for:
- Data models
- Service boundaries
- Public APIs
- Event contracts
- Ledger architecture
- Identity models
- Organizational ownership
Not every decision deserves a six-week review.
But the decisions that are expensive to reverse deserve more thought before they become embedded in the organization.
6. Revisit Decisions When the Context Changes
An architecture decision is not a permanent truth.
It is a decision made under a particular set of assumptions.
Traffic changes.
Teams change.
Products change.
Regulations change.
Business models change.
Technology changes.
Therefore:
A decision that was correct three years ago can become wrong without anyone having made a mistake.
The important question isn’t:
“Who made the original decision?”
It’s:
“Are the assumptions behind the decision still true?”
That changes architecture from a one-time approval process into a continuous learning process.
The Leadership Responsibility
This is where the subject moves beyond architecture.
A senior leader cannot personally make every technical decision.
Nor should they.
The leader’s responsibility is to create the context in which those decisions are made.
That means deciding:
- What behavior gets rewarded
- What metrics matter
- Who has decision authority
- How disagreement works
- What risks are acceptable
- How failures are discussed
- How teams are organized
- How architecture connects to business outcomes
A leader may never write the retry logic.
But the leader can create the environment that determines whether someone feels comfortable asking:
“Are we sure retries are actually making this system more resilient?”
That’s leadership.
Leadership isn’t making every architectural decision.
Leadership is shaping the conditions under which architectural decisions are made.
When Organizational Design Becomes Architecture
And there is another level above leadership.
The organization itself creates constraints.
If teams are organized around individual components, the architecture may become component-oriented.
If teams are rewarded independently, local optimization becomes natural.
If the organization prioritizes short-term delivery above resilience, technical debt becomes predictable.
If architecture decisions require excessive approval, teams may avoid challenging the status quo.
If failure is punished, people hide problems.
If failure is studied, organizations learn.
These aren’t simply cultural observations.
They eventually become technical properties of the system.
Organizational structure becomes service boundaries.
Incentives become engineering priorities.
Communication patterns become integration patterns.
Decision rights become architecture governance.
Culture becomes operational behavior.
What looks like a technology problem may therefore have an organizational cause.
And what looks like an organizational problem may eventually appear as a technology failure.
The Bigger Lesson
This is why I don’t think architecture can be separated completely from human behavior.
A distributed system is built from services, databases, queues, APIs, and networks.
But the architecture itself is produced by:
people making decisions,
inside
teams with incentives,
inside
organizations with structures,
operating under
business constraints.
Change the technology and you change the system.
But sometimes:
Change the human system, and the technology changes with it.
That is the connection between technical architecture and what I call Human Infrastructure.
Human Infrastructure isn’t separate from technology.
It is the layer underneath it.
Five Questions I Want Every Important Architecture Decision to Answer
Before approving a significant architectural decision, ask:
1. What are we optimizing for?
If the answer is “everything,” we haven’t made the trade-offs explicit.
2. Who benefits from this decision?
And equally:
Who carries the risk?
3. What assumptions are we making?
And which assumption, if wrong, would cause the greatest damage?
4. Who is missing from this conversation?
The person with the missing context may be more important than another person with another opinion.
5. What happens to the business process when this component fails?
Because customers don’t experience your architecture diagram.
They experience the outcome.
The Final Thought
Smart engineers will make bad decisions.
Experience won’t eliminate bias.
Architecture reviews won’t eliminate uncertainty.
Deadlines won’t disappear.
Business pressure won’t disappear.
The goal isn’t to create perfect engineers.
The goal is to create an environment where:
good decisions become easier to make,
bad assumptions become easier to challenge,
and
mistakes become easier to detect before they become expensive.
Because good architecture isn’t simply the product of smart engineers.
It is the product of a system that helps smart people make better decisions.
And that system is part of what I call: Human Infrastructure.