The system was becoming increasingly difficult to change. Every time we added a new capability, another service needed to know about it. A payment completed. The ledger needed to update. Notifications needed to be sent. Fraud analytics needed the transaction.…
It was 2AM. Most of the engineering team was asleep. The payment system was processing transactions normally. Then an alert fired. Duplicate charges were increasing. Not thousands. Not enough to immediately bring the entire system down. But enough to tell…
We had a performance problem. One part of the application was doing too much work during a synchronous request. The obvious solution was to move the work out of the request path. Instead of: we changed the flow to: The…
A downstream service started timing out. So we did what seemed like the responsible thing. We added retries. The thinking was straightforward. If a request fails because of a temporary network problem or a momentary downstream slowdown, try again. The…
