For 18 months, we built our own fraud detection system. The reasoning seemed sound. Fraud was important. Payments were core to the business. We had strong engineers. We had access to transaction data. And building the capability ourselves would give…
The requirement sounded simple. We needed to add a new column to a production table. The table had more than a billion rows. The business couldn’t afford hours of downtime. And we couldn’t simply stop the application, modify the database,…
It was 2AM. Most of the engineering team was asleep. The payment system was processing transactions normally. Then an alert fired. Duplicate charges were increasing. Not thousands. Not enough to immediately bring the entire system down. But enough to tell…
The system was slow. But the dashboards looked healthy. We had CPU utilization under 50%. Memory was stable. Error rates were below 1%. Our monitoring alerts weren’t triggering. And yet customers were complaining about slowness. This was the contradiction we…
We had a performance problem. One part of the application was doing too much work during a synchronous request. The obvious solution was to move the work out of the request path. Instead of: we changed the flow to: The…
A downstream service started timing out. So we did what seemed like the responsible thing. We added retries. The thinking was straightforward. If a request fails because of a temporary network problem or a momentary downstream slowdown, try again. The…
