Tag Distributed Systems

How We Handled a Payment System Incident at 2AM

It was 2AM. Most of the engineering team was asleep. The payment system was processing transactions normally. Then an alert fired. Duplicate charges were increasing. Not thousands. Not enough to immediately bring the entire system down. But enough to tell…

The Org Structure That Broke Our Architecture

Three teams. One database. And almost every deployment required coordination between them. The system was technically working. Payments were processing. Settlement jobs were running. Ledger entries were being created. But engineering was becoming slower every month. A change that should…

When a Database Became the Hidden Bottleneck

We had a system that looked healthy. Application CPU was reasonable. Memory was stable. The number of application instances had increased. The load balancer was distributing traffic normally. And yet response times kept getting worse. The first instinct was to…