In a static universe, database sharding is a solved problem. You define your hash modulus, spin up your physical nodes, and route the traffic. But a core banking ledger does not exist in a static universe.
Data volumes grow exponentially. Hotspots emerge—perhaps a massive corporate payroll client monopolizes a partition, or a specific shard’s NVMe drives degrade, shifting I/O latency. Eventually, every sharded ledger faces the same architectural nightmare: you must move millions of active, event-sourced accounts from Shard A to Shard B, and you cannot drop a single transaction to do it.
Changing a hash modulus (e.g., from % 16 to % 32) is trivial in a stateless web application. In an event-sourced financial ledger, it is a high-wire act of distributed state mutation.
1. The Immutable Barrier
You cannot simply SELECT * millions of historical events from Shard A and bulk-insert them into Shard B.
Because the ledger relies on cryptographic hash chaining and strictly ordered event IDs, a bulk insert scrambles the physical write order. Even if you perfectly copy the data, the new shard’s log sequence numbers will not match the original. Any derived balance projections on the new shard will fail their cryptographic verification.
Furthermore, while you are copying historical data, the source shard is actively processing new debits and credits. If you copy the data at 12:00 PM and the cutover happens at 12:05 PM, the new shard is permanently missing five minutes of financial reality.
2. Phase 1: The Dual-Write Interceptor
The architecture for live resharding begins by turning the JVM’s routing layer into a transparent proxy.
When the orchestration engine flags an account range for migration, it updates the distributed configuration. The JVM routing layer immediately enters a dual-write state for those specific accounts. When an AccountDebitedEventis processed, the routing logic calculates the current shard and the targetshard. It writes the event to both Shard A and Shard B simultaneously.
During this phase, all reads are still served exclusively from Shard A. Shard B is building a live, real-time tail of events, but it remains completely dark to user traffic.
The Dual-Write Failure Trap: Dual-writes are notoriously dangerous. What happens if the write to Shard A succeeds, but Shard B throws a network timeout? If the system rolls back the transaction to maintain atomicity, it sacrifices the availability of the source ledger for a migration task—which is unacceptable.
Instead, the architecture must treat Shard B as an eventual replica during the migration window. The write to Shard A (the source of truth) must succeed and commit independently. If the write to Shard B fails, it is recorded in a dead-letter queue, and a background reconciliation worker continuously retries the mutation until Shard B acknowledges it. The migration is allowed to lag, but the primary ledger is never blocked.
3. Phase 2: The Asynchronous Backfill
While the dual-write interceptor handles the real-time “tail” of events, a separate, heavily throttled batch process must copy the massive historical bulk.
This backfill pipeline reads the historical event log from Shard A and replays it into Shard B. Because this is an append-only log, the backfill can be executed using simple, sequential I/O.
To prevent the backfill from starving the OLTP workload on Shard A, the pipeline must implement aggressive pacing—reading batches of events and sleeping between network round-trips to cap disk IOPS.
4. Phase 3: The Consistency Watermark
How does the system know when the backfill has caught up to the dual-write tail?
The architecture requires a deterministic watermarking system. Every event appended to the ledger has a monotonically increasing sequence number. The orchestration engine continuously queries the maximum sequence ID on Shard A (the live ledger) and compares it to the maximum sequence ID on Shard B (the target ledger).
Because the dual-writer is actively feeding new events to Shard B, the gap between the two sequence IDs will steadily shrink. The moment the Shard B sequence ID equals the Shard A sequence ID, the system has achieved state synchronization. The historical backfill has seamlessly merged with the live tail.
5. Phase 4: The Atomic Cutover
The cutover must be instantaneous and fleet-wide.
The orchestration engine flips a feature flag in the configuration store. All JVM instances simultaneously invalidate their local routing caches.
The routing algorithm shifts. The hash modulus is updated. The next time a request arrives for the migrated accounts, the routing layer calculates the new hash, points directly to Shard B, and begins serving reads.
The dual-write interceptor is deactivated. Shard A is left in a dormant state, retaining the immutable historical data for regulatory compliance until it is safely archived to cold object storage.
The Architecture Revealed
Resharding a financial ledger is not a database administration task; it is a distributed systems orchestration problem. By decoupling the migration into a strict sequence of phases—a live dual-write tail, a throttled historical backfill, watermark verification, and an atomic cutover—the system treats its own physical topology as an eventually consistent state machine, capable of rewriting its own layout without ever pausing the flow of money.