The End-of-Day Batch Job: Architecting the Midnight Core Banking Processing Window

In a 24/7 core banking system, the End-of-Day (EOD) batch window is not merely a scheduled script; it is a hard state transition. The ledger must shift from a high-concurrency, random I/O-bound OLTP state (processing real-time payments) to a sequential, compute-heavy OLAP state (interest accrual, fee generation, and General Ledger reconciliation).

The architectural challenge of the midnight window is executing massive, monolithic data mutations across millions of accounts without violating ACID properties, thrashing the database I/O subsystem, or bleeding into the next business day’s OLTP peak.

The JVM Memory Trap: Cursor-Based Streaming

The most common failure mode in EOD processing is the OutOfMemoryError induced by loading entire account tables into JVM heap space. A 20-million-row account table cannot be materialized into a List<Account> or a standard JPA/Hibernate @Entity graph.

EOD processing must be strictly stream-based. Using plain JDBC, this requires explicit configuration to disable the default behavior of the JDBC driver (which often attempts to fetch the entire ResultSet into memory regardless of fetchSize):

With a server-side cursor, the database only materializes the 5,000 rows requested per network round-trip. The JVM heap footprint remains flat, regardless of whether the job processes ten thousand or ten million accounts.

Chunk-Oriented Processing and Transaction Boundaries

Streaming data dictates the transaction strategy. Wrapping 20 million updates in a single database transaction is architecturally impossible—it would generate an unbounded WAL (Write-Ahead Log), exhaust undo segments, and guarantee a catastrophic rollback time if the JVM crashes at row 19 million.

The standard architectural pattern is Chunk-Oriented Processing. The stream is consumed in fixed sizes (e.g., 1,000 records).

1. Read: Stream 1,000 accounts into a lightweight DTO (not a JPA entity).
2. Process: Apply the business logic (e.g., calculate daily overdraft interest).
3. Write: Execute a bulk UPDATE using a single prepared statement with batched parameters.
4. Commit: Commit the transaction.
5. Checkpoint: Record the successful commit in a framework-specific ExecutionContext.

This creates a highly deterministic boundary: if the JVM dies, it only loses the work of the current incomplete chunk. Upon restart, the job queries the checkpoint table, repositions the server-side cursor to the last committed offset, and resumes exactly where it left off.

Partitioning for Parallelism Without Contention

A single-threaded EOD job running at 50ms per account will take 11.5 days to process 20 million accounts. Parallelism is mandatory, but parallelism in a financial batch introduces severe lock contention if not architected correctly.

The dataset must be partitioned by a natural, non-overlapping key—typically account_id ranges or the underlying database sharding key. If the system uses 8 partitions, 8 worker threads are spawned.

Crucially, these threads must operate on physically distinct database partitions or index leaf nodes. If two threads attempt to update accounts that share the same database block, the RDBMS buffer pool will encounter buffer busy waits, and the Latch contention will degrade performance below single-threaded baseline speeds.

To prevent thread starvation and manage database connection pool exhaustion, the partitioning executor must be strictly bounded:

ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(8);
executor.setMaxPoolSize(8); // Bounded to match partition count
executor.setQueueCapacity(0); // Force rejection if a thread is blocked, rather than queuing

The Snapshot Isolation Problem

EOD jobs—particularly interest accrual—require a mathematically consistent view of the ledger. If an online credit card payment sneaks in via the OLTP API at 00:14 AM while the batch job is streaming account balances, the batch job will read an inconsistent state (the balance before the payment for some accounts, and after the payment for others).

Relying on REPEATABLE READ or SERIALIZABLE isolation levels across a 3-hour batch job is dangerous. Long-running transactions under strict isolation will block the OLTP system’s garbage collection or vacuuming processes, eventually halting the entire database.

The engineering solution is As-Of Temporal Tables or a Ledger Snapshot Copy. Before the EOD window opens, a highly optimized CREATE TABLE eod_snapshot AS SELECT * FROM accounts is executed. Alternatively, if the database supports it (like PostgreSQL with temporal tables or Oracle Flashback), the batch job queries the data using a specific System Change Number (SCN) or transaction timestamp:

SELECT * FROM accounts AS OF TIMESTAMP TO_TIMESTAMP('2023-10-25 00:00:00', 'YYYY-MM-DD HH24:MI:SS')

This completely decouples the batch read-set from the OLTP write-set, allowing the online system to continue processing real-time transactions without locking conflicts, while the batch job operates on a frozen, consistent ledger state.

Coexistence: Throttling the Batch to Protect OLTP

While modern architectures push EOD to read-replicas or separate columnar stores, many legacy and mid-tier systems still run batch against the primary OLTP database. In this topology, the batch job is a hostile neighbor.

If 8 parallel batch threads aggressively push bulk updates, they can monopolize the database’s I/O bandwidth and CPU, causing latency spikes for early-morning mobile banking logins.

The batch architecture must implement Pacing. Rather than executing executeBatch() as fast as the CPU allows, the writer thread must instrument its throughput and dynamically sleep to cap IOPS.

if (System.currentTimeMillis() - lastPaceCheck > 1000) {
    long currentIops = getProcessedCount() - lastCount;
    if (currentIops > TARGET_IOPS_LIMIT) {
        Thread.sleep(calculatedBackoffMillis); // Yield I/O bandwidth
    }
}

Furthermore, connection pooling must be strictly segmented. The EOD batch must be assigned a dedicated DataSource and connection pool. If the batch job exhausts its pool, it stalls its own workers; it does not starve the OLTP API’s connection pool.

General Ledger (GL) Posting: The Final Roll-up

The culmination of the EOD window is the aggregation of millions of customer-level ledger entries into a summarized General Ledger.

Iterating through the customer ledger table to sum balances is an O(N) table scan that is aggressively penalized by the database optimizer. Instead, the EOD job must maintain running totals in memory during the chunk-processing phase.

As each chunk of 1,000 accounts is processed, the resulting debits and credits are aggregated into a local Map<GL_Account_Code, BigDecimal>. At the end of the partition processing, a single bulk INSERT is executed into the GL table. This reduces the GL write I/O from millions of rows to a few hundred rows, completing the EOD window and handing the system back to the OLTP domain with zero residual performance debt.