Batch Jobs Fail Quietly
When a screen breaks, users tell you right away. When the settlement batch that runs at 2 a.m. fails, nobody notices. The problem only surfaces the next morning, when the sales team asks why yesterday's revenue is missing.
The cause is usually structural.
Scheduling and Dependency Management
The most common incident comes from using clock time to enforce order. Schedule order aggregation at 01:00 and settlement at 02:00, and everything works while aggregation finishes in its usual 40 minutes. On the day data growth pushes it to 70 minutes, settlement runs against half-aggregated data. It finishes as a "success" with no error, which makes it more dangerous.
A central scheduler is worth considering once you have more than 20 jobs, dependency chains three or more levels deep, or jobs running on two or more servers. Before that, build the job inventory. Putting job name, run time, upstream jobs, typical duration, and owner in a single table is often enough to expose missing jobs and jobs with no owner.
Building Batches That Are Safe to Fail
Design on the assumption that the batch will fail. The key question is whether it is safe to run again.
Monitoring and Alerting
Failure alerts are not enough. If the server was off or the cron entry was deleted, there is no failure to report. Four things need detecting.
Alerts need a primary owner and a secondary who is paged after 30 minutes without a response. A message posted only to a channel everyone receives gets read by no one.
Operations Handover
The person who built a batch is rarely the person who runs it. Each job needs a reprocessing runbook that covers:
Plan for a growing batch window as well. With data growing 30% a year, a 40-minute job takes about 68 minutes two years later. Splitting it into four partitions by account or date range and running them in parallel brings it down to roughly 17 minutes in theory. In practice database load makes it slower, so tune the degree of concurrency by measurement.
How POLYGLOTSOFT Helps
Batch operations are not something you build once. They need attention every time data and business rules change. Within a maintenance (SM) contract, POLYGLOTSOFT continuously improves your batch estate: organizing the job inventory, redesigning jobs for idempotency and restart, setting up monitoring for missed runs, delays, and abnormal counts, and writing reprocessing runbooks. If checking the settlement results is the first thing you do every morning, get in touch with POLYGLOTSOFT.