Glossary
Batch processing
Processing data in large, scheduled groups rather than continuously as each record arrives.
Batch processing collects data over a period, an hour, a day, or until a file reaches a certain size, and then processes it all at once as a single job. A nightly job that reads the previous day's orders, aggregates them, and loads the results into a warehouse table is a typical batch pipeline.
This contrasts with stream processing, which handles each event within seconds of its arrival instead of waiting for a batch to accumulate. Batch jobs are usually run by workflow orchestration tools on a fixed schedule or after an upstream dataset lands, and engines built for distributed computing, such as Apache Spark or Hadoop MapReduce, split a batch job's work across many machines to process large volumes within a bounded window.
Batch processing remains the default choice for most analytics because it is simpler to write, test and reason about than a streaming job, and it is cheaper per record when near-real-time results are not required. The main tradeoff is latency: results from a daily batch job are, by definition, up to a day old, which matters for use cases like fraud detection or live dashboards where data freshness requirements can't tolerate that lag.
Last reviewed September 22, 2026