Micro-batching

Micro-batching processes data in small batches that run every few seconds rather than every few hours, trading the simplicity of ordinary batch processing for latency much closer to true streaming, without adopting a dedicated event-by-event stream processing engine. Spark Structured Streaming's default mode is a well-known example: it internally slices an incoming stream into small batches and runs its normal batch engine on each one.

The distinction from true stream processing is architectural: a stream processor like Flink handles each event individually as it arrives, while a micro-batch system still waits, briefly, to accumulate a small group before processing it, which adds a small fixed delay but lets the system reuse a batch engine's execution model, fault tolerance and tooling rather than building a separate one. As the batch interval shrinks toward zero, the practical difference between the two approaches narrows.

Micro-batching matters as a pragmatic middle ground: many use cases need data freshness measured in seconds, not milliseconds, and don't justify the added operational complexity of a dedicated streaming engine. It is widely used with Apache Spark for exactly this reason. The main pitfall is treating micro-batch latency as equivalent to true streaming latency when a use case genuinely needs sub-second reaction time, where the batch interval itself becomes an unacceptable bottleneck.