Apache Spark

Apache Spark is an open-source engine for processing large volumes of data by splitting the work across many machines in a cluster and running it in parallel. It exposes this through high-level APIs, most commonly the DataFrame API, so engineers write transformations, filter, join, aggregate, without manually managing how work is distributed across the cluster.

Spark keeps intermediate data in memory across steps of a computation where possible, which is what made it much faster than its predecessor, Hadoop MapReduce, for iterative workloads like machine learning training that repeatedly reuse the same data. It supports both batch processing, its original and still most common use, and stream processing through Structured Streaming, which internally runs streaming data through the same batch engine in small increments, and can read directly from sources like Apache Kafka.

Spark matters because it is one of the most widely adopted engines for large-scale distributed computing in data platforms, supported by every major cloud provider and integrated into most modern data stacks. The main pitfalls are under- or over-provisioning cluster resources for a job's actual size, and writing transformations that trigger unnecessary data shuffles across the network between machines, which is usually the biggest performance cost in a Spark job.