Install
Big Data / Large-Scale terms
Lakes, lakehouses and distributed engines for data too big for one machine.
Apache Spark An open-source distributed computing engine for processing large datasets in parallel across a cluster of machines. Batch processing Processing data in large, scheduled groups rather than continuously as each record arrives. Data deduplication Identifying and removing records that represent the same real-world entity or event more than once. DataFrame A two-dimensional, table-like data structure with named, typed columns, used to manipulate data in code. Distributed computing Splitting a computation across multiple machines that coordinate over a network to solve it faster or at larger scale.