Apache Parquet

Apache Parquet is an open-source file format that stores data in columnar storage layout, with a self-describing schema and metadata embedded in the file itself. It is the de facto standard format for storing analytical datasets in object storage such as a data lake, and is read and written by essentially every modern query engine and processing framework.

A Parquet file organizes data into row groups, each holding column chunks that are compressed and dictionary-encoded independently, and it stores per-column statistics like min and max values, which lets a reading engine skip whole chunks that cannot match a query's filter without decompressing them, a technique called predicate pushdown. This differs from row-oriented formats like CSV or Avro, which must scan whole rows even when only one column is needed.

Parquet's popularity comes from being open, widely supported, and efficient for the read-heavy, wide-table scans typical of analytics, but a raw Parquet file has no built-in support for transactional updates, deletes, or concurrent writers — that gap is what open table formats such as Apache Iceberg add on top of a set of Parquet files. Schema changes across files, such as a renamed or newly added column, need careful handling and are a common source of read errors if not managed explicitly.