Schema evolution

Schema evolution is the capability of a table format or system to handle structural changes over time — a new column added, an old one dropped, a type widened, a field renamed — without requiring every existing file or row to be rewritten, and without breaking queries or pipelines that reference the table.

In older file-based systems, a schema change often meant rewriting the entire dataset or maintaining brittle, manual compatibility logic; modern open table formats such as Apache Iceberg track schema changes as metadata operations, mapping old and new field definitions by a stable ID rather than by column position, so historical Parquet files written under an old schema remain readable alongside new ones written under the current schema.

Schema evolution matters wherever source systems or upstream teams change their data shape independently of the pipelines and reports that consume it, which is constantly true in most organizations; without it, a single upstream change can silently break downstream jobs or reports. Formalizing what changes are allowed and how consumers are notified is the job of a data contract, which is increasingly paired with schema evolution support as organizations adopt data mesh-style ownership, where each team controls its own data's structure.