Change data capture (CDC)

Change data capture is a technique for identifying and capturing changes, inserts, updates and deletes, in a source database as they happen, and delivering only those changes downstream instead of repeatedly extracting the entire dataset. Implementations commonly read a database's transaction log, though simpler approaches poll for updated timestamps.

CDC is usually a component within a larger data pipeline, feeding an ETL or ELT process rather than replacing it. The difference from a full batch extract is efficiency and freshness: a nightly full reload of a large table can be slow and resource-intensive, while CDC keeps a downstream data warehouse close to real time by moving only the delta. This makes near-real-time reporting and operational analytics practical without constantly re-scanning source systems.

CDC matters most for large, frequently updated tables and for use cases where freshness matters, such as fraud detection or operational dashboards. Pitfalls include log-based CDC being sensitive to source database configuration and retention settings, difficulty handling schema changes in the source system gracefully, and the added operational complexity of running and monitoring a streaming component, which benefits from data observability tooling.