Data pipeline

A data pipeline is the automated sequence of steps that moves data from one or more source systems to a destination, typically a data warehouse or data lake, applying extraction, transformation and loading logic along the way. It is the general term that ETL and ELT both describe specific orderings of.

Pipelines can run in batches on a schedule, be triggered by events, or stream continuously using change data capture. They are usually coordinated by an orchestration tool that manages dependencies between steps, retries failures and alerts when something breaks, since a pipeline is only as useful as its reliability. Modern pipelines increasingly separate ingestion, transformation with tools like dbt, and orchestration into distinct, composable layers rather than one monolithic job.

Pipelines matter because almost all analytics work depends on data arriving correctly and on time; a broken or silently wrong pipeline undermines every dashboard and model downstream of it. Common pitfalls include insufficient monitoring, so failures or subtly wrong data go unnoticed without data observability, undocumented dependencies between steps, and tightly coupling business logic to a specific tool instead of keeping transformations portable, which also supports better data quality.