Workflow orchestration

Workflow orchestration is the practice of scheduling, sequencing and monitoring a set of interdependent data tasks so that each one runs only after the tasks it depends on have completed successfully. An orchestrator such as Apache Airflow, Dagster or Prefect does not do the data processing itself; it decides when and in what order other systems, a Spark job, a SQL transformation, an API call, should run.

Pipelines are almost always modeled as a directed acyclic graph, where nodes are tasks and edges are dependencies, which rules out circular waits. The orchestrator tracks the state of every run, retries failed tasks, alerts on missed SLAs, and can trigger a backfill to reprocess historical periods. This differs from a plain scheduler like cron, which fires jobs on a clock with no awareness of whether upstream tasks actually succeeded.

Orchestration matters because production pipelines chain many dependent steps across different systems, and a failure partway through needs to be caught and made visible rather than silently propagating bad or missing data downstream. A common pitfall is building tasks that are not idempotent, so a retry after a partial failure produces duplicate or incorrect results; another is letting orchestrator configuration become the primary home for business logic, which makes pipelines hard to test in isolation.