How to choose a workflow orchestrator

A workflow orchestrator schedules and sequences tasks that depend on each other — extract, then transform, then load; train a model, then evaluate it, then deploy it — and retries or alerts when a step fails. Anyone running more than a handful of cron jobs needs one: cron has no concept of "run B only if A succeeded," no retry logic, and no UI showing what ran and what broke. Teams that don't need one are those with a single, simple, linear job — a nightly script with no branching and no dependents is often fine as-is.

The orchestrators in this category differ less in whether they can run a DAG of tasks — nearly all can — and more in how you author pipelines, where they run, and what happens when something fails at 3 a.m.

Task-centric vs. asset-centric

Apache Airflow and most of its predecessors model a pipeline as a graph of tasks: step A runs, then step B. Dagster inverts this and models a pipeline as a graph of assets — the tables, files, or models a pipeline produces — with tasks as the mechanism that materializes them. The practical difference shows up when something breaks: an asset-centric tool can tell you which downstream tables are now stale because an upstream one failed, because it tracks the data, not just the run. A task-centric tool tells you which task failed and leaves you to work out what that means for the data. If lineage and freshness tracking matter as much as the scheduling itself, weight this heavily; if you just need reliable sequencing, either model works.

How you write pipelines

• Static DAGs in Python. Airflow pipelines are Python that produces a DAG object at parse time. This is battle-tested and every operator you'll ever need already exists, but dynamic, data-dependent pipeline shapes are awkward.
• Plain functions. Prefect turns ordinary Python functions into flows with decorators, so branching, loops, and runtime-determined fan-out read like normal code. This suits pipelines whose shape depends on the data they're processing.
• Declarative YAML. Kestra defines flows in YAML with embedded scripts where needed, which lowers the bar for people who don't want to own a Python codebase and version-controls cleanly as plain text.
• Notebook-style blocks. Mage lets you write and preview each step as an interactive block before chaining it into a scheduled DAG, closer to a Jupyter workflow than a scheduler.
• Visual, no-code canvases. Apache NiFi and Keboola build flows by dragging and connecting pre-built components rather than writing code, which suits integration-heavy routing and teams without engineers to spare.

None of these is strictly better; they trade code-first control for approachability in different amounts. Match the choice to who will actually maintain pipelines in a year, not to who evaluates the tool today.

Batch analytics pipelines vs. everything else

Most tools in this category assume the job is moving and transforming data on a schedule. Two don't:

• Argo Workflows and Flyte run each step as its own Kubernetes container, defined as Kubernetes custom resources. They fit teams that already operate on Kubernetes and want orchestration to inherit the same deployment, RBAC, and scaling model as everything else — and Flyte adds strong typing between tasks, which matters for ML pipelines that need to be reproduced and audited exactly.
• Temporal is not a data-pipeline tool at all. It's a durable-execution platform for long-running application logic — payment processing, order fulfillment, multi-step microservice transactions — where the guarantee that matters is that a workflow's state survives a crash and resumes exactly where it left off. Some data teams use it as an orchestration backend, but evaluate it against that job, not against Airflow.

If your pipelines are DAGs of SQL and Python moving data through a warehouse, look at the batch-oriented tools first. If you're orchestrating containerized ML jobs on Kubernetes, or reliability-critical application logic, the Kubernetes-native and durable-execution tools are a different, better-fitted category.

Self-hosted, managed, or a packaged platform

Most of the open-source orchestrators here — Airflow, Dagster, Prefect, Kestra, Argo, Flyte, Luigi — can be self-hosted at no licensing cost, but running the scheduler, database, and workers is real operational work: upgrades, scaling, and on-call for the orchestrator itself. Three paths exist for avoiding that:

1. A managed distribution of an open-source tool. Astronomer runs actual open-source Airflow — not a fork — and handles infrastructure, upgrades, and support, layering on observability and RBAC Airflow lacks natively. The tradeoff is operational burden versus a recurring bill, not a different pipeline-authoring experience.
2. A vendor's own cloud control plane. Dagster+, Prefect Cloud, and Kestra Cloud offer scheduling, history, and alerting as a hosted layer, usually with the option to keep execution in your own infrastructure.
3. An all-in-one SaaS platform. Keboola bundles ingestion connectors, transformation, and orchestration into one managed product, so there's no separate orchestrator to operate at all — at the cost of building on its platform rather than assembling best-of-breed pieces.

Open source vs. commercial, and what "free" really costs

Most of this category is open source, which is not the same as free. Airflow, Apache NiFi, and Argo Workflows cost nothing to license and everything to run at scale: someone owns upgrades, capacity planning, and incident response. Luigi is open source but has seen its pace of development slow since Spotify's own team moved to other tools — still shipping releases, but worth confirming the project's current activity before betting a new project on it. Commercial and usage-based tools (Astronomer, Temporal Cloud, Flyte's Union.ai) convert that operational cost into a metered bill, which is easier to budget but keeps scaling with usage.

A shortlist by situation

• If you want the deepest ecosystem and the safest hire, Apache Airflow remains the default; pair it with Astronomer if you don't want to run it yourself.
• If you want lineage and freshness tracked around the data a pipeline produces, look at Dagster, especially if you already use dbt.
• If your pipelines have runtime-dependent branching and you want code that reads like ordinary Python, look at Prefect.
• If you want declarative, event-driven flows that non-Python teammates can read, look at Kestra.
• If you're already on Kubernetes and want orchestration and containers to share one operational model, look at Argo Workflows for general batch/CI work or Flyte specifically for typed, reproducible ML pipelines.
• If the job is real-time routing and mediation between systems, not scheduled batch transforms, look at Apache NiFi.
• If you want ingestion, transformation, and orchestration in one managed platform without assembling separate tools, look at Keboola or Mage.
• If the problem is durable execution of business logic, not data pipelines, look at Temporal instead of any of the above.

Questions to ask vendors or in a trial

• Can pipeline structure change dynamically at runtime, or must the DAG be fixed at parse/deploy time?
• What happens to an in-flight run during a deploy or a worker crash — does it resume, restart, or fail?
• How does the tool represent backfills for historical date ranges, and can you re-run just the failed portion of a large DAG?
• Is scheduling idempotent by default, or is that left to the pipeline author?
• What does the self-hosted operational burden actually look like — database sizing, worker scaling, upgrade cadence?
• If evaluating a managed offering, what's billed: deployment-hours, worker-hours, actions, or a flat seat fee — and how does that scale as pipeline count grows?

Common mistakes

• Choosing based on the demo DAG rather than your messiest real pipeline — the one with conditional branches, backfills, and a flaky upstream API.
• Underestimating the ops cost of self-hosting an open-source orchestrator, then discovering a managed option would have been cheaper once on-call time is counted.
• Picking a Kubernetes-native tool without already running Kubernetes, just because it looks modern — the operational overhead of standing up a cluster for orchestration alone rarely pays off.
• Treating Temporal as an Airflow alternative for ETL when it solves a different problem (durable application logic, not data movement).
• Ignoring how backfills and reruns work until the first large historical load is needed in production.

For a closer look at two specific pairings, see Apache Airflow vs Dagster, Apache Airflow vs Prefect, and Dagster vs Prefect. Every tool in this category is listed at /tools/category/workflow-orchestration/.