Query engines & federation · Dask (NumFOCUS-sponsored open-source project)

Dask

Open-source Python library that parallelizes NumPy, pandas and scikit-learn workflows across cores or a cluster.

Visit Dask Alternatives

Dask is an open-source Python library for parallel and distributed computing that extends familiar libraries such as pandas, NumPy and scikit-learn to datasets and workloads too large for a single machine's memory, while keeping the same programming style analysts and data scientists already use. It builds a task graph of operations and schedules them across available cores on a laptop or across a distributed cluster, without requiring a rewrite into a new API the way moving to Spark often does. Dask includes Dask DataFrame and Dask Array for scaling pandas/NumPy-style code, and Dask-ML for scaling scikit-learn-style model training. It is self-hosted by default and integrates with Kubernetes, YARN and HPC schedulers; the company Coiled, founded by core Dask maintainers, offers managed cluster hosting with usage-based pricing. Dask is generally preferred over Spark by teams already standardized on the Python/pandas ecosystem.

At a glance

Vendor Dask (NumFOCUS-sponsored open-source project)
Pricing model Open source + paid options
Free tier Yes
Deployment Self-hosted
Open source Yes (BSD-3-Clause)
Best for Python/pandas teams scaling existing analysis or ML code beyond a single machine without switching ecosystems.

Pricing

Free and open source; managed cluster hosting for Dask is sold separately by Coiled on a usage-based basis.

Pricing has not been verified yet — see the vendor's site.

Features

  • Parallelizes pandas, NumPy and scikit-learn code with minimal rewrites
  • Dynamic task-graph scheduling
  • Scales from a laptop to a distributed cluster
  • Dask-ML for distributed model training
  • Integrates with Kubernetes, YARN and HPC job schedulers
  • Lazy evaluation for large-than-memory datasets

Integrations

Profile last reviewed September 21, 2026

Alternatives

Ray Open-source Python framework for distributing machine-learning training, tuning and general compute across a cluster. Apache Spark Open-source distributed processing engine for large-scale batch, SQL, streaming and machine-learning workloads.

Dask in the index now

Terms to know

Machine learning Building models that learn patterns from data to make predictions or decisions, rather than following hand-coded rules. Data pipeline The automated sequence of steps that moves data from source systems to a destination, transforming it along the way. Distributed computing Splitting a computation across multiple machines that coordinate over a network to solve it faster or at larger scale.

Related guides

How to choose a distributed query engine This category bundles two different jobs — federating SQL across existing systems, and distributed compute for pipelines and ML — start by telling them apart.