ELT & data integration · Google

Google Cloud Dataflow

Fully managed, serverless service for running unified batch and streaming data pipelines built on Apache Beam.

Visit Google Cloud Dataflow Alternatives

Dataflow is Google Cloud's managed execution engine for Apache Beam pipelines, handling both batch and streaming data processing with the same programming model. Beam pipelines can also run on other engines (Spark, Flink), but Dataflow is the fully managed, autoscaling option with no clusters to provision. Its Streaming Engine and Dataflow Shuffle features offload state and shuffle operations from worker VMs to a managed backend, improving autoscaling and reducing worker resource needs. Flexible Resource Scheduling (FlexRS) delays batch job start within a window to use cheaper preemptible-mixed capacity. Billing is metered across several independent dimensions (worker vCPU/memory time, Shuffle or Streaming Engine data processed, and Persistent Disk), which reflects usage precisely but is less predictable than fixed-fee tools.

At a glance

Vendor Google
Pricing model Usage-based
Free tier No
Deployment Cloud
Open source No
Best for Teams running Apache Beam pipelines that need one engine for both batch and streaming without managing infrastructure.

Pricing

Metered pricing billed for worker vCPU-hours and memory GiB-hours, plus separate per-GB charges for Dataflow Shuffle (batch) or Streaming Engine (streaming) data processed; rates vary by region and machine type.

Plan Price Notes
Batch, streaming & FlexRS Usage-based: billed per vCPU-hour and per GiB-hour of worker memory FlexRS offers a lower blended rate by delaying batch job start within a scheduling window; no published flat rate
Dataflow Shuffle / Streaming Engine Usage-based: billed per GB of data processed Offloads shuffle/state from workers to a managed backend, separate from vCPU/memory charges

Prices read from the vendor's own page on September 21, 2026. Vendors change prices; check the source before you budget.

Features

  • Unified batch and streaming via Apache Beam
  • Fully managed autoscaling, no cluster management
  • Streaming Engine for offloaded state management
  • Dataflow Shuffle for offloaded batch shuffle
  • Flexible Resource Scheduling (FlexRS) for cheaper batch runs
  • Native integration with BigQuery and Pub/Sub

Integrations

Profile last reviewed September 21, 2026

Alternatives

AWS Glue Serverless AWS ETL service using Apache Spark under the hood, billed per Data Processing Unit-hour with no infrastructure to manage. Azure Data Factory Microsoft's cloud ETL/ELT and orchestration service for building and scheduling data pipelines across Azure and hybrid sources. Apache Kafka Open-source distributed event streaming platform for high-throughput publish-subscribe messaging and durable event logs. Google Cloud Pub/Sub Google Cloud's fully managed, serverless messaging service for asynchronous event ingestion between applications and analytics pipelines.

Google Cloud Dataflow in the index now

Terms to know

ETL (extract, transform, load) A data integration pattern that transforms data before loading it, unlike ELT which transforms after loading. ELT (extract, load, transform) A data integration pattern that loads raw data first and transforms it afterward using the warehouse's own compute. Data pipeline The automated sequence of steps that moves data from source systems to a destination, transforming it along the way.

Related guides

How to choose an ELT tool ELT tools differ less on connector count than on who operates them, how pricing scales, and whether transformation happens in the tool or downstream.