Data quality testing · Amazon Web Services (AWS Labs)

Deequ

Open-source Scala/Spark library from AWS for defining unit tests that measure data quality on large datasets.

Visit Deequ Alternatives

Deequ is a library built on Apache Spark that lets engineers write "unit tests for data": declarative constraints such as completeness, uniqueness and value-range checks that run as a Spark job over large datasets. It computes quality metrics incrementally so repeated runs on growing datasets stay efficient, and it includes an automatic constraint-suggestion feature that profiles a dataset and proposes a starting rule set. A built-in anomaly detection component tracks metrics over time to catch quality drift even without an explicit rule. PyDeequ exposes the same functionality through a Python/PySpark API for teams that prefer not to write Scala. As a library rather than a hosted service, Deequ has no UI, no built-in alerting and no managed offering; it is meant to be embedded directly into existing Spark pipelines on platforms like AWS Glue, EMR or Databricks.

At a glance

Vendor Amazon Web Services (AWS Labs)
Pricing model Open source + paid options
Free tier Yes
Deployment Self-hosted
Open source Yes (Apache-2.0)
Best for Spark-based data engineering teams on AWS wanting code-native quality checks at scale.

Pricing

Free, open-source library with no vendor pricing or hosted product.

Pricing has not been verified yet — see the vendor's site.

Features

  • Declarative data quality 'unit tests'
  • Automated constraint suggestion
  • Anomaly detection on quality metrics over time
  • Incremental metric computation on Spark
  • PyDeequ Python API
  • AWS Glue and EMR integration
  • Data profiling

Integrations

Profile last reviewed September 21, 2026

Head to head

Alternatives

Great Expectations Open-source Python framework for defining, running and documenting data quality tests ("expectations") in pipelines. Soda Data quality testing platform combining a check-as-code language (SodaCL) with a SaaS UI for monitoring and collaboration. DQOps Data quality and observability platform with 150+ built-in checks, configurable via UI or YAML, with daily automated monitoring.

Deequ in the index now

Terms to know

Data quality The degree to which data is accurate, complete, consistent and fit for the purpose it is being used for. Data pipeline The automated sequence of steps that moves data from source systems to a destination, transforming it along the way. Anomaly detection Identifying data points or patterns that deviate significantly from expected or normal behavior. Apache Spark An open-source distributed computing engine for processing large datasets in parallel across a cluster of machines. Data profiling Automatically scanning a dataset to summarize its structure, value distributions, and quality before it is trusted or modeled.

Related guides

How to choose a data quality testing tool Pick by how checks get authored — code library or check-as-code platform — and whether you need scheduling, dashboards and alerting, or just a test suite.