Install
Orchestration & Operations terms
Schedulers, DataOps, MLOps and keeping pipelines alive.
Backfill Reprocessing historical data through a pipeline, typically after a bug fix, schema change or new metric definition. Batch processing Processing data in large, scheduled groups rather than continuously as each record arrives. Change data capture (CDC) A technique for detecting and streaming only the rows that changed in a source system, instead of reloading everything. Data contract A formal, versioned agreement on a dataset's schema and semantics between the team producing it and its consumers. Data fabric An architecture that uses metadata and automation to integrate and provide unified access to data across many sources. Data freshness How current a dataset is relative to real time, measured as the lag between an event happening and it being queryable. Data mesh A decentralized approach where domain teams own and publish their own data as products, instead of one central team. Data observability Continuous automated monitoring of pipelines and datasets to catch freshness, volume and quality problems early. Data pipeline The automated sequence of steps that moves data from source systems to a destination, transforming it along the way. Data product A dataset or data service built, owned and maintained with product-level rigor for a defined set of consumers. Data SLA (service-level agreement) A formal commitment about a dataset's freshness, availability or quality, with defined targets and consequences. DataOps Applying DevOps practices, such as automated testing, version control and continuous delivery, to data pipelines. Directed acyclic graph (DAG) A graph of one-way dependency edges between tasks, with no cycles, used to define the run order of a pipeline. Event-driven architecture (EDA) A system design where components communicate by producing and reacting to events, rather than calling each other directly. Idempotency The property where running an operation multiple times leaves the same result as running it once. Micro-batching Processing data in very small, frequent batches to approximate streaming without a dedicated stream processor. MLOps (machine learning operations) The set of practices for deploying, monitoring and maintaining machine learning models reliably in production, extending DevOps to ML systems. Workflow orchestration Scheduling, sequencing and monitoring interdependent data tasks so each step runs only after the ones it depends on succeed.