Idempotency

Idempotency is the property of an operation that produces the same end state no matter how many times it is executed with the same input. Setting a customer's status to "active" is idempotent: running it five times leaves the same result as running it once. Incrementing a counter by one is not, since each execution changes the outcome.

In data pipelines, idempotency is usually engineered rather than assumed. A load step becomes idempotent by overwriting a partition instead of appending to it, by using an upsert keyed on a unique ID instead of a plain insert, or by having a task check whether its output already exists before writing. This is distinct from exactly-once processing, which is a guarantee about message delivery in a streaming system; idempotency achieves a similar practical effect, no duplicate results, but at the level of how a task is written rather than how messages are transported.

Idempotency matters because failures and retries are routine in workflow orchestration: a task can crash halfway through, a network call can time out after the write actually succeeded, or an operator can rerun a backfill by hand. If tasks are not idempotent, any of these produce duplicate rows or double-counted metrics, and the resulting data deduplication cleanup afterward is often harder than making the task idempotent in the first place.