Data deduplication

Data deduplication is the process of finding and removing records that represent the same underlying thing more than once, the same customer entered twice with slightly different spellings, the same event ingested twice after a retry, the same transaction logged by two systems. It ranges from trivial, dropping byte-for-byte identical rows, to genuinely hard, recognizing that "Jon Smith" and "Jonathan Smith, Jr." at similar addresses are probably the same person.

Exact deduplication typically works by comparing a unique key, an event ID, a transaction ID, and discarding repeats; this is the usual fix for duplicates introduced by retries in a pipeline that isn't fully idempotent or that relies on an at-least-once delivery guarantee rather than exactly-once processing. Fuzzy deduplication, matching records that refer to the same entity despite differing text, is a harder, closely related problem usually called record linkage or entity resolution, and typically requires similarity scoring rather than exact key matching.

Deduplication matters because duplicate records silently inflate counts, revenue, users, events, in ways that are easy to miss until a number looks implausibly high. It is a core part of data quality work, and pipelines that ingest from systems with at-least-once delivery or from multiple overlapping sources need a deliberate deduplication strategy rather than assuming duplicates won't occur. A common pitfall is deduplicating too aggressively and discarding legitimately repeated events, like a customer placing two genuinely separate but similar orders in the same minute.