Record linkage

Record linkage is the process of identifying which records in two or more datasets, a school enrollment file and an employment file, say, belong to the same underlying person or entity, when there is no common ID field that lets the datasets be joined directly.

Deterministic linkage matches records that agree exactly on a set of fields, such as name, date of birth, and address; probabilistic linkage instead scores the likelihood that two records refer to the same entity based on partial agreement across several fields, weighting the more distinctive fields more heavily, which makes it more tolerant of typos, name changes, or missing values. It is closely related to data deduplication within a single dataset, but record linkage specifically connects separate datasets that were never designed to be joined together.

The technique lets researchers and agencies build a fuller picture of an individual's outcomes across systems, for example linking administrative data from a job-training program to later earnings records to evaluate its impact, without commissioning a new survey. Pitfalls include false matches, linking two different people, and missed matches, failing to link the same person across files, both of which bias downstream analysis. Because linkage often requires identifying information, it also raises privacy concerns that typically call for formal data-sharing agreements and steps such as anonymization before any linked file is analyzed or released.