Impact evaluation

Impact evaluation measures the causal effect of a program or policy by comparing what actually happened to participants against a credible estimate of what would have happened without the program, the counterfactual. This distinguishes it from simply tracking outcomes over time, which cannot separate the program's effect from other trends, such as a broader economic recovery, that would have moved the outcome anyway.

The strongest design is a randomized controlled trial, where random assignment to treatment and control groups guarantees the two groups are comparable except for the program itself. Where randomization is not feasible, evaluators build a counterfactual from observational data using quasi-experimental methods: difference-in-differences compares change over time between treated and untreated groups, regression discontinuity exploits a cutoff rule that assigns treatment, propensity score matching pairs treated units with similar untreated ones, and synthetic control builds a weighted composite of untreated units to approximate the treated case.

Impact evaluation sits within the broader practice of monitoring and evaluation but answers a narrower and harder question, not whether things improved, but whether the program caused the improvement. The most common pitfall is treating a before-and-after comparison as evidence of impact; without a counterfactual, an evaluator cannot rule out that the same change would have happened anyway, a problem sometimes compounded by selection bias in who chose to participate.