How to evaluate a public program

Most program evaluations fail at the same step: they measure whether an outcome improved after a program launched, and quietly assume the program caused it. program evaluation done well separates those two claims, because outcomes move for reasons that have nothing to do with the program — a good economy, a demographic shift, or simple regression to the mean. This is a methods guide, not a verdict on any specific policy or program; what a rigorous evaluation actually requires depends heavily on the program's design, data, and the questions the evaluation is meant to answer.

Start with a logic model, before you measure anything

A logic model lays out the causal chain a program is supposed to produce, from resources through to long-term impact:

Inputs → Activities → Outputs → Outcomes → Impact
(funding,   (what the      (units of      (short-term    (long-term,
 staff)      program does)  service        change in       population-
                             delivered)     participants)   level change)

Building this before you design an evaluation forces two useful decisions: which link in the chain you actually have the data and the time horizon to measure, and which assumptions the program's theory depends on that you should test rather than take on faith. A job-training program's logic model might assume that a completed course (an output) leads to employment (an outcome) — an assumption worth checking directly, not inferring from enrollment numbers alone.

The counterfactual problem

The central difficulty in any program evaluation is that you can never observe what would have happened to the same people without the program — the counterfactual. Every method below is a different strategy for approximating it credibly.

Before-and-after comparison, on its own, is not evaluation. Participants' outcomes might have improved regardless of the program — from a recovering economy, from aging out of a life stage the metric is sensitive to, from simply regressing toward an average after being selected for the program precisely because they looked unusually bad on that metric at the start. Without a credible comparison group, you cannot separate the program's effect from everything else that changed at the same time.

Building a credible comparison

Randomized controlled trials. A randomized controlled trial assigns eligible participants to the program or to a control group by chance, which — if done well, at sufficient scale — makes the two groups comparable on everything, observed and unobserved, except the program itself. This is the strongest design for causal inference and, where ethically and operationally feasible, the preferred approach. It is not always feasible: withholding a program from an eligible population raises real ethical and political questions, and many programs (a policy change, a new tax rate) cannot be randomized to individuals at all.

When randomization isn't possible, quasi-experimental methods approximate it:

• difference-in-differences compares the change over time in an outcome for a group affected by the program against the change over the same period for a similar, unaffected group — the logic being that both groups would have trended similarly absent the program, so the difference in their trends isolates the program's effect. It depends on that "parallel trends" assumption holding, which is worth testing against pre-program data, not asserting.
• regression discontinuity design exploits a sharp eligibility cutoff — an income threshold, a test score, an age limit — comparing people just above and just below the line, who are otherwise similar but differ in program access. Strong where a clean cutoff exists; useless where eligibility is fuzzy or gamed.
• propensity score matching pairs each participant with a non-participant who looked statistically similar on observable characteristics before the program started, building an artificial comparison group from people who didn't get the program. It only controls for what you measured and included in the match — unmeasured differences between participants and non-participants can still bias the result, which is its main limitation relative to randomization.
• Instrumental variables, a more advanced technique, use a factor that affects program participation but has no direct effect on the outcome except through that participation, to isolate the program's causal effect when neither randomization nor a clean cutoff nor good matching data is available. It is powerful and easy to misuse — a weak or invalid instrument produces a confidently wrong answer, so this method deserves a specialist's involvement.

Choosing a method in practice

Beyond causal impact: cost and reach

A program can produce a real, credible effect and still be a bad use of funds if the cost per unit of outcome is high relative to alternatives, or if it only reaches a small share of the eligible population. A complete evaluation reports:

• Cost-effectiveness — cost per unit of outcome achieved, ideally compared against at least one alternative use of the same funds.
• Reach and take-up — what share of the eligible population the program actually served, since even a strong per-participant effect matters less if few eligible people enroll.
• Fidelity of implementation — whether the program was delivered as designed; a null result from a program implemented poorly tells you about the implementation, not the underlying theory.

Where the data comes from

For programs involving federal funds, USAspending.gov publishes the underlying contract, grant, and spending records for free, useful for tracing where money actually went before evaluating what it achieved. For evaluating or benchmarking nonprofit-delivered programs specifically, Charity Navigator's public ratings on financial health and, for a growing set of charities, results and impact, offer an independent starting point rather than relying solely on a program's own self-reported outcomes. For engaging residents in the budget trade-offs behind public programs, tools such as Balancing Act let a government show constituents the real revenue and spending line items rather than an abstract summary.

Common mistakes

• Reporting a before-and-after change as the program's effect, with no comparison group.
• Using difference-in-differences without checking that the comparison group's pre-program trend actually looked similar to the treatment group's.
• Matching on convenient variables rather than the ones that actually predict both participation and the outcome, leaving propensity-score matching vulnerable to selection bias on anything unmeasured.
• Evaluating impact without evaluating cost, and reaching a "it works" conclusion that says nothing about whether it works better than the next-best use of the money.
• Treating a single evaluation, on one cohort in one place, as generalizable to every context the program might run in next.

For running an evaluation inside a digital product or service specifically — an A/B test rather than a population-level program — see the broader treatment in how to set up an experimentation program and how to read an A/B test result. For public finance and government spending tools more broadly, see every tool in this category.