Install
Experimentation & Causal Inference terms
A/B tests, randomized trials and the methods that separate cause from correlation.
A/A test An experiment that splits users into two identical groups with no real difference, used to check that the testing system itself is unbiased. A/B testing A controlled experiment that compares two variants to measure which one produces a better outcome. Average treatment effect (ATE) The mean difference in outcome between treated and untreated units across a full population, the standard headline result of an experiment. Causal inference The set of methods used to determine whether one variable actually causes a change in another, not merely correlates with it. Clinical trial analytics Analytics applied to designing, running, and analyzing clinical trials, from site selection through final results. Conversion lift study A randomized test comparing a holdout who saw no ads against an exposed group, to measure advertising's true incremental effect on conversions. Counterfactual What would have happened to a unit or group without a treatment, the unobservable baseline every causal claim is measured against. CUPED (controlled-experiment using pre-experiment data) A variance-reduction technique that uses each user's pre-experiment behavior to sharpen an experiment's estimate without adding more traffic. Difference-in-differences (DiD) A causal method that compares the before-and-after change in a treated group against the before-and-after change in an untreated group. Effect size A standardized measure of the magnitude of a difference or relationship, independent of sample size. False discovery rate (FDR) The expected share of "significant" results that are actually false positives, when many statistical tests are run and checked at once. Feature flag A configuration switch that turns a piece of functionality on or off, or for specific users, without deploying new code. Geo experiment An experiment that randomizes treatment by geographic region rather than by individual user, often to measure marketing's incremental impact. Guardrail metric A secondary metric watched during an experiment to catch harm the primary metric would not show, such as load time or churn. Heterogeneous treatment effects (HTE) Differences in a treatment's true effect across subgroups, showing an intervention can help some segments and harm or not affect others. Holdout group A segment of users deliberately kept out of a treatment, campaign or feature, so its true incremental impact can be measured against them. Impact evaluation Measuring a program's causal effect on outcomes by comparing what happened to a credible counterfactual. Incrementality testing An experiment measuring the true causal lift of a marketing activity by comparing it against a held-out control. Instrumental variables (IV) A causal-inference technique that uses a variable affecting the outcome only through treatment, to work around unmeasured confounding. Marketing mix modeling A statistical technique estimating how much each marketing channel and external factor contributes to results. Minimum detectable effect (MDE) The smallest true effect an experiment is designed to reliably distinguish from no effect, given its sample size and power. Multi-armed bandit (MAB) An adaptive experiment that shifts traffic toward better-performing variants in real time, instead of splitting it evenly until a fixed end date. Multiple comparisons problem The increased risk of a false positive that comes from running many statistical tests and highlighting any one significant result. Multivariate testing (MVT) An experiment that varies several page or product elements at once, to measure each element's individual effect and how elements interact. Novelty effect A temporary spike in engagement with a new feature or design that fades once users stop reacting to its newness and it becomes routine. Overall evaluation criterion (OEC) The single metric or composite formula a team agrees in advance will determine whether an experiment's treatment is judged a success. Preregistration Publicly documenting a study's hypotheses, methods and analysis plan before collecting or examining the data. Program evaluation Systematic assessment of whether a public or nonprofit program achieves its intended outcomes, and why. Propensity score matching (PSM) A method that pairs treated and untreated units with similar odds of having received treatment, to approximate a randomized comparison. Randomized controlled trial (RCT) An experiment that randomly assigns subjects to treatment and control groups to isolate a causal effect. Regression discontinuity design (RDD) A causal method that compares outcomes just above and below a cutoff, such as a score threshold, that determines who receives a treatment. Sample ratio mismatch (SRM) A significant gap between an experiment's actual traffic split and its intended split, signaling a bug rather than a real treatment effect. Sample size The number of observations in a study or test, which determines how precisely and reliably an effect can be measured. Sequential testing An analysis method that lets experiments be checked continuously without inflating the false-positive rate from repeated peeking. Split URL testing An A/B test that sends visitors to entirely separate page URLs for each variant, instead of changing content dynamically on one shared URL. Statistical power The probability that a hypothesis test correctly detects a real effect, given that one truly exists. Statistical significance A threshold judgment that an observed effect is unlikely to be due to chance alone, given the data and test used. Switchback experiment An experiment design that alternates a whole system between treatment and control over time, for cases where individual users can't be split. Synthetic control A method that builds a weighted composite of untreated units as the counterfactual for a single treated unit, such as one market or region. T-test A statistical test that compares the means of one or two groups to judge whether an observed difference is likely real. Uplift modeling Predictive modeling that estimates how much a treatment changes an individual's outcome, to target only the people it will actually influence.