How to read an A/B test result

Most experimentation dashboards reduce a test to a single word: significant, or not. That word hides most of what you need to know before you act on it — whether the test had a real chance of detecting the effect you cared about, whether the win is big enough to matter, and whether the result would replicate. This guide is about reading past the green checkmark.

Start with what "significant" actually means

statistical significance answers one narrow question: if there were truly no difference between variants, how likely is it we'd see a gap this large by chance alone? That likelihood is the p-value. A p-value under the usual 0.05 threshold means "unlikely to be chance" — it does not mean "large," "important," or "certain." A test can be significant and commercially trivial, and a test can show a meaningful-looking lift that isn't significant because the sample was too small to tell noise from signal.

The confidence interval around the effect is more useful than the p-value alone, because it shows you the range of true effects the data is consistent with. A result reported as "+2% conversion, 95% CI [+0.1%, +3.9%]" is a very different decision than "+2%, 95% CI [-1.5%, +5.5%]" — the second interval includes zero and even a loss, even though the point estimate looks identical.

Check whether the test could have won at all

statistical power is the probability a test detects a real effect of a given size, if one exists. Low power is the quiet failure mode of experimentation: a true, valuable improvement gets reported as "no significant difference" not because it didn't work, but because the test never had enough traffic or runtime to see it. Before trusting a null result, ask what minimum detectable effect the test was powered for. If the MDE was 8% and the real effect was 3%, "no significant difference" tells you almost nothing about whether the change helped.

A rough intuition, not a substitute for a calculator: required sample size scales roughly with 1 / effect_size². Halving the effect you need to detect roughly quadruples the sample size required — which is why small, incremental changes on a low-traffic page can take months to test properly, and why teams under time pressure quietly accept underpowered tests without saying so.

Look for sample ratio mismatch before you look at the metric

A sample ratio mismatch (SRM) means the traffic split that arrived is not close to the split you configured — 55/45 when you set 50/50, for instance. SRM is a bug alert, not a statistics problem: it usually means the randomization broke, a bot filter hit one variant differently, or a redirect added latency to one arm. Any result from a test with SRM should be treated as untrustworthy until the cause is found, no matter how clean the headline metric looks. Most experimentation platforms flag this automatically; check for it first, every time, before reading anything else.

Understand why peeking is dangerous — and when it isn't

Checking a fixed-horizon test's p-value every day and stopping the moment it crosses 0.05 inflates the true false-positive rate well above the nominal 5%, because you are effectively running many tests (one per look) and taking the luckiest one. This is the single most common way legitimate-looking "wins" turn out not to replicate.

Two things fix it. sequential testing methods (mixture sequential probability ratio tests, always-valid confidence intervals) are built to be checked continuously without inflating false positives — if your platform explicitly supports one, continuous monitoring is safe. Otherwise, commit to a sample size or a calendar date in advance and do not act on the result before then, even if it looks decisive on day three.

CUPED (controlled-experiment using pre-experiment data) is a different, complementary technique: it uses a user's pre-experiment behavior to reduce the variance of the metric, which shrinks the sample size — or the time — a test needs to reach a given power, without changing what significance means.

Read the result in context, not in isolation

• Segment cuts after the fact are exploration, not proof. If the overall result is flat but "new mobile users in one region" looks great, that is a hypothesis for the next test, not evidence on its own — you searched many subgroups and found the one that moved.
• Novelty and change-aversion fade. A redesign can spike or dip in week one purely because it's different, then settle. Where you can afford it, let tests run long enough to see the effect stabilize, particularly for changes users will see repeatedly.
• Check the guardrails, not just the goal. A checkout flow that lifts conversion 3% while page load time doubles has a real cost the primary metric won't show.
• A test that ran on 2% of your normal traffic during a holiday week is not the same population you'll ship to. Match the test window to how you'll actually use the result.

Common mistakes

• Treating "not significant" as "proven no effect," rather than "insufficient evidence given this sample size."
• Stopping a fixed-horizon test the moment it looks good, then reporting it as if it had run the planned duration.
• Reporting relative lift ("+20%") without the base rate — a 20% lift from 0.5% to 0.6% conversion is a different decision than 20% from 10% to 12%.
• Ignoring sample ratio mismatch because the headline metric happened to be favorable.
• Comparing results across tests run in different seasons, channels or traffic mixes as if the audiences were the same.

Setting up the process that produces trustworthy results in the first place is covered in how to set up an experimentation program. For the platforms that implement these statistical methods, see experimentation and feature-flag tools.