Experimentation validates causal relationships — but it's only as trustworthy as the statistics behind it. Significance, power, sample size, and confidence intervals are what separate a real result from noise.
▸ Try the interactive toolExperimentation is how data-informed PMs validate causal relationships between product changes and outcomes — the discipline that turns hypotheses into proven results. But experimentation is only as trustworthy as the statistics behind it, and a PM who runs experiments without understanding the fundamentals will draw confident conclusions from noise.
The essentials a PM must grasp: statistical significance (the probability the result isn't just chance — the p-value), statistical power (the test's ability to detect a real effect if one exists), sample size (how many users you need, calculated before running), and confidence intervals (the range the true effect likely falls in). The PM doesn't need to be a statistician — but they must understand these well enough to know when a result is real, when it's noise, and when an experiment was doomed before it started by being under-powered.
Significance (p-value) — chance the result is a fluke.
Power — ability to detect a real effect.
Sample size — calculated before running.
Confidence interval — the range the true effect likely sits in.
Without the statistical fundamentals, a PM can't tell a real effect from random noise — and most experimentation disasters are confident conclusions drawn from results that never reached significance.
| The shortcut | What it costs | What it gives you instead |
|---|---|---|
| Reading noise as signal | Declaring a winner before the result is statistically real. | Significance tells you whether the result is more than chance. |
| Under-powered experiments | Too small a sample to detect the effect; doomed from the start. | Power and sample-size calculation ensure the test can find a real effect. |
| Ignoring the interval | Reporting a point estimate as if it were exact. | Confidence intervals show the real range of uncertainty. |
| No pre-calculated sample size | Stopping whenever the numbers look good (peeking). | A pre-set sample size prevents peeking-driven false positives. |
Determine how many users you need to detect a meaningful effect with adequate power — before starting. An experiment too small to detect its effect is wasted before it begins.
Decide what p-value counts as significant before seeing data. Setting it after the fact invites rationalising noise into a 'result.'
Power is the test's ability to detect a real effect. An under-powered test that shows 'no effect' may simply have been too weak to see one — a false negative, not a true null.
Resist stopping early when results look good. Every peek-and-consider-stopping is effectively a new test, inflating false positives. Run to the pre-set sample or duration.
Communicate the range the true effect likely falls in, not a single number presented as precise. The interval is the honest expression of what the experiment actually established.
A team ran an experiment, saw the variant ahead after a few days, and declared victory — shipping the change. The result evaporated in production. Two statistical failures had combined: the experiment was under-powered (too small a sample to reliably detect the real effect), and they'd stopped early on a favourable peek rather than running to a pre-calculated sample.
Doing it properly meant calculating the required sample size up front, setting the significance threshold in advance, and running to that sample without peeking. The honest result, with its confidence interval, was far less exciting than the early bump — and far more trustworthy. The early 'win' had been noise that a properly-powered, peek-free experiment would never have mistaken for a result.
The deliverable is an experiment with pre-calculated sample size, a pre-set significance threshold, adequate power, and results reported as confidence intervals.
| Fundamental | Guards against | Discipline |
|---|---|---|
| Significance | Reading noise as signal | Pre-set p-value threshold |
| Power | False negatives | Adequate sample for the effect |
| Sample size | Under-powered tests | Calculate before running |
| Confidence interval | False precision | Report the range, not a point |
The statistics aren't academic decoration — they're what separates a real result from random noise. A PM who can't tell the difference will ship changes based on flukes and kill good ideas that an under-powered test failed to detect.
The two failures that cause the most damage are under-powering and peeking. Under-powering means the experiment is too small to reliably detect the effect you care about — so a 'no significant difference' result tells you nothing, yet teams read it as proof the change didn't work. Peeking means looking at results repeatedly and stopping when they look good — which mathematically inflates the false-positive rate, because each look is effectively another roll of the dice. Both are prevented by the same discipline: calculate the required sample size in advance and run to it without stopping early. The PM doesn't need to derive the statistics, but must understand them well enough to demand pre-calculated samples, resist early stopping, and read a confidence interval — because the alternative is confident decisions built on noise.
Each peek-and-stop inflates false positives. Run to the pre-set sample.
Too small to detect the effect means a meaningless result. Calculate sample size first.
Report the confidence interval — the honest range of the effect.
Decide significance in advance, or you'll rationalise noise into a result.
The checklist (Tool 18) operationalises these fundamentals into a routine.
Many of the six mistakes (Tool 19) are statistical failures this tool prevents.
These foundations make A/B tests (Module 3, Tool 12) trustworthy.
When fixed-horizon testing doesn't fit, bandits (Tool 20) take a different statistical approach.
A team runs an experiment, sees the variant ahead on day 3, and ships it. Name the two statistical failures most likely at play.
Then describe the discipline that would have prevented each — what should they have calculated and decided before starting?
If you identified under-powering and early stopping (peeking), and prescribed a pre-calculated sample run without peeking, you've understood the two failures behind most false experimental wins.