Six experimentation failures that are common, expensive, and entirely preventable. Where the design checklist is the proactive discipline, this is the diagnostic awareness of how experiments go wrong.
▸ Try the interactive toolThis tool catalogues six experimentation failures that are common, expensive, and entirely preventable. Where the design checklist (Tool 18) is the proactive discipline of setting experiments up well, this catalogue is the diagnostic awareness of how experiments go wrong — the failure modes to recognise in your own and others' work.
The six are deadly because each produces a confident, wrong conclusion that looks legitimate from the inside. Peeking and novelty effects manufacture false wins; underpowered tests and sample ratio mismatch produce meaningless results that masquerade as findings; multiple comparisons guarantee a spurious winner; ignoring guardrails lets you 'win' while doing damage. Recognising them is most of the defence — and the running of an experiment, when the temptation to peek and stop early is strongest, is exactly when this awareness matters most.
Each produces a confident, wrong conclusion that looks legitimate from the inside. The temptation peaks while the experiment runs — especially the urge to peek and stop early. Awareness is most of the defence.
These mistakes are dangerous precisely because the flawed experiment looks successful to the people running it — the false win feels like a real one.
| The shortcut | What it costs | What it gives you instead |
|---|---|---|
| Peeking & early stopping | Stopping on a good peek inflates false positives invisibly. | A pre-set sample and no-peeking rule prevent it. |
| Novelty effect | A week-one win reverses once the newness fades. | A minimum duration lets novelty wash out. |
| Multiple comparisons | Test enough things and one 'wins' by pure chance. | One primary metric and correction for multiple comparisons. |
| Ignoring guardrails | Winning the primary while harming retention or revenue. | Guardrail metrics catch the hidden damage. |
Each is common, expensive, and preventable — and each produces a confident conclusion that looks right from the inside, which is what makes them dangerous.
| Mistake | What goes wrong | How to prevent it |
|---|---|---|
| Peeking & early stopping | Stopping when results look good inflates false positives — each peek is a new test | Pre-set sample size and duration; never stop early on a favourable peek |
| Novelty effect | Users engage more with anything new; a week-one win can reverse by week three | Run a minimum duration (e.g. two weeks) so the novelty fades |
| Underpowered tests | Too small a sample to detect the effect; 'no result' is meaningless | Calculate required sample size before running (Tool 17) |
| Multiple comparisons | Testing many metrics/variants means something 'wins' by chance | One primary metric; correct for multiple comparisons |
| Ignoring guardrails | Winning the primary metric while quietly harming another | Track guardrail metrics alongside the primary |
| Sample ratio mismatch | Traffic isn't splitting as intended — the experiment is broken | Check the actual split; investigate any imbalance before trusting results |
An experiment had been live only a few days when the early activation numbers looked good for the variant, and the pull to call it and ship was immediate. That pull — peeking and early stopping — is the single most common cause of false positives, because every time you look and consider stopping, you're effectively running a new test and inflating the odds of a fluke.
Two of the six mistakes were in play at once. Even if the early numbers held, they might be a novelty effect — users engage more with anything new for a week or two, and a variant winning early can normalise or reverse by week three. The discipline that defeats both is the same: a pre-set minimum duration and sample, run without peeking, so the result reflects sustained value rather than early shine or random noise.
The deliverable is using this catalogue diagnostically — before trusting any experiment, checking it against the six failure modes.
| If you're tempted to… | Recognise it as… |
|---|---|
| Stop early on a good result | Peeking — run to the planned sample |
| Trust a week-one win | Novelty — wait out the minimum duration |
| Celebrate a small-sample result | Underpowered — it may be noise |
| Pick the metric that won | Multiple comparisons — one primary only |
These six mistakes share one trait that makes them lethal: each produces a result that looks like success from the inside. The flawed experiment doesn't feel flawed — the false win feels exactly like a real one, which is why awareness is the primary defence.
The most important practical insight is when the danger peaks: while the experiment is running and early numbers look favourable. That's the moment the pull to peek, stop early, and ship is strongest — and it's the moment the discipline matters most. Peeking inflates false positives because each look is effectively another test; novelty effects make early wins evaporate as the newness fades; both are defeated by a pre-set minimum duration and sample run without interruption. The other mistakes — underpowering, multiple comparisons, ignored guardrails, sample ratio mismatch — round out the failure modes that turn experiments into confident, wrong conclusions. This catalogue is the diagnostic counterpart to the design checklist: the checklist prevents these in setup, this recognises them when the temptation arrives.
Knowing the mistakes doesn't stop them mid-experiment. Pre-commit to the disciplines that prevent each.
The strongest temptation and the most common false-positive cause. Run to the planned sample.
Novelty fades. Wait out the minimum duration before believing a result.
A 'win' that harms another metric isn't a win. Watch the guardrails.
The checklist (Tool 18) prevents these proactively; this recognises them diagnostically.
Most of the six are statistical failures the fundamentals (Tool 17) explain.
Like Tool 27, this is a catalogue of failures to recognise and avoid.
Avoiding these keeps A/B tests (Module 3, Tool 12) trustworthy.
Imagine an experiment three days in, with the variant looking good. Write down the argument for stopping and shipping now — then identify which two of the six mistakes that argument represents.
Describe the single discipline that defeats both.
If you spotted peeking and the novelty effect — and prescribed a pre-set minimum duration run without peeking — you've learned to recognise the false wins exactly when they're most tempting.