The six most dangerous measurement mistakes — the recurring ways teams misuse data that look like good measurement from the inside but quietly mislead. The negative counterpart to everything else in the module.
▸ Try the interactive toolMetrics anti-patterns are the six most dangerous measurement mistakes — the recurring ways teams misuse data that look like good measurement from the inside but quietly mislead. This tool is a catalogue of failures to recognise and avoid, the negative counterpart to the constructive measurement tools throughout the module.
The six are dangerous precisely because a sophisticated team commits them while feeling rigorous. Vanity metrics celebrate numbers that only go up; averages hide bimodal distributions; correlation gets read as causation; cherry-picking reports the flattering cut; gaming (Goodhart's law) optimises a metric destructively once it's a target; and ignoring confidence reads noise as signal. The defence is a deliberate, honest self-audit — because these mistakes don't feel like mistakes; the dashboard looks healthy right up until the misleading number drives a bad decision.
Each looks like good measurement from the inside — even a sophisticated team commits them while feeling rigorous. The dashboard looks healthy until a misleading number drives a bad decision. The defence is an honest self-audit.
A team can have a strong data practice and still be fooled by its own metrics, because the most dangerous measurement mistakes are the ones that look like good measurement.
| The shortcut | What it costs | What it gives you instead |
|---|---|---|
| Vanity metrics | Numbers that only go up feel like success and mean nothing. | Metrics that can go down (active users) actually carry information. |
| Averages hiding distributions | One average smears two different populations together. | Showing the distribution reveals the real picture. |
| Correlation as causation | ‘X correlates with retention’ read as ‘X causes it’. | Causal testing separates the marker from the cause. |
| Gaming (Goodhart's law) | A metric made a target is optimised destructively. | Guardrails catch the metric rising as value falls. |
Each looks like good measurement from the inside — which is exactly what makes a deliberate, honest self-audit the only reliable defence.
| Anti-pattern | What it looks like | The fix |
|---|---|---|
| Vanity metrics | Celebrating numbers that only ever go up (total registered users) | Use metrics that can go down — active users, connected accounts |
| Averages hiding distributions | Reporting one average that smears two populations together | Show the distribution, or median + percentiles |
| Correlation as causation | ‘Users who do X retain better, so X causes retention’ | Test causally (experiment); X may just mark already-engaged users |
| Cherry-picking | Reporting the metric or timeframe that flatters | Pre-commit to the metric; report it whichever way it lands |
| Gaming / Goodhart's law | A metric becomes a target and is optimised destructively | Use guardrails; watch for the metric rising as value falls |
| Ignoring confidence / noise | Reading random fluctuation as a real trend | Account for significance and intervals before acting (Tool 17) |
A team with a genuinely strong data practice ran an honest self-audit against the anti-patterns — and found it had fallen into several. A 'total registered users' slide in the board deck always went up and told them nothing about whether people were active: a classic vanity metric. An 'average time to first action' was reported as one number, but was actually bimodal — fast for activated users, never for the rest — the average smearing two populations into a meaningless middle.
They also caught themselves treating a correlation ('users who complete a certain step retain better') as causation, when the step might simply mark already-engaged users. The fixes were direct: replace the vanity metric with active users (which can go down and therefore means something), show the distribution instead of the average, and test the correlation causally. The point was that a sophisticated team had committed these while feeling rigorous — only a deliberate audit surfaced them.
The deliverable is a periodic honest audit of your own metrics practice against the six anti-patterns.
| Audit question | Catches |
|---|---|
| Any metric that only goes up? | Vanity metrics |
| Any average hiding two populations? | Distribution-smearing |
| Any 'X causes Y' from correlation? | Correlation-as-causation |
| Reporting the flattering cut? | Cherry-picking |
| A target being gamed? | Goodhart's law |
These six anti-patterns are dangerous for the same reason discovery's biases are: they operate while the practitioner feels rigorous. The dashboard looks healthy, the analysis feels sound, and the misleading number drives a bad decision anyway.
The defining trait — that they look like good measurement from the inside — is why awareness alone isn't enough and a deliberate self-audit is required. A team won't stumble onto its own vanity metrics by feeling uneasy; the vanity metric feels great, because it always goes up. It won't notice its average is hiding a bimodal distribution unless it deliberately looks at the distribution. It won't catch correlation-as-causation without explicitly asking whether it tested the causal link. The audit format — walking through each anti-pattern and honestly asking 'are we doing this?' — is the only way to surface mistakes that, by their nature, don't announce themselves. This is the measurement counterpart to discovery's research biases and process anti-patterns: a structured catalogue whose value is recognition, applied honestly to one's own work first.
Sophisticated teams commit these while feeling rigorous. Audit yourself honestly.
Vanity metrics feel like success. Use metrics that can go down.
One number smears two populations. Show the distribution.
A correlate may just mark an already-engaged user. Test causally.
These are the failures of the constructive metrics tools (Tools 01–26).
Goodhart's law here is the same mistake as treating velocity as a target (Module 4, Tool 03).
Ignoring noise is the statistical failure of Tool 17.
Like Module 3's biases and anti-patterns, this is a catalogue to recognise and avoid.
Take a dashboard or report you know. Walk it against the six: any vanity metric? any average hiding a distribution? any correlation treated as cause? any cherry-picked cut? any gamed target? any noise read as trend?
Be honest — find at least one. Which one would have felt most like 'good measurement' until you looked?
If the anti-pattern you caught felt rigorous until you audited for it, you've understood why these are so dangerous — and why honest self-audit, not good intentions, is the only defence.