How to Tell If a Statistic Is Misleading

A statistic can be factually accurate and still mislead. The number itself is rarely the whole claim. What usually determines whether a statistic informs or distorts is what surrounds it: the comparison, the baseline, the population, the period and the definition of what is being measured.

This guide describes a practical way to read statistics critically, especially when they appear in news articles, product claims, social-media posts or AI-generated answers.

1. Ask what is actually being measured

Two statistics can look similar and mean very different things. "Customer growth" might refer to new accounts, active accounts, paying accounts, or accounts created but never used. "Traffic" might mean page views, unique visitors, sessions, or links clicked.

Before evaluating a number, identify the underlying measure and how it is defined. If the definition is missing, the statistic cannot be compared meaningfully with anything else.

2. Look for the comparison group

A percentage usually implies a comparison. "Sales rose 30%" is incomplete without knowing what they rose from, and over what period.

Useful questions include:

• Compared with what — a previous month, a previous year, a competitor, a forecast, or a baseline of zero?
• Is the baseline unusually low or unusually high?
• Is the comparison group representative, or was it chosen because it produces a flattering result?

Percentage growth off a small base is easy to produce. A company that sells 10 units one month and 20 the next has "doubled sales" but is still a small operation.

3. Check whether the number is absolute or relative

Relative numbers — percentages, ratios, multiples — can create a stronger impression than the underlying absolute numbers warrant.

A claim that "risk increased by 50%" can describe two very different situations:

• A risk rising from 1 in 10,000 to 1.5 in 10,000 (a 50% increase, but a small absolute change).
• A risk rising from 1 in 10 to 1.5 in 10 (also a 50% increase, but a large absolute change).

Both statements are accurate. Only one is usually worth acting on. When a relative change is presented, ask for the absolute numbers behind it.

Note what a scoring tool can and cannot do here. The Supience Information Check gives more weight to numbers that appear with units, comparisons or named entities, and less to bare numbers. But even a misleading statistic such as “risk increased by 50%” contains a number and a comparison phrase, so it will contribute to the specificity signal. The tool cannot tell whether the number is meaningful — that check is described in the steps above.

4. Check the sample size

Percentages based on small samples can be accurate and meaningless at the same time. "70% of users preferred the new design" is very different if the sample was 10 people versus 1,000.

When the sample size is not given, the statistic is harder to evaluate. A well-reported statistic usually identifies how many observations it is based on.

5. Check the time period

Many statistics depend on the period they cover. A claim about "record profits" may be based on a single quarter, a year, or a decade. A statement about "rising prices" may reflect a short-term spike or a long-term trend.

Ask:

• What period does the statistic cover?
• Is that period typical, or was it chosen because it produces a notable result?
• Would a longer or shorter period change the conclusion?

6. Watch for cherry-picked baselines

Two analyses of the same dataset can reach opposite conclusions depending on where the measurement starts. A chart that begins at a recession trough will look very different from one that begins at a boom peak.

When a striking growth or decline is presented, check whether the starting point is representative or unusually favourable to the conclusion.

7. Check the denominator

Percentages have an implicit denominator, and that denominator is sometimes more informative than the percentage itself.

"50% of respondents" could mean 5 out of 10 or 5,000 out of 10,000. "One in three users" could refer to a small group of early adopters or the entire user base.

Where possible, identify the population the statistic is drawn from. If the population is unclear or unusual, the percentage may not generalise.

8. Check whether averages hide variation

An average can be accurate while concealing a very uneven distribution. If half of a group earned nothing and half earned a high amount, the group's average may look healthy while most individuals are struggling.

When a mean is given, ask whether the median or a distribution would tell a different story. Where the distribution is skewed, an average may not describe any individual case.

9. Watch for correlation presented as causation

A correlation is a pattern. A causal claim is a stronger statement about why the pattern exists.

"Cities with more bookshops have higher literacy rates" may be true without supporting the conclusion that opening bookshops raises literacy. Both facts may share a common cause, such as higher average income or education levels.

Where a causal claim is made, look for evidence that supports the causal direction — for example, a controlled study, a natural experiment or a documented mechanism.

10. Check whether the measurement matches the conclusion

A statistic can measure one thing and be used to support a conclusion about another. This gap is one of the most common sources of misleading claims.

Examples:

• A study measuring short-term memory performance presented as evidence about "intelligence".
• A survey about reported satisfaction presented as evidence about actual product quality.
• A measure of clicks presented as evidence about reader engagement.
• A percentage of surveyed users generalised to the whole market.

The statistical work may be entirely sound. The problem is the inference.

11. Watch for changes in definitions

Definitions sometimes change between periods, between sources or between organisations. A rise in "unemployment" may reflect a real change or a change in how unemployment is counted.

When comparing statistics across time or sources, check whether the underlying definition is stable. If it is not, the comparison may not be valid.

12. Consider the missing context

A statistic presented without its context can create a misleading impression even when every part of it is accurate.

Questions worth asking:

• Is there a related statistic that would change how this one reads?
• Are there known limitations on the data source?
• Are there parallel trends that make this statistic less surprising than it appears?
• Are there base rates that make the finding more or less remarkable?

A short routine for reading a statistic

1. Identify what is being measured.
2. Identify the comparison group and the baseline.
3. Check whether the number is absolute or relative.
4. Find the sample size and time period.
5. Check the denominator and the population.
6. Ask whether the measurement supports the conclusion drawn from it.
7. Look for missing context that would change the reading.

Where Supience fits

The Supience Information Check can help identify whether a passage containing statistics includes verifiable signals — such as dates, sources, measurable units and comparison context — but it cannot tell whether the statistic itself is meaningful or honestly presented. That requires the reading process above.

For a broader process covering claims of any kind, see the Information Verification Checklist.

Open the Supience Information Check

Related guides

• Information Quality: What Can Go Wrong? — common failure patterns that make information look more reliable than it is.
• Information Verification Checklist — a reusable process for checking a claim.
• How to Verify a Source — checking whether a source actually supports a claim.