Statistical significance
A label for a test result whose p-value falls below a threshold set in advance, meaning the data would be unusual if there were no real difference.
By Azande PorterUpdated 3 min read
What is statistical significance?
A result is statistically significant when its p-value falls below a threshold chosen before the test, most often 0.05, meaning data this extreme would be unusual if there were truly no difference. It is a statement about chance, not about size or importance.
It shows up most often in A/B testing: two versions, randomly split audiences, and the question of whether the gap between them is bigger than luck would usually produce.
What does a p-value actually mean?
A p-value is the probability of seeing a difference at least as large as the one you observed, assuming there is no real difference between the versions. For example, a p-value of 0.03 means that if both signup pages truly performed the same, a gap this big or bigger would appear by chance about 3% of the time.
The American Statistical Association's statement on statistical significance and p-values sets out six principles for using and reading p-values. Three matter most for marketers:
- A p-value does not measure the probability that your hypothesis is true.
- Significance does not measure the size of an effect or the importance of a result. A tiny, useless lift can be significant with enough traffic.
- Decisions should not rest only on whether a p-value crosses a threshold. A result at 0.049 and one at 0.051 carry nearly the same evidence.
How big a sample do you need?
It depends on four choices: your current rate, the smallest difference worth detecting, the significance level, and the power you want, which is the chance of detecting a real difference of that size. 80% power is a common choice.
For example, to detect a signup conversion rate moving from 4% to 5% at a 0.05 level with 80% power, the standard two-proportion calculation needs roughly 6,700 visitors per version. To detect a jump from 4% to 8%, it needs about 550 per version. The smaller the difference you care about, the faster the required sample grows, which is why small audiences can only reliably detect large effects.
Why does peeking at results cause false positives?
If you check a test every day and stop the first time p dips below 0.05, you give chance many attempts to cross the line. The real false-positive rate ends up far higher than the 5% you think you accepted. Johari, Pekelis, and Walsh showed that standard inferences are "wholly unreliable" when people choose sample sizes by continuously monitoring their tests, and proposed sequential methods built for repeated checks.
The simple fix is to set the sample size before you start and read the result once. If you must monitor, use a tool that applies a sequential method rather than a standard p-value.
Can social media results be statistically significant?
Rarely in the formal sense. Significance tests assume random assignment, and social posts do not have it: each post goes out at a different time, to an audience the platform's ranking chooses, with one post per version.
Email is closer. A subject line split sends each version to a random part of the list. Judge it on clicks, because an open rate can include opens no person made.
For posts, compare each one with your baseline and look for repetition. A change that beats your usual four times out of five is a pattern worth acting on, labeled as an observation rather than a significant result. The guide to reviewing social post performance shows that method step by step.
How Distinctful helps
Distinctful does not run significance tests on posts. It compares each post with your account's own usual results, which is the honest comparison for content that cannot be randomly split, and it keeps every post's result so you can see whether a change holds across several posts before you treat it as real. See social media analytics.
Related terms
Go deeper
Guide
How to Review Social Post Performance With Claude
Analyze social media post performance with Claude by comparing similar posts, checking missing metrics, and turning one clear observation into a content test.
Guide
How to Track Newsletter Signups From LinkedIn
Track newsletter signups from LinkedIn with separate tagged links for post, Contact Info, and Featured placements, then confirm your provider stores the values at signup.
Judge posts against your own usual
Compare each post with your baseline and repeat a change before calling it a pattern.