There is a number that ends most e-commerce AB tests. It sits in the testing platform dashboard, ticks upward over days or weeks, and the moment it crosses a threshold, the conversation stops. The variant wins. The change gets released. The result gets reported upward as evidence that the programme is working.

That number is statistical significance, and it is the most confidently misread metric in digital experimentation. Not because it is wrong, but because what it actually says is much narrower than what most teams hear. Significance tells you the result is unlikely to be random. 

Statistical significance does not tell you the result is real, stable, commercially meaningful, or safe to act on. 

And for e-commerce teams making release decisions on the back of it, the gap between those two things is where real value gets lost.

What Statistical Significance Actually Tells You

At 95% confidence, statistical significance means one specific thing: 

If there were genuinely no difference between your control and your variant, you would see a result this large or larger by chance 5% of the time, or 1 in 20.

That is all it means. 

It is a statement about the probability of your data given the assumption of no effect, not a statement about whether your variant actually works.

The 5% threshold is not a safety net. It is an accepted error rate. Run twenty tests on your site and you would expect one winner by pure chance, even if none of your variants made any real difference to customer behaviour. Run a programme of tests over two years and the false positives accumulate. Each one looks like a win. Each one justifies the next round of investment. And none of them move the commercial needle, because they were never real in the first place.

This is not an argument against significance testing. It is an argument for understanding what it does and doesn’t protect you from, and for treating a significant result as the beginning of the right questions, not the end of them.

The High-Value Transaction Problem: When a Single Order Falsifies Your Results

Average order value and revenue per session are among the most commercially important metrics in e-commerce experimentation. They are also among the most vulnerable to distortion, for a reason that has nothing to do with your test design.

A single anomalous transaction, one £3,000 order on a site with an average AOV of £300, does not just skew the average. It can manufacture statistical significance where none exists. If that transaction happens to fall in the variant group, it inflates the variant’s revenue per session figure, widens the gap between control and variant, and can push a borderline result across the 95% threshold in a single session. The platform reports a winner. The team releases the change. The next month’s revenue looks nothing like the test result, because the uplift was never there, it was one customer buying a large order who happened to land in the variant.

This is not an edge case. For any e-commerce site with a non-uniform order value distribution, which is most of them, high-value transactions are a persistent source of noise in revenue-based metrics. 

The fix is to exclude large orders from your analysis and to treat revenue-based significance with more scepticism than conversion rate significance, cross-check against order count uplift as well as revenue uplift, and look at what happens to the result when you remove the top one or two percent of transactions by value.

The Variation Problem: Impact Is Never Flat During a Test

A test that delivers a 2% conversion rate uplift in aggregate is not delivering a 2% uplift every day. The daily impact of any variant fluctuates, driven by day-of-week effects, traffic mix changes, promotional activity, seasonal patterns, and run-of-the-mill random chance. On some days the variant will be up 8%. On others it will be down 6%. The aggregate number is the average of a volatile series, not a stable signal.

The problem arises when significance is manufactured by a small number of exceptional days rather than a consistent pattern of outperformance. A test that ran for three weeks might show a significant aggregate uplift driven almost entirely by four days where the variant dramatically outperformed, perhaps coinciding with a promotional email that drove higher-intent traffic into the variant cohort, or a competitor going out of stock and sending unusual demand to your site. Strip those four days out and the result disappears.

This matters because a genuine behavioural improvement, a variant that actually makes the experience better for customers, should produce a reasonably consistent pattern of outperformance over time. Not perfectly flat, but directionally stable. A significant result built on a handful of exceptional days is a fragile one. It may replicate in production, but the conditions that produced it are unlikely to persist.

The Peeking Problem: Why Calling Tests Early Is One of the Most Reliable Ways to Ship False Positives

Most teams check their testing platform more often than they should. 

The dashboard is there, the data is live, and the temptation to look, and to act on what you see, is hard to resist (and in some cases practically impossible). The problem is that significance fluctuates during a test in ways that make early readings unreliable.

A test might cross 95% confidence on day five, drop back to 88% on day nine, recover to 96% on day fourteen, and settle at 91% on day twenty-one. If the team calls the test on day five, they ship a result that the full data would not have supported. This is one of the most common sources of false positives in e-commerce experimentation, not because teams are being careless, but because the testing platform makes early significance visible and the commercial pressure to release winners is real.

The discipline required is straightforward in principle and difficult in practice: decide the test duration before the test runs, based on the traffic volume needed to reach adequate power, and do not call the result until that duration is complete. A significant result on day five of a planned twenty-one day test is not a winner. It is an early reading on a noisy signal.

The Novelty Effect: Why Early Conversion Lifts Often Don’t Survive

A variant that is new and visually different gets disproportionate attention from returning customers simply because it is unfamiliar. Returning visitors, who typically convert at higher rates than new visitors and represent a commercially significant share of traffic on most e-commerce sites, engage more actively with something they haven’t seen before. Click rates go up. Conversion rates follow. The test looks like a strong early winner.

By week three, the novelty has worn off. Returning customers have seen the variant and their behaviour has normalised. The uplift decays, sometimes partially and sometimes entirely. A test called at ten days on a site with significant returning traffic may be measuring curiosity rather than genuine preference, and the change that looked like a 4% conversion lift in the first week of testing delivers 1% in production, if that.

The practical guard against novelty effect is time: tests should run long enough to include multiple exposures for returning customers and to allow early novelty inflation to settle. As a rule, if your returning customer rate is above 30%, you should be sceptical of any result called in the first ten days regardless of what the significance number says.

Segment Contamination: Significant Overall, Damaging Underneath

A test that shows a 3% overall conversion uplift may be masking sharply different effects across customer segments. New visitors might be responding positively while returning customers are not. Mobile users might be lifting while desktop users are declining. High-intent organic search traffic might be responding well while paid social traffic shows no effect or a negative one.

Averaging across segments produces a significant aggregate result that is simultaneously good news and bad news, and releasing the variant will deliver the aggregate average to all users, including the segments it actively harms. For an e-commerce site where returning customers represent a disproportionate share of revenue, a variant that lifts new visitor conversion while suppressing returning customer conversion can look like a winner in the test and quietly erode the most commercially valuable part of your customer base in production.

Segment-level analysis is not optional on significant results, it is the check that tells you whether the aggregate number is safe to act on. A result that holds across your primary segments has a fundamentally different character to one that holds only in aggregate.

Statistical Power: The Problem Significance Can’t See

Statistical significance has a sister metric that many e-commerce teams never think about: statistical power. Where significance protects you from seeing effects that aren’t there, power protects you from missing effects that are. They are different failure modes, and a test can fail both simultaneously.

The convention in experimentation is to run tests at 80% power, meaning that if a real impact exists, your test has an 80% chance of detecting it. The uncomfortable implication is that even a correctly powered test will miss a real effect one time in five. For teams running tests on modest traffic volumes, power is almost always the binding constraint: the minimum sample size needed to detect a meaningful lift reliably is often larger than the traffic available in a sensible test window.

The connection to significance is this: an underpowered test that happens to reach significance is more likely to be a false positive than the 5% error rate implies, and more likely to overestimate the size of the effect. A result that looks like a 3% lift from an underpowered test may be real at 1%, or may not be real at all. Statistical power deserves its own rigorous treatment, and we’ll address it fully in a dedicated piece. The point here is simpler: significance without power is an incomplete picture, and most teams are only looking at half of it.

Consistency: The Sense Check That Statistics Can’t Replace

Outside the formal statistical framework, there is a quality check that experienced experimenters apply to every significant result before they release it: consistency.

Not a mathematical test, a pattern check. Does the daily performance of the variant actually look like a genuine behavioural shift, or does it look unreliable and inconsistent, and luckily happened to average out in the variant’s favour?

A genuine effect tends to produce a variant that leads on the majority of days the test runs, not every day, but consistently more often than not. It produces a daily uplift that fluctuates around a common impact rather than swinging wildly between large positive and large negative values. And it survives the simple test of removing the two or three best days from the variant data: if significance holds after stripping these outliers, the result has a more reliable foundation than one that depends on them.

Consistency analysis is not a substitute for significance or power, it is the layer on top that turns a statistically valid result into one you can actually trust. 

A result that passes significance, is adequately powered, and shows a consistent daily pattern is one worth releasing. A result that passes the first two but fails the third deserves more data before the release decision is made. Like statistical power, consistency warrants its own dedicated treatment, but the principle is worth holding alongside every significant result you see.

AB Testing Statistical Significance Is a Threshold, Not a Verdict

The 95% confidence threshold is not the finish line. It is the point at which the data becomes interesting enough to take seriously, and the point at which the real interrogation of a result can begin. A significant result that was manufactured by a single large transaction, driven by two days of anomalous performance, called three weeks early, or inflated by novelty effect is not a winner. It is a number that crossed a line under the wrong conditions.

The teams that build genuinely effective experimentation programmes are not the ones that release every significant result quickly. They are the ones that understand what significance does and doesn’t tell them, apply the checks that the platform dashboard never will, and hold the bar high enough that the results they release actually hold in production.

If your current experimentation programme treats 95% confidence as the end of the conversation, it is leaving the most important questions unasked. 

Statistical significance is where rigour starts, not where it ends.