Repeating the same test and hoping for a different result isn't a strategy. It's an expensive way to stand still.

There is a version of e-commerce A/B testing strategy that looks like progress. Tests are running. Results are being recorded. The programme is active, visible, and defensible in a slide deck. The problem is that running tests is not the same thing as learning from them (or making commercial progress) and for too many digital teams the two have quietly become confused.

Running the same experiment again, on the same page, with the same hypothesis, because the first result was disappointing and external factors seem like a plausible excuse is not an experimentation strategy. It is wishful thinking with a confidence interval attached. And the cost of it, test by test, is not visible until you look back at six months of activity and ask what, exactly, has changed.

This confusion has a name: Insanity

The Definition of Insanity in E-Commerce Experimentation

In digital experimentation, insanity is systematic. An organisation runs a campaign. Engagement is poor. Rather than interrogating the campaign, the team attributes the result to a difficult trading period, a competitor promotion, or some other external variable beyond their control. The same campaign runs again. The same result follows.

The pattern repeats in conversion optimisation. A team invests weeks in an A/B test; data-backed, stakeholder-approved, briefed with genuine rigour. The test loses. Or worse, it delivers no measurable impact. The instinct is to re-run it: the creative was strong, the hypothesis felt right, perhaps the test window coincided with an anomaly. So, it runs again.

In the overwhelming majority of cases, running the test again simply yields the same result. 

Not only does this waste the resources committed to the test, it holds back the entire experimentation programme. Occupying traffic, time, and analytical capacity that could be directed at the questions the data is actually asking.

Why an E-Commerce A/B Testing Strategy Built on Volume Fails

The volume game is seductive. More tests feel like more progress. Agencies with an interest in activity over impact will happily frame testing velocity as a measure of programme maturity, after all fifty tests a month sounds considerably more impressive than five, regardless of what those tests are actually discovering.

But volume without evolution is both inefficient and ineffective. 

A programme that re-runs failed experiments in search of a better result is not running fifty tests. It is running one test fifty times. The result is a testing pipeline that is busy but not diagnostic, generating what might generously be called academic wins while the underlying reasons why customers are failing to buy remain entirely unaddressed.

For e-commerce organisations, where traffic is finite and every test window consumes meaningful resource, this is not a theoretical problem. A poorly framed re-run that occupies four weeks of test traffic and tells you nothing is four weeks of opportunity cost. 

Scaled across an entire quarter, or year, and the sunk cost of a testing strategy that prioritise volume represents a significant drain on both time and budget.

The Difference Between Iteration and Insanity

The distinction that separates an effective e-commerce A/B testing strategy from an insane one is the purpose of repeated activity. 

In an iterative programme, repetition is never aimed at achieving a different result; it is aimed at understanding why the original result happened, and using that understanding to construct a better question.

The difference is not subtle. Imagine a landing page test built from genuine customer data. On paper, the test should have won. In practice, it had no measurable impact. 

The insane response is to run it again. The iterative response is to dig into why.

Deeper analysis might reveal that while the page content was strong, fewer than 30% of users ever reached the relevant section. For those who did, conversion was meaningfully higher. The failure was not the content; it was the failure to surface it. That insight produces an iteration: a call-to-action that anchors users into the content rather than directing them away from it. The next test is not a repetition. It is a logically connected evolution, built from what the first test actually taught.

This is what a genuine e-commerce A/B testing strategy looks like. Not a library of pre-packaged interventions run on repeat until one of them lands, but a disciplined process of asking why, drawing inference from the answer, and constructing the next experiment from evidence rather than hope.

The Cost of Getting it Wrong

The cost of an insane experimentation programme is not visible test by test, that’s why it’s so costly.

It accumulates. Every re-run that delivers the same result occupies traffic that could have been used to test something meaningful. Every hypothesis framed as a prediction rather than a question produces a result that tells you only whether the prediction was right, not why customers are behaving as they are.

Over time, this dynamic erodes the team's confidence in experimentation itself. Tests run. Results are inconclusive. The programme produces activity but not impact. The natural conclusion, that A/B testing doesn't work for businesses of this size, is the wrong one. The programme was working correctly. It was simply asking the wrong questions.

For a digital team operating on £5M annual revenue, a 1% genuine improvement in conversion rate is worth £50,000. A re-run of a failed test that reaches no significance and tells you nothing is worth exactly that, nothing, simply consuming weeks of the traffic and resources that could have found the real answer.

What an Iterative E-Commerce A/B Testing Strategy Requires

Moving from insanity to iteration requires three deliberate changes to the operating model, none of which involve running more tests.

The first is slowing testing velocity. 

Unless you are operating at enterprise scale with millions of weekly sessions, running dozens of concurrent experiments produces noise, not knowledge. A smaller number of highly focused, evidence-informed tests, each grounded in a specific customer failure identified through behavioural data, on-site engagement, and customer voice, will deliver more commercial impact than a high-volume programme built on borrowed best practice and re-runs.

The second is defining the measurement strategy before the test runs. 

Knowing what you are trying to move, at what stage of the funnel, and for which segment of customer is the prerequisite for a hypothesis worth testing. 'Conversion will improve' is not a measurement strategy. 'Add-to-cart rate on mobile product pages for first-time visitors' is.

The third is aligning every experiment with customer insight. 

The data that should drive your e-commerce A/B testing strategy is not a best-practice library and it is not the preferences of senior stakeholders. It is the evidence of what your customers are actually doing, where they are failing, and what they are telling you about why. When that evidence is the starting point, hypotheses write themselves and when a test loses (and some of them will, even the ones you most confidently thought would win), the loss is informative rather than wasted.

Your E-Commerce A/B Testing Strategy Needs to Ask Why, Not Just What

The fundamental failure of an insane experimentation programme is that it measures what without ever asking why. 

It records results without generating understanding. It produces a pipeline of inconclusive or re-run tests that accumulates activity and delivers no diagnostic value.

A genuinely effective e-commerce A/B testing strategy is not a delivery pipeline. It is a scientific process, one that starts with the question of where the largest volume of commercial value is being lost in the funnel, working backward to understand what is causing that loss, and creates experiments that test specific, evidenced solutions to specific, evidenced failures.

The win rate of such a programme is not the measure of its success. The measure is whether the revenue that was leaking has actually stopped, and whether, when a test loses, the team knows enough about why to make the next test better. That is the difference between an experimentation programme that accumulates knowledge and one that accumulates tests.

Real commercial growth is not found by doing the same thing twice and hoping for a different result. It is found by building a disciplined, evidence-based response to real customer failure, one that evolves with every test, win or lose, rather than repeating itself in search of an outcome the data has already delivered.