Why your AB test doesn’t win and what to do about it.

I am confident anyone can agree with me that at least once in your career, you found yourself into this situation: no matter what you do on a page, you can’t get a win. And if your AB test is left longer, you still see inconclusive results.

This is what scientists call the Sensitivity Analysis, the process of recalculating outcomes under alternative assumptions and finding parameters that will change the outcome.

This is the typical mistake that we all have done. But now it’s time to stop that and learn a new thing today. In this article you will learn how to spot irrelevant pages, understand why they give inconclusive results and how to fix it.

It’s a lot to cover, let’s start.

I have done it so many times, thinking that simply changing a text, a CTA, the copy on the page would change the results of my AB test! Then have quit my tests after the third attempt. This is what I thought until recently, when I have discovered a new way of AB testing pages.

I like to make the comparison with Volleyball, sport that I play like there is no tomorrow. When I pass the ball to my very experienced defender, hoping him to hit it and make the point, I actually find out (duh!) that he always gets blocked, or his ball gets out and we lose the point. Then I try to pass the ball at him quicker to prevent blocking, with the same result, no point for the team. No amount of great passing to the defender can change the result because the defender is not an attacker, he is not built for attacking.

This comparison equals a page on the website with no visitors like the “Terms and conditions”. No amount of amending will increase your profits because the page is not connected to your goals.

Both examples are obvious, the defender will never be able to hit the ball strong enough to make a point, because he is not your attacking guy.

You might find out that your home page is disconnected from your goals. I have done the Sensitivity Analysis for a client and found out interesting things that I will share with you.

I have recently worked on the highest conversion page for a company, that page alone generates 24,000 leads a year, which equal approx. to £3,500,000 in revenue per year. I have changed this page twice in 2 AB tests, but the conversion rate hasn’t changed.

I have done so much qualitative research, asked users to answer questions before and after the conversion to find frictions and objections. I have done a lot of comparisons with competitors’ landing pages. And when I have developed the new landing page, I was confident I have done such a great job with the copy, that I would increase conversions by 20%, in the worst-case scenario.

I ran the AB test and suddenly, after a couple of days, I realised there wasn’t any difference. I was discouraged to say the least.

My guess was that the landing page would give no measurable uplift to the number of people who would convert.


Because I have asked users how they decided to convert, they answer wasn’t “I liked the landing page”, it was more like this:

  • “A friend recommended this company”

  • “Your courses are industry accredited”

  • “I have read your great reviews on Trustpilot”.

The main consideration is that the landing page might have had an impact on conversions, but it wasn’t the main lever. In Volleyball terms, the landing page wasn’t my attacker, so how could I expect the page to produce more leads (or make more points for the team??

The landing page might have played a role near the conversion but didn’t influence the decision of my users. The decision happened in the customers’ mind when they talked to their friends, saw the great reviews and accreditations and realised they can trust the company.

So, to influence the decision, you shouldn’t change your landing page, you should identify the persuasive content.

How to identify persuasive content

Before we get into the details of this, I tell you why you need to identify persuasive content.

It’s because this content is the place where conversions happen. And this place might be far away from your landing page.

It’s easy to give up at this point because as marketers we love landing pages, tweaking and amending, improving copy and CTAs. But in the eyes of your users, the landing page is only used to get something valuable for them. When they get there, they have already made their decision.

Let’s take WhatsApp as a good example. If you have a look at WhatsApp home page, it contains very little.

50 words of copy and two images.

Most of all, the page shows only CTAs to download WhatsApp on various platforms and not much else. You might wander why WhatsApp doesn’t need anything else. But visitors don’t get to this page below and think “oh, it looks great, let me convert!”.

By the time they get to WhatsApp’s homepage, they may have already used WhatsApp, so they know what it is and what it does. Or maybe a friend recommended it. Maybe they have read reviews elsewhere.

WhatsApp homepage
WhatsApp homepage

So, what the team at WhatsApp have discovered is that it doesn’t help to optimise the home page on every detail. This page isn’t where the conversion happens. And so, all WhatsApp team need to do is to simplify the sign-up process.

Because WhatsApp is loved by users, then the team at WhatsApp does the right thing. And it continues to optimise not the landing page, but the brand mechanism that makes users land on their home page to download it.

Users make decision outside the homepage or any landing page that WhatsApp is building.

They actually don’t have any.