Use preference testing to increase CRO

Use Preference Testing to Increase Your Conversion Rate – Not Just A/B Testing!

Conversion rate optimization (CRO) is now being used by many smart online businesses to increase the percentage of users that convert into sales or leads on their website, and significantly improve their revenue.

And many of them are now doing A/B testing to help them find out which version of their proposed improvements converts better. 

However, there are three major problems with A/B testing:

1. It doesn’t tell you why your visitors preferred the winning version. It only tells you which version won (if any).

2. It doesn’t give you any feedback from your visitors about the improvements you were A/B testing.

3. Most websites don’t have enough traffic to conduct A/B testing, or sometimes only enough to do it for pages that get the most traffic.

Do any of those sound frustrating issues familiar to you? 

This is why you need to start doing preference testing instead of relying on A/B testing to gain insights and feedback about what you want to improve on your website. 

What’s so great about preference testing?

The preference test at Lyssna is a great alternative to traditional A/B testing because it allows you to gather feedback from your target audience on which page version they prefer, and most importantly, why they prefer it – the current page version or the versions that you’ve tried to improve. And unlike A/B testing, you don’t need any website traffic to do this either!

Your go-to user research platform

The best teams use Lyssna so they can deeply understand their audience and move in the right direction — faster.

Sign up for free

Sign up for free

Sign up for free

How to create a preference test 

First, you need to design one or two improvements to a page on your website that you want to convert better into orders or leads. This could include things like an improved headline, page layout, or imagery. Ideally, the improvements should be based on insights from conversion research like customer surveys, web analytics reports, visitor session recordings, user testing, or CRO audits, rather than just guessing at what to improve, as you will get better results. 

Once you’ve created at least one improved page design, take a screenshot of each version. Then open up the Lyssna Dashboard and create a new test. The first step is to add a ‘preference test’ as the first section of your test and then upload your screenshots.

Next, change the instruction at the top of your preference test to be something that relates to what you’re improving, for example: ‘‘Which version of this page do you prefer?”

In the preference text example below, you can see two different proposed layout changes for a product page.

Next, you need to add the all important ‘short text question,’ which will give you a goldmine of insights. This is what you need to add:

“Why do you prefer that version?”

The answers to this question give you exactly what you don’t get from doing A/B testing – highly useful feedback and insights from your target audience about what you’re trying to improve, and ideas to improve it even further!

That’s the most important question to ask, but to gain further insights, I suggest you also add additional questions about exactly what you have tried to improve, for example:

• What is your favourite part about the version you chose?

• What didn’t you like about the version you didn’t choose?

• How easy was it to notice the differences between the two versions?

Next, you need to go to the ‘recruit’ section of the test and get many users from the Lyssna panel to take your preference test. To make sure they match your target audience, you need to also select the demographics you want. You can target by age, gender, country, household income, education status, employment status, and even B2B demographics like job title and industry.

I suggest getting at least 100–200 users, as this will give you a good amount of responses to analyze. Ideally more is better, but it depends on what your budget is. Each tester will cost between $1–5, depending on if you ask further questions or if you have a very specific target audience. If you ask just 2 questions and have simple demographics for your target audience, this will cost you just $200 for 100 testers, which will be the best money you can spend!

You can also recruit your own testers if you want to save money. For example, you could email your preference test to a random sample of your customers, or to a list of friends or partners of anyone that works at your business. Bear in mind you will get different feedback from users that already know your business, so you may want to do a few rounds of preference testing to include users who may not know about your business.

Run your preference test and analyze the results

After you have chosen your participants, it’s time to get excited and launch your preference test!

Depending on how common the demographics are that you selected, you could receive the results within just a few hours. 

Now, you’ll need to spend time reviewing the results to see which page version your users seem to prefer, and the reasons why.

To help you determine the winning page version, you should create a simple document that highlights the positives and negatives for each page included in the preference test, and list each time it was mentioned. 

You can also use tags in the Lyssna results page, which will help you to identify patterns in common feedback and insights. Filters also help you to drill down and analyze by specific audience groups, like by gender or income bracket, or those who preferred a specific version. 

Make further improvements based on feedback and launch your new design

This analysis gives you great insights and ideas for enhancing the winning page even further, like improving the usability or clarity on the page. And if many users said they didn’t notice the difference between the two pages, you should make the improvement more prominent.

If the new page doesn’t get very good feedback, you need to conduct more conversion research (customer surveys, web analytics reports, visitor session recordings, user testing, or CRO audits) to create and design a better version, and then do another round of preference testing on your new, improved version. 

Once you’ve decided on a final version of the improved page based on the insights and feedback, it’s time to launch your new design. 

To ensure the improved page sees positive results, you’ll need to monitor the impact of your conversion rate and revenue in Google Analytics 4 over the next few weeks, and compare it to the previous weeks and the same period last year.

If you’re lucky enough to have enough traffic (at least 5,000 visitors per week to the page you’re testing), you should ideally run an A/B test on this winning page to determine that it has a positive impact on your conversion rate and revenue.

Preference test all your key web pages

Don’t stop after you’ve done this preference test for one page you want to improve.

To get the best results, do a preference test for each of your key pages to improve them too, particularly your homepage, product pages, and your checkout or signup flow.

So go ahead and try this for your next website improvement. I’m sure you will love the goldmine of insights that you will get, which will lead to launching improvements with a much bigger increase in your website conversion rate and revenue!

Get even more feedback by doing ‘User-Focused A/B Testing’ 

There are even more ways to gain additional, high-impact feedback and insights about your proposed website improvements, which also go into more depth.

This is because the preference test is actually just one part of my great new technique called ‘User-Focused A/B Testing,’ which includes these other high-impact ways of gathering feedback and insights on proposed improvements, in additional to preference tests:

• Record your visitors interacting with the current page version and the improved version

• Set up a visitor survey to gain further feedback on both versions

• Create a user test with 5 users to get detailed feedback on both versions

Read all about the ‘User-Focused A/B Testing’ technique.

Elevate your research practice

Join over 320,000+ marketers, designers, researchers, and product leaders who use Lyssna to make data-driven decisions.

Sign up for free

Sign up for free

Sign up for free

--

Rich Page has 15 years of experience with CRO. With his CRO services, he has increased conversion rates and revenue for hundreds of websites including JosephJoseph.com, ManlyBands.com and UnboundMernio.com. He has also written two popular books about CRO.