Preference testing

Preference testing is a powerful tool for assessing user preferences among various design options, making it essential for anyone involved in web or app design.

In this video, we'll cover the basics of preference testing, when and why to use it, and how to set up your own preference test using Lyssna. To provide deeper insights, we'll also show you how to ask follow-up questions, analyze the results, and filter responses effectively. Whether you're new to preference testing or looking to refine your skills, this tutorial has you covered.

Resources mentioned in the video:

• Lyssna templates library

• Preference testing guide

Transcript

Introduction (0:00)

Hey everyone, this is Matt here and today I'm going to walk you through a quick demo of preference testing using Lyssna. Now, before we get started, I just want to give a brief overview of what preference testing is and when you might use it. So, preference testing is exactly what it sounds like. You can use it to test two or more design options and see which one your users prefer.

And you can test a whole bunch of different things. So, you might test graphic designs, such as logos or colorways. You can test marketing assets like ads, product concepts, or UI elements like iconography. There are a lot of different options. Another thing that preference testing is really good for is resolving conflicts.

So, if you have a couple of design concepts and your team is divided on which way to go, you can get your users to help you make that decision. Okay, so now that we know a little bit about what preference tests are, let's jump in into Lyssna and I'll show you how to set them up.

How to create a preference test in Lyssna (1:05)

So here we have the Lyssna dashboard, and I'm going to go ahead and create a new test.

So the first thing that we want to do is set our test name, and here it's going to be Logo Preference Test. And then we're going to scroll down to our sections and select Preference test. Next thing we're going to do is upload some designs. We can see that we are uploading two different logos. W can preview those.

So we've got sort of a blue on green version and then a slightly different treatment which is green on blue with a little bit of additional copy. And what we're going to do is just ask people to please choose the logo they prefer. Okay, now to help us out with our data analysis and provide a little bit more additional context, we're going to ask people a follow-up question.

So here what we're going to do is we're going to ask them why did they choose this design? And to give them a little bit more room to provide their answer, we're going to select long text. Now one thing to point out here is this randomize order of choices checkbox. You can turn this off and when you do that, the images are going to be presented in the order that you uploaded them in, or you can reorder them to suit whatever order you want.

But generally we suggest that you keep this randomization toggle turned on, and that helps remove any biases that might occur from the sequence of the designs presented. The other thing to mention is obviously, Lyssna is a modular test builder, so you can add additional sections as well, either before or after.

So for instance, we might ask a five second test to get people's first impressions. We might ask some additional survey questions. Here we're going to keep it really simple and straightforward and just have a preference test section. And, also it's good practice to preview the test, to see what the participant experience is like. Super simple this time around. So we're just going to go ahead and continue.

How to recruit preference testing participants from the Lyssna research panel (3:10)

And diving into our recruitment options, there are two ways that we can recruit with Lyssna. One is using a link, so each test has a unique link that you can turn on and use that to invite your own participants. Or you can leverage our panel, which has over half a million participants that you can order from.

So we're going to go ahead and do that. And this is super straightforward as well. All we need to do is just specify the number of participants that we want. So here we might say 25. And then there are a bunch of different demographic options as well that we can choose from. So, let's say that we're interested in folks from the United Kingdom, we can do that.

And maybe there's a specific gender that we're after, or age group. A lot of different options for targeting. And then, as we add our demographic options, they're summarized on the right hand side here, and additionally, we give you an estimate of turnaround time as well. So for this particular order, we're seeing a turnaround time of 43 minutes to get 25 participants from the United Kingdom.

We're obviously not going to wait that entire time, so I've gone ahead and created this test already and recruited some participants. So let's check that out.

How to analyze and share preference test results in Lyssna (4:24)

So here is our test. We've got our two logos here, and we can see clearly that 72 percent of participants actually chose logo one over logo two. Now, when you're uploading two designs, we actually calculate statistical significance for you. So, in this case, this result is statistically significant, and we can be 99 percent certain that this particular result is not due to just simple, random chance.

So this gives us a sense of confidence in these particular results. The other thing that we did was we asked this follow up question, why did you choose this design? Now, here, obviously, you know, this is just providing answers for everyone. And what we actually want to do is see why people chose design one, or why people chose design two.

And we can do that by using filters. So, if we're interested in the folks that selected logo one, we can filter down to those. And so now these 18 participants are the ones that are shown here. So these are all the responses that are shown by the folks that picked logo one. And the other thing that I've done here is I noticed when I was looking at these results that color was mentioned quite a bit.

Now, these are obviously just free text responses, so they're qualitative in nature, but we can actually make these a little bit more quantitative. So what I've done is I've tagged up all the instances of color. And then I selected those and added the color tag so we can see, for instance, if we remove this filter, there might be some other instances of color.

So yes, there are, there are indeed. So we'll go ahead and select all of those and just make sure that every instance of color is mentioned or is tagged rather, then we'll go ahead and deselect that. And we can see that 28 percent of participants said something to do with color. And we can compare this quantitatively by saying, rather by filtering and seeing that 14 percent of participants mentioned color when choosing option two. And 33 percent mentioned color when choosing option one. So it seems that color was one of the determining factors when making this particular choice. So by combining tags and filters between different sections, you can really dive deep into the specifics of what your users have said and why they've said it.

And you can also make that a lot more quantifiable as well. On the left hand side here we have our demographics, and the nice thing is that you can filter on these as well. So, you know, we've got a cross section of men and women in this test, so if we're interested in just the preferences of women in this sample, we can filter down to those versus the preferences of men, and we can filter down to those. So this is just an example of how you might use demographic filters to dive deeper into your data as well. Last, but certainly not least, you can export all of this data out as a CSV for analysis in another application if you so choose. And, additionally you can also make comments.

So if you have team members that you'd like to invite to collaborate with on analyzing results. You can do that and simply at mention them, and they'll receive a notification. So, that's preference testing in Lyssna. And as I mentioned, there are many ways that you can use it to help you make design decisions, and I hope this has been helpful.

Please feel free to give us a like and subscribe for more updates. And if you're looking for more advice on preference testing in general, we have a bunch of resources linked in the description below and also links to preference test templates that you can modify to suit your needs. So, thanks so much for watching and we'll see you again soon. Cheers.