Preference Testing

When choosing is tough, let somebody else do it!

Introduction

Preference tests help you to choose between design variations by simply asking users which one they prefer. When taking part in a preference test, the participant is shown a number of design options to consider and is asked to choose the one they most prefer.

These tests are commonly used to measure aesthetic appeal, but participants can also be instructed to judge designs based on their perceived usability, trustworthiness, or how well they communicate a specific message or idea.

There is no limit to what can be tested, with common test subjects including website designs, mock-ups, prototypes, icons, logos, color palettes, and even copy.

Preference tests produce both quantitative and qualitative feedback.

Quantitative feedback includes the number of people who preferred each design. From this feedback you can calculate the statistical significance of the result. In this context, statistical significance can be defined as the likelihood that the best-performing design is actually the favorite, and isn’t outperforming the other designs by random chance. The level of significance you can obtain will vary depending on your sample size, with larger sample sizes giving you greater significance. It will also depend on the degree of difference between the designs’ performance, with large differences in performance giving greater significance.

Qualitative feedback is obtained by asking participants why they chose the design they did. This qualitative feedback is particularly important because it allows you to find further areas for design improvement and can inform future decisions. Sometimes it’s possible to use this feedback to select the best parts or attributes of the tested designs to produce a hybrid.

A key benefit of preference testing is that it can be done before a product or design is completed, meaning you can gain feedback early on in the process and adjust it as needed. This is different to traditional split testing, which requires an audience (which you may not have for an unfinished or very new product). In fact, preference testing is often most valuable early in the design cycle as it informs the direction of wireframes or mock-ups before an investment is made to create a final design.

Testing

There are a number of common questions that are asked in preference tests. If you’re unsure what you should be asking, these questions can help to guide your test setup.

  • Design aesthetics – which design does a user prefer visually?
  • Perceived usability – which layout is easiest to interact with, or makes the most sense?
  • Favored choice – which option would you be most likely to buy from, or sign up to?

Or test for a particular theme or attribute:

  • Which design looks the most trustworthy?
  • Which design looks the most modern?
  • Which design looks the most serious?
  • Which design looks the most friendly?

Mobile Testing

Testing the design of your mobile website or app can allow you to find user preferences on smaller devices. By testing wireframes or mock-ups early on in the process, you can uncover new ideas and gain guidance on design decisions before the design is finalized.

Key questions

  • What kind of mobile layout makes sense to users?
  • What does a user like about a competitor's design compared to mine?
  • Would people buy from this design?
  • Is the app icon eye-catching and enticing in the App Store?

Icon Testing

Icons are extremely varied in their design and use. Preference testing allows you to gain insight into which icons participants prefer and which icons are most intuitive to use.

Key questions

  • Do users like the appearance of an icon or icon set?
  • Do users understand the meaning and function of an icon?

Example tests

Logo Testing

Business logos are essential in the digital age. Preference testing logo options can provide valuable insight into how people respond to a logo before it is taken to the wider world.

When commissioning a new logo, clients are often presented with a number of options or directions to choose from before revisions are made and a final design is produced. This presents a perfect opportunity to run a preference test, where you can measure the preferences and sentiments of a larger audience rather than simply choosing the option that appeals most to the client.

Rebranding is another process that can benefit from preference testing, particularly for businesses and organizations with an established logo and brand image. Testing new candidate logos and brand assets with a select panel of existing customers helps to avoid some of the pitfalls of a rebrand that can surface after a big reveal.

Key questions

  • What do people like about a logo?
  • Do people like a new logo more than the old one?
  • Which logo best projects the image you want your business to project?

Example tests