An introduction to preference testing

 There’s no accounting for taste, but numbers don’t lie.

Introduction

Preference tests can help you to choose between design variations, by simply asking users which one they prefer. 

When taking part in a preference test, a participant is shown a number of design options and is asked to pick a favorite.

These tests are commonly used to measure aesthetic appeal, but participants can also be instructed to judge designs based on their trustworthiness, or how well they communicate a specific message or idea.

There are no restrictions around what can be tested. Common test subjects include logos, color palettes, icons, website designs, mock-ups, prototypes, copy, and t-shirt designs — the sky is the limit.

A key benefit of preference testing is that it can be done before a product or design is completed, meaning you can gain feedback early on in the process and adjust it as needed.

Preference testing produces both quantitative and qualitative feedback, which makes it a very time-effective evaluation tool.


Quantitative feedback

The number of participants who preferred each design is shown on the results page, which is useful for a straightforward comparison of designs. From this, you can calculate the statistical significance of the result. 

In this context, statistical significance is defined as the likelihood that the best-performing design is actually the favorite, and isn’t outperforming the other designs by random chance. 

The level of significance you can obtain will vary depending on your sample size, with larger sample sizes giving you greater significance. It will also depend on the degree of difference between the designs’ performance, with large differences in performance giving greater significance.


Qualitative feedback 

Qualitative feedback is obtained by asking participants why they chose the design they did. You can then categorize this feedback into groups to get a high-level view of how many participants had similar feedback. 

This type of feedback is particularly important because it allows you to find further areas for design improvement and can inform your future decisions. 

Sometimes it’s possible to use this feedback to select the best parts of the tested designs to produce a hybrid design.


Common questions

There are a number of common questions that are asked in preference tests. If you’re unsure what you should be asking, starting with one of these questions can help guide your test setup.

  • Which design does a user prefer?
  • Which design looks the most trustworthy?
  • Which design looks the most modern?
  • Which design looks the easiest to use?

Examples