Summative usability testing

Depending on where you’re at in the product development process, your UX research will fall into either of these categories: formative or summative. 

As explained in our post on formative usability testing, there may be some debate about which type of testing is better. However, there’s no doubt that using both methods is good practice for successful UX design and research.

This guide will dive into the details of summative usability testing, including:

• What is summative usability testing?
• What’s the difference between summative and formative usability testing?
• When you should use summative testing
• Benefits of summative testing
• Summative usability testing methods
• Summative testing best practices

Like formative usability testing, there’s a small clue in the name — ‘sum’-mative testing helps to test the ‘sum’ of the product, i.e. its overall design or the big picture. The goal of summative usability testing is to figure out how well a current iteration of a product is working compared to previous iterations, or against competitors in the market. 

What’s the difference between summative and formative usability testing?

The main difference between summative and formative usability testing is the stage of product development at which you use these testing methods (more on this in a moment). Another big difference between them is that summative usability testing tends to be more on the quantitative side, whereas formative testing typically uses more qualitative methods of data collection and analysis.

As explained in our formative usability testing article, you can think of formative testing as a ‘bottom-up’ approach and summative testing as a ‘top-down’ approach. 

When you should use summative usability testing

You should use summative usability testing when you have an existing product or highly-developed/complex prototype to test against, i.e. at the end of the product development process, before the cycle begins again. The main reason for this is because of the testing methods used for summative evaluations – they typically need an existing product or usable prototype for the tests to actually work (though there can be exceptions).

However, another reason is that summative usability tests often work best when there’s a large sample size – which means a product with an established customer base will likely see the best test response rates. 

Benefits of summative usability testing

There are a few obvious benefits to doing summative testing, including:

• They’re generally cheaper than formative testing methods. That being said, the cost efficiency of summative testing depends on the scale of the tests you do. If you need thousands of global responses, this could end up costing you more than a series of 1:1 in-person interviews.
  
  
• They’re easier to scale. If you have the means for it, summative testing methods are far easier to scale than formative methods – you create the test once and then send it out to however many participants you need.
  
  
• They’re nearly always quicker to complete. You can reasonably receive results from a summative usability test (especially when you do them remotely) within a couple of days, as opposed to needing to source potential interviewees and organize for interviews to take place.

While I mention the comparison to interviews as a method (for formative testing), you can also use interviews as a summative testing method, providing you ask particular kinds of questions – I’ll cover this in more detail shortly.

Summative usability testing methods 

As you might have guessed, many summative testing methods are unmoderated and you can do them remotely. Here we’ll look at some common methods and what makes them particularly useful.

Card sorting 

If you need to test the information hierarchy of your site, such as item categories on an ecommerce store, a card sorting test is a great option. With a card sorting test, you can ask your participants to organize items into product categories that make sense to them. Another use case for card sorting can include figuring out how your audience would categorize support documentation or helpful information in a knowledge base.

Card sorting also has the benefit of being highly visual, and you don’t need to use a lot of mental effort to complete them. When tests are easier to complete, you have better chances of getting higher completion rates.

First click tests

Another way to test your information architecture or the effectiveness of your CTAs is using a first click test. These tests measure the usability of a website, app, or design by finding out how easy it is to complete a task. For instance, you could ask your participants, “Where would you click to see a list of your saved content?”. You could then ask follow-up questions about what drew your participants to click where they did.

First-click tests are important for helping you figure out how your audience interacts with your product or website. The First Click Usability Testing study by Bob Bailey and Cari Wolfson found that if the first click was the correct option, users had an 87% chance of completing the action correctly, as opposed to 46% if the first click was incorrect.

Five second tests

Remember how I said most summative methods are quantitative? Well, five second tests are one of the few unmoderated, remote testing methods that are more qualitative (that being said, you could also do this test in person). 

A five second test measures what your audience recalls about your product or webpage in the first five seconds of exposure. An example question here would be “Please write down all the main elements you remember from the page you saw.”

From the responses you get, you can analyze the results using word cloud visualizations to see if there are any trends (both expected and unexpected). 

Interviews and/or focus groups

The final methods you can use (that I’m detailing in this post) are interviews and focus groups. Again, this method is more qualitative, and it mostly suits formative usability testing, but you can still do summative-based interviews and focus groups.

For instance, I recently took part in a focus group for a utility company that wanted to figure out how people would react to changes in its compensation criteria (for bad service). This session was very much summative as there was already an existing set of criteria, but they were testing potential changes. 

While interviews and focus groups can be slow and expensive, if you’re looking to make big changes where the wrong decisions could cost the business even more in the long run, they can be worth handing the cash over for those deep insights.

Summative usability testing best practices

It’s all well and good to know what kinds of testing methods are at your disposal, but it’s another to know how to run summative usability tests in the best possible way to get meaningful results. Here, we’ll look at a few summative testing best practices to help you hit the ground running. 

Ask the right questions

We have an in-depth guide on usability testing questions that covers what to ask throughout the product development process, however it’s easy to steer off the path of summative testing by accidentally asking questions that are better suited to early discovery work.

For example, the questions you’d ask in a formative usability test would be along the lines of:

• “What type of product would you use to complete X task?”
• “Can you give an example of when you would use X type of product?”

For summative testing, you need to get more specific about the use of your product/website. For example:

• “How would you describe the language on this page?”
• “What features do you find most valuable about this product?”
• “Can you name any competitors that you currently use to do X? [related to existing feature]”

These questions can help you narrow the scope of your research and make sure you’re not doing extra, unnecessary work during your analysis.

Avoid giving hints to participants

While providing some guidance during formative usability tests is okay (e.g. a participant tries to click on a non-functional part of a prototype design), you want to avoid this practice during summative testing.

The reason for this is to make sure participants take the paths that are natural to them. By doing this, for example during first click tests, you reveal whether your existing information hierarchy has any issues you need to address in later product updates.

Thoroughly plan and prepare

I said earlier that with some summative testing methods you can build the test, send it out, and get responses within a matter of hours (or even minutes!). However, don’t let this potential for quick turnarounds get in the way of good planning and preparation. 

On this subject, David Watkins, Director of Product Development at EthOS, explains:

‍“By carefully planning and preparing for the testing process and approaching it with an open mind and a willingness to learn from feedback, you can gather valuable insights that can help to improve the user experience and drive the success of your product.”

He also gives some practical tips on preparing to conduct summative tests by saying:

‍“It would help if you first started by defining clear objectives, selecting appropriate participants (based on personas if they are available), creating relevant tasks, and selecting a straightforward platform that will be easy for your staff and for participants to use.”

Start using summative testing for your UX research 

Summative usability testing is a vital component of the product development process. Just because you’ve launched your product doesn’t mean the work is finished! There are always improvements you can make. As you gain feedback from customers and practice a continuous product discovery mindset, you’ll give your product the best chances of success in the long run.

If you’re ready to gather actionable data with summative UX research methods, sign up for a free UsabilityHub plan and get access to unlimited active tests! 

Frequently asked questions about summative usability testing

What is summative usability testing?

Summative usability testing refers to evaluating the overall design of a product. It’s used to assess how well the current iteration of a product performs compared to previous iterations or competitors in the market. Unlike formative usability testing, which focuses on qualitative methods of data collection and analysis, summative testing tends to be more quantitative in nature.

What’s the difference between summative and formative usability testing?

The key distinction between summative and formative usability testing lies in the stage of product development at which these testing methods are employed. Summative testing is typically conducted at the end of the product development process, when there’s an existing product or a highly-developed/complex prototype to test. This approach allows for evaluating the product's performance and gathering feedback before starting a new development cycle. Formative usability testing refers to the evaluation of a product or design during the early stages of its development. It aims to gather qualitative feedback and insights that can inform the design. This type of testing focuses on understanding user behavior, preferences, and pain points to guide the decision-making process.

What methods can you use to conduct summative usability testing?

Summative usability testing can be conducted using various methods. Card sorting allows testing of information hierarchy by asking participants to organize items into categories. First click tests evaluate the effectiveness of information architecture and the ease of task completion by analyzing the initial click participants make. Five second tests capture participants' recall of main elements from a product or webpage within the first few seconds. Interviews and focus groups, while commonly associated with formative testing, can also provide qualitative insights in a summative context, facilitating in-depth understanding of features, language, value, and competition. These methods can be conducted remotely and at scale, but careful planning and preparation remain important for effective summative testing.

Alexander Boswell is a freelance writer specialising in B2B SaaS and eCommerce marketing with a particular interest in the world of data, as well as a business Ph.D. candidate. When he’s not writing, he’s nerding out playing D&D and Magic: The Gathering.