01 Nov 2023
|11 min
Unmoderated usability testing
A quick guide to understanding unmoderated usability testing, including remote testing and the benefits, drawbacks, and tools available to run unmoderated tests.
When you think of tests, your mind might take a trip down memory lane to school exams – students sat at their desks, diligently completing test questions (or so teachers hoped), with the presence of an external moderator or a member of staff.
Unmoderated usability tests are sort of the opposite of that (especially remote ones, more on that later). There’s a test, sure – but there’s no active moderator.
Now you might be thinking “how do any of these unmoderated tests get done properly?”, with the Lord of The Flies making an appearance in your mind. But of course, this is a reasonable question. What are unmoderated usability tests really? And what are they for?
This quick guide will help you understand what unmoderated usability testing is (and its remote method), the benefits and potential drawbacks you can expect from conducting these kinds of tests with your users, as well as tools you can use to help properly manage them.
What is unmoderated usability testing?
As you might have inferred from the introduction, unlike moderated usability testing, unmoderated tests don’t involve a researcher actively asking questions to participants in real time.
Instead, these types of tests are less hands-on. Typically, the team conducting research will design a test and invite participants to take part, who then complete the test at their own pace.
Unmoderated usability tests might include some prompts or specific tasks to complete. But otherwise, participants have free roam to explore the test subject (e.g. a pre-launch app design or feature of an existing app).
What is remote unmoderated usability testing?
For the most part, users complete an unmoderated usability test remotely using the internet or an app you’ve designed specifically for testing.
This means you’ll build a test that you need to personally monitor, and the software being used will provide any basic instructions needed (for the test to function), and it will record the user’s behavior as they explore the test subject.
Your go-to user research platform
The best teams use Lyssna so they can deeply understand their audience and move in the right direction — faster.
The benefits (and drawbacks) of unmoderated usability testing
As you might expect, unmoderated usability testing comes with several benefits and some potential drawbacks. Here we’ll go into detail about these pros and cons, as well as how you can avoid those shortcomings.
Benefits
Saves time and money. One of the biggest benefits of unmoderated usability testing is the savings gained in both time and money. You can plan and conduct a remote unmoderated test within a few hours, whereas with moderated tests you need to set up schedules for users/researchers and likely need more incentive to block a specific amount of the users’ time.
Phillips Lighting, for example, was able to quickly set up five-second tests that showed issues with customers misunderstanding one of their homepages, and made changes to improve these issues (and convince stakeholders of their importance).
More data to make business decisions. With quantitative unmoderated usability tests, you’re able to gather data and gain more confidence in making decisions.
For example, after The Economist Group (TEG) started using remote unmoderated testing tools, they tested two contentious design choices and solved the dilemma using data from testing. As they continued to use unmoderated tests, they discovered that their previous research was being shared among new teams and projects, which promoted even quicker and easier decision-making.
Better able to scale. As well as only taking a few hours to set up, you can send unmoderated tests to much larger quantities of people compared to conducting in-person, and even remote, moderated tests. For example, Lyssna offers a panel of over 530,000 users ready to test your designs, with easy recruitment. On top of this, it’s easy to adjust and re-test with remote unmoderated testing tools.
For example, Carter Digital Research used remote unmoderated tests to figure out how long it took for customers to find an emergency roadside assistance number on the RACV website – after multiple iterations of designs and testing, they managed to improve the success rate of finding information (and average time spent finding it) by 75%.
Potential drawbacks
Little guidance for participants. By its nature, unmoderated usability testing deliberately doesn’t provide any supervision or real-time guidance on completing tests. As a result, sometimes users might get stuck and not know how to continue, thus being unable to complete the test. They might also behave in unexpected ways, which can be both a blessing and a curse.
To help avoid this drawback, be sure to share your testing goals with participants and provide clear and simple instructions.
Users might not behave realistically or authentically. Remember that Lord of The Flies reference I mentioned earlier? People simply behave differently when they know they’re being watched (for experimentation purposes). In social psychology circles, this is known as the Hawthorne Effect.
As a result of knowing about the experiment (and perhaps its purposes), users may behave in a way they feel you want them to, and be socially motivated to complete the experiment. While this is more of a pressing issue for moderated testing, it can be a problem for unmoderated testing too. Unfortunately, there’s little you can do about this effect, but you can account for it in your results and analyses.
Weighing up the pros and cons of unmoderated usability testing, you can decide whether or not the method is a good fit for your UX design projects.
What tools you can use to conduct unmoderated usability tests
If “Unmoderated usability testing sounds perfect!” is what I’m hearing from you, then you’re probably wondering about the next steps: what tools can you use to conduct these types of tests?
We have already published an extensive comparison guide on the top usability testing tools available, so I won’t cover what these different platforms offer. Instead, I’ll dive deeper into what kind of test and tool features to look out for to help you gather the best remote unmoderated test data.
1. Prototype testing
Prototype testing is an essential tool for testing your digital products, no matter if it’s conducted moderated or unmoderated. It’s pretty much non-negotiable if you’re building a new product or feature. One of the areas that might make a difference between one platform and another is its integration with other design tools such as Figma.
In Lyssna for example, you can import a prototype design from Figma in just a few clicks, send it out to users, and receive responses within minutes.
2. Design surveys
Another important remote unmoderated testing tool for any UX researcher and designer is the humble design survey. A design survey is similar to a prototype test, but you can build a design survey much earlier in the research process to test different static designs before moving forward with functional builds.
Design surveys can be quantitative (in the form of Likert scales, multiple choice, or sorting questions), or qualitative (short and long text response fields).
3. Five second tests
I mentioned five second tests earlier in this guide as a quick and easy method for gathering data on user behavior. These tests are a great way to measure users’ first impressions by seeing how well a design communicates a particular message. Another impressive feature of five-second tests is that they can also provide both quantitative and qualitative feedback from respondents.
You can use five second tests to answer questions like:
Do people understand the product/service?
Do people feel they will receive a benefit from the page?
Can people recall the company/product name?
4. Preference tests
Preference tests might sound similar to design surveys, however you can use preference tests to have users choose from multiple design options in response to specific questions. For example “of the following four logos, which design do you think is best, and tell us why you picked it”.
Using a preference test with multiple designs can help you understand what matters to your users and what doesn’t.
5. First click tests
Another essential tool for UX design and research is the first click test. These tests help you measure the physical usability of your product by finding out how easy it is to complete a given task.
Tools like web analytics can tell you where users clicked on your website and how long they stayed on the page, but they can’t tell you why or what they were trying to do. By using first click tests, you can get tangible data on user behavior so that you can isolate problem issues and fix them quickly.
6. Card sorting
Another useful tool for UX designers and researchers conducting remote unmoderated usability testing is card sorting. This type of test is a well-established research method that lets users sort cards into categories.
This method of testing can help you make more informed decisions about the information architecture (IA) of your site or app as well as discover the mental model of your target users.
All of these tests are incredibly useful tools for remote unmoderated usability research, and choosing a platform that has more than one of these options can be the most cost-effective approach to moving forward with practical design projects.
Start building unmoderated usability tests today
If you want more fast and scalable research data, unmoderated usability testing is the way forward for you. The different tests you can build and send out quickly make this type of testing a winner for many UX design and research projects.
But of course, it does have its limitations – which means remote unmoderated testing might not be the best fit for every project. Whenever you’re thinking about conducting new usability tests, you should assess which type of testing would best fit the needs of your research questions and outcomes.
Elevate your research practice
Join over 320,000+ marketers, designers, researchers, and product leaders who use Lyssna to make data-driven decisions.
Frequently asked questions about unmoderated usability testing
--
Alexander Boswell is the Founder/Director of SaaSOCIATE, a B2B SaaS, MarTech and eCommerce Content Marketing Service and a Business PhD candidate. When he’s not writing, he’s playing baseball and D&D.
You may also like these articles
Sign up to our newsletter
We'll keep you updated with the latest UX insights and more.
