Unmoderated studies can be an extremely valuable way to gain findings and insights (yes, these are different) into people’s attitudes and behaviors, and identify areas for improvement in your product or service. But it can be difficult to dig deep when you only have 10–15 minutes. To address this, without extending the timeframe, I often incorporate additional tactics.

These tactics are similar to making an ice cream sundae. The unmoderated study is the ice cream and the tactics are the toppings that enhance the overall experience. Just as a sundae becomes more delightful with a variety of toppings, unmoderated studies can provide richer insights when supplemented with these strategies.

Let’s explore various approaches that can be creatively combined to enhance the unmoderated study experience! 

Gather pre-study inputs

Similar to choosing the right bowl or cup for your sundae, pre-study inputs are a great way to screen respondents, learn more about them in general, and gather information to segment them for comparison later on. Many unmoderated research platforms, including Lyssna, allow you to ask participants questions before they begin the main activity. 

PRO TIP: Consider questions about attitudes and behaviors to understand how each respondent self identifies in regard to the study topic and beyond. I like to keep my pre-study questions close-ended so they don’t introduce too much friction before the main activity begins. Similar to surveys, make sure these questions are scrubbed for bias and always provide an escape hatch.

Collect qualitative data

While unmoderated studies can often focus on quantitative metrics, such as task completion rates and time on task, consider incorporating qualitative data collection methods too. For example, include open-ended questions to gather participants’ thoughts, opinions, and suggestions. Most mature platforms will allow you to gather these responses verbally or in writing. 

I think of this additional data as various sundae toppings (sweet, savory, chewy, colorful, etc.). It can provide valuable context into a participant’s subjective experiences and uncover underlying issues that quantitative data alone may not reveal. 

PRO TIP: Verbal responses will be easier and faster for participants to provide but may be more difficult to analyze! Transcribing the oral responses is very helpful!

Here are a few ideas of qualitative questions to ask:

• “What three adjectives would you use to describe ___?”

• “How successful, or not, were you in completing that task? Provide context on your response.” 

• “How is ____ similar or different from ___? Please elaborate …”

• “What is most/least enticing to you about ____? Tell me more about that.”

• "Were there any moments where you felt unsure or confused? If so, where/when? Provide context …”

Employ think-aloud protocols 

Using think-aloud protocols is akin to someone verbalizing their thoughts while eating their ice cream sundae. Oftentimes, I see unmoderated studies focusing only on task completion or success. This is a missed opportunity. Encouraging participants to verbalize their thoughts and actions WHILE engaging with what they are providing feedback on is a more productive use of this precious time. This can help you understand their decision-making process, uncover usability issues, and gain a deeper understanding of their mental model (including motivations, frustrations, and expectations). Think-aloud protocols can be combined with unmoderated studies when participants record their sessions and you provide prompts or instructions to guide their narration.

Here are some ideas:

• “Please describe in more detail what led you to make that particular decision/choose that approach?"

• “What were your initial thoughts or feelings when you encountered this feature/element/option/name?”

• “What, if any, additional information or context would have been helpful for you while completing this task/activity?”

• “What aspects of the experience stood out to you the most, positively or negatively?”

Mix qualitative and quantitative analysis 

Mixing qualitative and quantitative analysis is like mixing different ice cream flavors or toppings. Consider combining the strengths of qualitative and quantitative data to gain a comprehensive understanding of behavior and attitude.

Analyze the quantitative metrics from the unmoderated study, such as completion rates and time on task, and complement them with qualitative data collected through open-ended questions or think-aloud protocols. By triangulating the qual and quant findings, you can uncover patterns, identify potential problems, and generate more meaningful and actionable insights.

Compare segments 

If you have a diverse set of respondents, consider analyzing the data from different user segments separately (e.g. group participants based on demographics, experience levels, attitudes, or any other relevant criteria that support your study goals). Then compare the interactions and feedback within each segment and against the other segment(s).

You may discover variations in issues, preferences, needs, etc. These learnings can help you tailor the experience you're designing to better meet each segment's specific expectations or requirements. 

PRO TIP: See Gather pre-study inputs above for ideas on how to segment in an unmoderated study if you have not established segmentation previously in a distinct recruiting process.

Utilize session recordings

Unmoderated studies often involve participants recording their sessions while engaging with your concept or design. Take advantage of these recordings to observe respondents’ behavior in detail. Watch them closely and slowly. In other words, study the sundae – its shape, structure, vessel, flavors, presentation, etc.

Look for emotional cues (such as facial expressions), physical hesitation, and other non-verbal communication. Analyze the recordings to identify trends, problems, delight, and/or confusion. This approach allows you to get a firsthand view of how people interact with what you are testing and identify specific areas for improvement.

Combine unmoderated and moderated methods

Unmoderated studies often lack the depth and context that can be obtained through direct conversation and interaction with participants. Alone, they’re similar to a plain vanilla ice cream with fudge sauce. No additional toppings. No interesting vessel. 

Consider combining unmoderated studies with moderated methods, such as in-depth interviews or in-person moderated usability testing. By using a mixed-methods approach, you can gain a more comprehensive understanding of participant behavior and uncover things that might be missed with unmoderated studies alone.

Gather post-study inputs 

This serves as the finishing touch to your sundae. Earlier, we talked about gathering pre-study inputs. In addition to asking questions before the main activity, I often query participants after they’ve completed the central undertaking. These “wrap-up” probes can be a great way to learn more about how your participants think about what was tested in a summary format. You can even ask them to reflect on the experience and recap the highlights and lowlights, what’s working well, what’s not, etc. To me, these are akin to the whipped cream and nuts! 

HEADS UP: These “summarized reflections” can be incredibly valuable time savers when they include the type of feedback you seek. Make sure to focus the wrap-up prompts to clearly align with your study’s goals. 

Conduct post-study interviews

Sometimes, I follow up with participants after they have completed an unmoderated study, specifically to dig deeper. These convos can be the cherry on top! Consider scheduling brief conversations to discuss their experiences, clarify any confusion, and delve into their feedback. 

These sessions can provide substantial context and help you uncover underlying reasons behind observed behaviors or issues that emerged during the unmoderated study.

By considering the unmoderated portion of the study as the main ingredient, you can create a richer research experience that's more meaningful, well-rounded, satisfying and enjoyable. Incorporating a variety of the tactics (or toppings), and an interesting vessel, will add depth and richness to the learning. 

Elevate your research practice

Join over 320,000+ marketers, designers, researchers, and product leaders who use Lyssna to make data-driven decisions.

Sign up for free

Sign up for free

Sign up for free

---
This article was authored by Michele Ronsen, Founder and CEO of Curiosity Tank. Michele is a user research executive, coach and educator. She teaches design and user research to people around the world. Her corporate trainings and workshops are inspired by working with Fortune 500s and start-ups for more than twenty years. Fuel Your Curiosity is her award winning, free, user-research newsletter. In 2020, LinkedIn honored Michele with a TopVoices award in the Technology category. She is the first and only researcher to receive this award.