Planning a Test
The first thing to know about planning a usability test is that every test is different in scope, and results will vary a lot depending on the purpose and context of the test. Testing a single new feature will look very different from testing several key scenarios in a new Application.
What Are You Going to Test?
Next, you need to decide what you’re going to test. The best way to do this is to meet with the design and development team and choose features that are new, frequently used, or considered troublesome or especially important. After choosing these features, prioritize them and write task scenarios based on them. A task scenario is a story that represents typical user activities and focuses on a single feature or group of related features. Scenarios should be:
- Short - Time is precious during usability testing, so you don’t want to spend too much time on reading or explaining scenarios.
- Specific - The wording of the scenario should be unambiguous and have a specific end goal.
- Realistic - The scenario should be typical of the activities that an average user will do on the Application.
- In the user’s language and related to the user’s context - The scenario should explain the task the same way that users would. This emphasizes the importance of the pre-session discussion, which gives you the opportunity to understand the participant’s relationship with the Application.
Here’s an example scenario for an Application that sells images:
Ex: You’re looking for an image that you can use on your company’s support Application. Find an appropriate image and add it to your basket. Be sure to let me know when you’re done.
Who is going to evaluate the Application?
Who you choose to evaluate the Application will have a massive effect on the outcome of the research. It’s very important to develop a thoughtful screener for recruiting your participants.
Imagine that you’re creating an Application that sells images. Your customers are people who want to buy images—a huge group of people. Narrow your focus to a short and concise user profile, a picture of your ideal test participants. This profile should be based on your primary user (customer) segment and contain characteristics that those users share.
In this scenario, our participants are graphic designers or other people who use graphic design software and purchase images online. Create and order a list of these users’ characteristics. While you’re creating the user profile, you may realize that you have two or more equally important subgroups—people who buy images for business use and people who buy images for home use. This is fine as long as you can justify the relevance of each subgroup to the features that you’ll be testing.
- Test with a reasonable number of participants —The best results come from testing no more than 5 users and running as many small tests as you can afford.
The most striking truth of the curve is that zero users give zero insights.
As soon as you collect data from a single test user, your insights shoot up and you have already learned almost a third of all there is to know about the usability of the design. The difference between zero and even a little bit of data is astounding.
When you test the second user, you will discover that this person does some of the same things as the first user, so there is some overlap in what you learn. People are definitely different, so there will also be something new that the second user does that you did not observe with the first user. So the second user adds some amount of new insight, but not nearly as much as the first user did.
The third user will do many things that you already observed with the first user or with the second user and even some things that you have already seen twice. Plus, of course, the third user will generate a small amount of new data, even if not as much as the first and the second user did.
As you add more and more users, you learn less and less because you will keep seeing the same things again and again. There is no real need to keep observing the same thing multiple times, and you will be very motivated to go back to the drawing board and redesign the Application to eliminate the usability problems.
After the fifth user, you are wasting your time by observing the same findings repeatedly but not learning much new.
You need to test additional users when the Application has several highly distinct groups of users
Article is prepared by:
Anushka Wickramaratne
Senior QA Engineer
Anushka Wickramaratne
Senior QA Engineer
